Project

General

Profile

Actions

Bug #1528

open

ctfestimate,py deadlock on subprocess.wait()

Added by Anchi Cheng almost 13 years ago. Updated over 12 years ago.

Status:
In Code Review
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
01/18/2012
Due date:
% Done:

0%

Estimated time:
Affected Version:
Appion/Leginon 2.1.0
Show in known bugs:
No
Workaround:

Description

According to python doc, this can happen when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data.

This starts happening on CentOS installation and new ctffind3 that I downloaded. Maybe the buffer is smaller here or the output is larger.

Actions #1

Updated by Anchi Cheng almost 13 years ago

  • Status changed from Assigned to In Code Review
  • Assignee changed from Anchi Cheng to Jim Pulokas

r16257 fixed this particular case. Not sure how many of other cases where we use wait() need to be changed. According to python doc communicate() can only be used when the returned data can be handled by memory.

Actions #2

Updated by Jim Pulokas over 12 years ago

The warning in the python docs only indicate that it will block if the child stdout or stderr are connected to PIPE, but in this case only the stdin is connected to a PIPE. The output is connected to an open file object (logf), so not sure how the warning applies in this case. I guess the doc is wrong because it obviously fixes it to use communicate.

Other cases where this could hit us again in appion:

Checking where we use proc.wait():

grep -r "proc.wait()" * |grep -v svn
returns 165 lines. Those are not necessarily all bad because many do not redirect stdout,stderr to pipes or files, they just let it out through the parent process stdout/err. Can also check how many Popen() were created with stdout/err/in redirection:
grep -r Popen * | grep -v svn | grep std
returns 63 lines

I will try to fix as many as possible, but I think we don't have to worry about getting them all since no other problems have been reported.

Actions

Also available in: Atom PDF