The seedbox.tasks.subprocessext Module

Extension of subprocess.Popen

Extends the subprocess.Popen class to provide logging threads for stdout and stderr of the child process such that it will log stdout through logging module to provided logger.info() and log stderr to provided logger.warn().

Because using subprocess.PIPE for stdout and stderr with any of the convenience methods or with the wait() command has potential for buffer issue, we leverage Popen directly with PIPE but then spawn a thread with access to the corresponding logger function on the specific pipe. And then leverage the poll method to wait for child process to complete. If the returncode is not 0, then just like check_call() we will raise the CalledProcessError to be as consistent as possible. This is done via the staticmethod execute(). Because we subclass Popen, you can simply create the instance and leverage Popen methods for interaction if you prefer.

The intent here is to have the output send directly to the logger vs. being buffered until the end with the communicate() method, or dealing with buffer conflict issues like wait(). The logger can handle the different levels, StreamHandler or even FileHandler and the corresponding subclasses to make it more manageable.

This was derived from several different responses on stackoverflow and blogs on this topic. After pulling them all together and doing several rounds of testing, this was the result.

class seedbox.tasks.subprocessext.ProcessLogging(cmd)

Bases: subprocess.Popen

Run a command as a subprocess sending output to a logger.

Initializes new instance.

Parameters:cmd (list) – command and options sent to subprocess to execute

complete()

Handle all the processing of the subprocess

static execute(cmd)

Simple convenience method.

Provide a convenience method for creating the object and waiting for the results; very similar to check_call() but not at module level.

Parameters:cmd (list) – command and options sent to subprocess to execute