Python subprocess: run commands and manage child processes

Python's subprocess module starts external programs and exposes their exit status and input/output streams. Use run() when you want to wait for a command to finish. Use Popen when you need to manage its lifetime or stream its output yourself.

The Python interface is portable, but command availability, shell syntax, signals, and process-tree behavior still depend on the operating system. The ls, grep, cat, and make examples below assume suitable Unix tools.

Run a command and inspect the result

import subprocess

result = subprocess.run(
    ["ls", "-l", "/tmp"],
    capture_output=True,
    text=True,
    timeout=10,
)
print(result.stdout)
print(result.returncode)

Each argument occupies one list element. Spaces within an element remain part of that argument; no shell parses the list by default. capture_output=True captures both stdout and stderr. text=True decodes them into strings using the selected or default encoding; specify encoding when the child's output contract requires one.

A zero exit status usually indicates success, but interpret nonzero statuses according to the program. For example, grep uses 1 for “no matches,” which is not an execution failure in every workflow.

Distinguish command failure from launch failure

This self-contained example deliberately exits unsuccessfully:

import subprocess
import sys

try:
    subprocess.run(
        [sys.executable, "-c", "import sys; print('failed', file=sys.stderr); sys.exit(2)"],
        check=True,
        capture_output=True,
        text=True,
        timeout=30,
    )
except subprocess.CalledProcessError as error:
    print("Exit status:", error.returncode)
    print("Standard error:", error.stderr)
except subprocess.TimeoutExpired:
    print("Command exceeded its timeout")
except OSError as error:
    print("Could not start the command:", error)

check=True raises CalledProcessError for an unsuccessful exit. Without it, check returncode yourself. A missing executable or an OS-level launch problem follows a different path, commonly an OSError subclass.

run(timeout=...) kills and waits for the direct child when the communication timeout expires. Process creation itself may not be interruptible, and descendants need separate handling. A 30-second setting is not a universal wall-clock bound on the whole process tree. See the subprocess reference.

Send data through standard input

import subprocess

log_text = "info: started\nerror: request failed\n"
result = subprocess.run(
    ["grep", "error"],
    input=log_text,
    capture_output=True,
    text=True,
    timeout=10,
)
if result.returncode not in (0, 1):
    result.check_returncode()
print(result.stdout)

Python supplies the input directly. You do not need a shell pipeline or a temporary file. The string input matches text=True; without text mode, supply bytes.

Use a shell only when its syntax is required

A fixed Unix shell pipeline can use expansion and redirection:

import subprocess

# Run only in a directory whose log files are intended inputs.
subprocess.run(
    "cat *.log | grep ERROR > errors.txt",
    shell=True,
    check=True,
    timeout=30,
)

This writes errors.txt, and the usual shell pipeline status reflects its last command. check=True does not reliably detect a failure of an earlier pipeline component. Use explicit subprocess connections or Python file processing when you need precise error handling for each stage.

Never build such a command by inserting untrusted text. For example, interpolating a filename into cat {filename} with shell=True allows shell metacharacters in the filename to become commands. Prefer an argument list:

import subprocess

filename = "application.log"
subprocess.run(["cat", "--", filename], check=True, timeout=10)

The -- ends option processing for tools that support it, preventing a leading hyphen from turning a filename into an option. Argument lists avoid shell parsing; they do not decide which executable or file paths a caller is allowed to use. Validate those separately. Windows batch files have additional shell-handling caveats described in the security considerations.

Stream a child's output with Popen

Create worker.py:

import time

for number in range(3):
    print(f"Step {number}", flush=True)
    time.sleep(0.05)

Then run this parent script from the same directory:

import subprocess
import sys

with subprocess.Popen(
    [sys.executable, "-u", "worker.py"],
    stdout=subprocess.PIPE,
    stderr=subprocess.STDOUT,
    text=True,
) as process:
    for line in process.stdout:
        print("Child:", line.rstrip("\n"))
    returncode = process.wait()
print("Exit status:", returncode)

Using sys.executable selects the current interpreter. The child flushes its output, and -u requests unbuffered Python streams. Merging stderr into stdout prevents an unread second pipe from filling, at the cost of losing a separate stderr stream.

Line iteration waits for a newline or EOF. This example has no deadline; production supervision needs an explicit timeout strategy. Do not assume a timeout passed to wait() after the read loop would bound time spent inside that loop.

For bounded output, use communicate() to drain both pipes. This is an alternative to the streaming loop, not something to run after partially consuming those streams:

import subprocess
import sys

process = subprocess.Popen(
    [sys.executable, "-c", "import time; time.sleep(10)"],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
    text=True,
)
try:
    stdout, stderr = process.communicate(timeout=0.05)
except subprocess.TimeoutExpired:
    process.kill()
    stdout, stderr = process.communicate()
print("Exit status:", process.returncode)

Unlike run(), communicate(timeout=...) does not kill the child automatically. The second call drains remaining output and waits after the kill. Capturing output buffers it in memory; use files or continuously drained streams for large output. Descendants can also keep inherited pipes open, so a process tree may require platform-specific group or job management.

Set the environment and working directory

import os
import subprocess

env = {**os.environ, "APP_ENV": "production"}
subprocess.run(
    ["make", "deploy"],
    cwd="/srv/myapp",
    env=env,
    check=True,
    timeout=300,
)

This deployment fragment requires your own reviewed Makefile and directory; do not run it as a generic test. Passing env replaces the inherited environment, so copying os.environ retains values such as PATH before overriding selected keys. cwd changes only the child's working directory.

Common questions

Why prefer subprocess to os.system()?

os.system() delegates a command string to a shell and offers less direct stream and error control. On Unix its result is an encoded wait status, whereas subprocess exposes the child return code directly.

Why does a child appear to hang?

An unread pipe can fill and block the child. Drain it with communicate() for bounded output, stream it correctly, or redirect to a file. Also check child-side buffering, missing newlines, input the child is waiting for, and inherited pipes held by descendants.

How can I run several commands concurrently?

You can start multiple Popen instances, provided you manage and drain all their streams rather than waiting on one while others fill their pipes. For async applications, asyncio subprocess APIs provide create_subprocess_exec(); this API predates Python 3.12.

Record failures without leaking command inputs

For scheduled or service-owned commands, record an operation name, duration, exit status, and bounded diagnostic output. Redact command arguments and stderr when they may contain credentials. Guance's log collection guide provides the collection path; keep an entire failure associated with its execution identifier.

Local examples were exercised in a temporary directory. No deployment, external Git push, Windows process-tree test, or Guance ingestion was performed. Related reading: Python scheduling (Chinese) and Python timeouts.