How do I execute a program or call a system command?

Executing External Programs and System Commands in Python

Learn how to reliably execute shell commands and external programs from your Python scripts using the subprocess module, covering basic calls, capturing output, and handling errors.

Python offers robust ways to interact with the underlying operating system, including executing external programs and system commands. This capability is crucial for tasks like automating system administration, integrating with command-line tools, or running scripts written in other languages. While older methods like os.system and os.popen exist, the subprocess module is the recommended and most powerful approach for modern Python development.

The subprocess Module: Your Go-To for External Commands

The subprocess module allows you to spawn new processes, connect to their input/output/error pipes, and obtain their return codes. It replaces older, less flexible functions and provides a unified interface for process management. The primary function you'll use is subprocess.run().

flowchart TD
    A[Python Script] --> B{"subprocess.run()"}
    B --> C[External Command/Program]
    C --> D[Standard Output (stdout)]
    C --> E[Standard Error (stderr)]
    C --> F[Return Code]
    D --> B
    E --> B
    F --> B
    B --> G[Python Script (Result)]

Flow of executing an external command using subprocess.run()

Basic Command Execution

The simplest way to execute a command is by passing it as a list of arguments to subprocess.run(). This is generally safer than passing a single string with shell=True because it avoids shell injection vulnerabilities. For commands without arguments, a single string can be used, but the list format is preferred for consistency.

import subprocess

# Example 1: Simple command without arguments
result = subprocess.run(['ls', '-l'], capture_output=True, text=True)
print("Stdout:", result.stdout)
print("Stderr:", result.stderr)
print("Return Code:", result.returncode)

# Example 2: Command with arguments
result = subprocess.run(['echo', 'Hello, World!'], capture_output=True, text=True)
print("Stdout:", result.stdout)

# Example 3: Running a command that might fail
result = subprocess.run(['non_existent_command'], capture_output=True, text=True)
print("Return Code for failed command:", result.returncode)
print("Stderr for failed command:", result.stderr)

Basic execution of shell commands using subprocess.run()

Capturing Output and Handling Errors

To capture the standard output (stdout) and standard error (stderr) of the executed command, you need to set capture_output=True. By default, subprocess.run() will print stdout and stderr directly to your console. Setting text=True (or encoding='utf-8') ensures that the captured output is decoded as text rather than bytes.

import subprocess

# Command that produces output
result = subprocess.run(
    ['grep', 'root', '/etc/passwd'],
    capture_output=True,
    text=True,
    check=False # Do not raise an exception for non-zero exit codes
)

print("Captured Stdout:\n", result.stdout)
print("Captured Stderr:\n", result.stderr)
print("Exit Code:", result.returncode)

# Command that fails and we want to check for it
try:
    subprocess.run(
        ['false_command'],
        capture_output=True,
        text=True,
        check=True # Raise CalledProcessError for non-zero exit codes
    )
except subprocess.CalledProcessError as e:
    print(f"Command failed with error: {e}")
    print(f"Stderr: {e.stderr}")

Capturing output and handling non-zero exit codes with check=True

Advanced Scenarios: Input, Shell, and Timeouts

The subprocess module offers more advanced features for complex interactions:

• Input (input parameter): You can pass data to the standard input (stdin) of the child process.
• Shell (shell=True): If you need shell features like wildcards, environment variables, or command chaining (e.g., command1 | command2), you can set shell=True. However, be extremely cautious as this can introduce security vulnerabilities if the command string comes from untrusted input.
• Timeout (timeout parameter): Prevent commands from running indefinitely by setting a timeout. If the command exceeds the timeout, a TimeoutExpired exception is raised.

import subprocess
import time

# Example 1: Providing input to a command
# This simulates 'cat' reading from stdin
result = subprocess.run(
    ['cat'],
    input='This is input for cat.',
    capture_output=True,
    text=True
)
print("Cat output with input:", result.stdout)

# Example 2: Using shell=True (use with caution!)
# This command uses shell redirection
result = subprocess.run(
    'echo "Hello from shell" > output.txt', 
    shell=True, 
    capture_output=True, 
    text=True
)
print("Shell command executed. Check output.txt")

# Example 3: Command with a timeout
try:
    # 'sleep 5' will run for 5 seconds
    subprocess.run(['sleep', '5'], timeout=2, capture_output=True, text=True)
except subprocess.TimeoutExpired:
    print("Command timed out after 2 seconds!")

# Clean up the created file
subprocess.run(['rm', 'output.txt'], capture_output=True)

Advanced subprocess.run() usage with input, shell, and timeout