Master Python System Programming: From Subprocess Commands to Advanced Process Control Techniques

#programming #devto #python #softwareengineering

As a best-selling author, I invite you to explore my books on Amazon. Don't forget to follow me on Medium and show your support. Thank you! Your support means the world!

Think of your computer's operating system as a busy office building. The receptionist handles calls, messengers run between departments, files get stored in cabinets, and cleaning staff work overnight. Python gives you the master keys and blueprints to this building. You can talk directly to the receptionist, send your own messengers, reorganize the filing system, and even listen to the building's internal announcements. This is system programming.

I write scripts to automate my work, and often, that means making Python talk to the system it's running on. Let me show you how I do it, moving from simple tasks to more direct control.

First, we often need to run other programs. Maybe you want to list files, convert an image, or start a database. You could do this by clicking icons, but Python can do it for you. The subprocess module is your tool here. It's like having a robotic assistant you can instruct from your code.

import subprocess

# The simplest way: run a command and wait for it to finish.
# Let's get a directory listing, like 'ls -la' on Linux or 'dir' on Windows.
result = subprocess.run(['ls', '-la'], capture_output=True, text=True)
print(f"The command finished with code: {result.returncode}")
if result.stdout:
    print("Here's what it found:")
    print(result.stdout)

The capture_output=True tells Python to grab whatever the command prints. text=True gives us a nice string instead of raw bytes. The returncode is crucial: 0 usually means success, anything else suggests an error.

Sometimes, you need to chain commands, like taking the output of one program and feeding it to another. Imagine you want to find all running Python processes. On a command line, you might type ps aux | grep python. Here's how you build that pipe in Python.

import subprocess

# Start the 'ps aux' command.
ps_process = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE)
# Start 'grep python', telling it to read from ps_process's output.
grep_process = subprocess.Popen(
    ['grep', 'python'],
    stdin=ps_process.stdout,  # This is the magic link.
    stdout=subprocess.PIPE,
    text=True
)
# Important: close the first process's output in our program.
ps_process.stdout.close()
# Now, get the final result from the grep command.
output, errors = grep_process.communicate()
print(f"Found these Python processes:\\n{output}")

What if a command gets stuck? You don't want your script to hang forever. You can set a timeout.

try:
    # This will try to sleep for 10 seconds...
    subprocess.run(['sleep', '10'], timeout=5)  # ...but we only allow 5.
except subprocess.TimeoutExpired:
    print("Good, we stopped it. It was taking too long.")

Running commands is useful, but real system work often means dealing with files and directories directly, not just through other programs. The os module is your direct line to the operating system's file manager.

Let's say you're writing a script to clean up a project folder. You need to check file sizes, see what's old, and understand permissions.

import os
import stat
import time

file_name = 'my_data.log'

# Get everything the OS knows about this file.
info = os.stat(file_name)
print(f"Size: {info.st_size} bytes")
# Convert the modification timestamp to something readable.
mod_time = time.ctime(info.st_mtime)
print(f"Last modified: {mod_time}")
# Permissions are stored as numbers. 'oct' shows them in octal, like 0o644.
print(f"Permissions: {oct(info.st_mode)[-3:]}")

# Maybe we want to make this file read-only for the owner.
os.chmod(file_name, stat.S_IRUSR)  # S_IRUSR means "user can read."

# Now, let's navigate directories.
# Create a whole chain of folders safely.
os.makedirs('project/data/2024/logs', exist_ok=True)  # 'exist_ok' prevents errors if it already exists.

# Walk through a directory tree. This is incredibly powerful for organizing files.
for current_dir, subdirs, files in os.walk('project'):
    print(f"\\nIn directory: {current_dir}")
    for file in files:
        full_path = os.path.join(current_dir, file)
        size = os.path.getsize(full_path)
        print(f"  - {file} ({size} bytes)")

When you work with very large files—think multi-gigabyte logs or data dumps—loading them entirely into memory can crash your program. Memory mapping is a clever trick. Instead of reading the file, you ask the operating system to make the file look like a big string in your computer's memory. You interact with the memory, and the OS handles the reading and writing from the disk in the background. The mmap module does this.

import mmap
import contextlib

def search_in_huge_file(filename, search_term):
    with open(filename, 'r+b') as f:  # Open for reading and writing in binary.
        # Map the file to memory. Length 0 means "map the whole thing."
        with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as m:
            # Now 'm' behaves like a giant byte string.
            # Searching is fast because it happens in memory.
            location = m.find(search_term.encode())  # Encode string to bytes.
            if location != -1:
                # Extract 100 characters around the found term.
                start = max(location - 50, 0)
                end = min(location + 50, len(m))
                context = m[start:end].decode('utf-8', errors='ignore')
                print(f"Found '{search_term}' near position {location}:")
                print(f"...{context}...")
            else:
                print("Term not found.")

# Imagine 'big_data.bin' is 20GB. This still works efficiently.
search_in_huge_file('big_data.bin', 'ERROR:')

Sometimes, one program isn't enough. You need multiple Python scripts working together. Perhaps one handles a web request while another crunches numbers. They need to talk. This is Inter-Process Communication (IPC). The multiprocessing module provides several ways, but I often start with a Queue. It's like a conveyor belt between processes.

import multiprocessing as mp
import time

def worker(my_queue, result_queue):
    """A separate process that does calculations."""
    print(f"Worker {mp.current_process().name} started.")
    while True:
        task = my_queue.get()
        if task is None:  # Our signal to stop.
            print(f"Worker {mp.current_process().name} stopping.")
            my_queue.task_done()
            break
        # Simulate some work.
        time.sleep(1)
        result = f"Processed: {task}"
        result_queue.put(result)
        my_queue.task_done()

# Create the communication channels.
task_queue = mp.JoinableQueue()
result_queue = mp.Queue()

# Start two worker processes.
processes = []
for i in range(2):
    p = mp.Process(target=worker, args=(task_queue, result_queue))
    p.start()
    processes.append(p)

# Send them tasks.
for item in ['task_a', 'task_b', 'task_c', 'task_d']:
    task_queue.put(item)

# Tell the workers to shut down by sending two 'None' signals.
for i in range(2):
    task_queue.put(None)

# Wait for all tasks to be marked as done.
task_queue.join()

# Collect results.
print("\\nResults:")
while not result_queue.empty():
    print(result_queue.get())

# Clean up the processes.
for p in processes:
    p.join()

When you run this, you'll see the two workers taking tasks from the shared queue, processing them in parallel, and putting results in another queue. This pattern is the backbone of many high-performance systems.

Now, what if your program is running and someone tries to stop it with Ctrl+C? Or the system wants to shut it down? These events are sent as signals. You can teach your Python program to listen for them and clean up nicely, like saving a document before quitting. The signal module handles this.

import signal
import sys
import time

# A flag to control our main loop.
keep_running = True

def handle_interrupt(signum, frame):
    """This function runs when SIGINT (Ctrl+C) is received."""
    global keep_running
    print("\\nOkay, okay, I got the interrupt signal. Shutting down gracefully...")
    keep_running = False
    # We could save state here.

def handle_terminate(signum, frame):
    """This runs on SIGTERM (a polite 'please stop' from the OS)."""
    print("\\nReceived terminate request from system.")
    sys.exit(0)  # Exit cleanly now.

# Connect our handler functions to the signals.
signal.signal(signal.SIGINT, handle_interrupt)
signal.signal(signal.SIGTERM, handle_terminate)

print("Program running. Press Ctrl+C to test. Or, in another terminal, run:")
print(f"  kill -TERM {os.getpid()}")

# Main work loop.
while keep_running:
    print(".", end="", flush=True)
    time.sleep(1)

print("\\nCleanup done. Exiting.")

This turns a jarring, immediate stop into a controlled shutdown. It's essential for scripts that manage data or connections.

Let's get a bit more low-level. Have you ever wondered how command-line tools like vim or htop read single keystrokes without you pressing Enter? Or how they hide your password input? They manipulate the terminal itself. We can do that with tty and termios. This is more specialized, but it's powerful for building interactive tools.

import tty
import termios
import sys

def get_character():
    """Reads one character instantly, without waiting for Enter."""
    # Save the current terminal settings.
    file_descriptor = sys.stdin.fileno()
    original_settings = termios.tcgetattr(file_descriptor)
    try:
        # Switch the terminal to 'raw' mode.
        tty.setraw(sys.stdin.fileno())
        # Read just one character.
        char = sys.stdin.read(1)
    finally:
        # No matter what, restore the original settings.
        termios.tcsetattr(file_descriptor, termios.TCSADRAIN, original_settings)
    return char

print("Press any key (you won't need Enter):")
user_key = get_character()
print(f"\\nYou pressed: '{user_key}'")

To watch your own program's resource use, or to monitor the whole system, psutil is an indispensable library. It's not built-in, so you'd install it with pip install psutil. I use it all the time to add simple monitoring to long-running scripts.

import psutil
import time

print("--- System Overview ---")
print(f"CPU Load: {psutil.cpu_percent(interval=1)}%")
memory = psutil.virtual_memory()
print(f"Memory Used: {memory.percent}% ({memory.used / (1024**3):.1f} GB)")

print("\\n--- My Process Details ---")
me = psutil.Process()
print(f"My PID: {me.pid}")
print(f"My Memory: {me.memory_info().rss / (1024**2):.1f} MB")
print(f"My Open Files: {len(me.open_files())}")

print("\\n--- Top 3 CPU-Using Processes ---")
for proc in psutil.process_iter(['pid', 'name', 'cpu_percent'])[:3]:
    try:
        info = proc.info
        print(f"  PID {info['pid']}: {info['name']} - {info['cpu_percent']}%")
    except (psutil.NoSuchProcess, psutil.AccessDenied):
        pass

Finally, you must remember that systems differ. A script for Linux might break on Windows. The sys and platform modules help you write code that knows where it is.

import sys
import platform

print(f"Platform ID: {sys.platform}")
print(f"OS Name: {platform.system()}")
print(f"OS Version: {platform.release()}")
print(f"Processor: {platform.machine()}")

# Writing conditional code for different systems.
if sys.platform == 'win32':
    print("Running on Windows. Maybe use 'dir' instead of 'ls'.")
    # Windows-specific code can go here.
    # import ctypes
    # ... do Windows things ...
elif sys.platform == 'darwin':
    print("Running on macOS.")
elif sys.platform.startswith('linux'):
    print("Running on Linux.")
    # Linux-specific code here.
    # import fcntl
    # ... do Linux things ...
else:
    print("Running on an unknown system. Proceed with caution.")

Putting it all together, system programming in Python is about reaching out from your script's comfortable world and touching the machinery underneath. You start with running simple commands using subprocess. You move to inspecting and organizing files with os. For heavy data, you use mmap. To build multi-tasking applications, you connect processes with multiprocessing. You make your programs robust with signal handlers. For advanced interfaces, you control the terminal with tty. To keep an eye on performance, you use psutil. And you always, always check your platform.

Each step gives you more control. You stop being just a visitor in the office building. You become the manager, with the keys to every door and the schedule for every task. It lets you build tools that are efficient, resilient, and deeply integrated with the system they run on. That's the real power of Python for system programming—it turns high-level ideas into low-level action.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!