DEV Community

Mr Vi
Mr Vi

Posted on

The Python Developer's Guide to Background Process Management

#python #programming #beginners #tutorial

As a Python developer, you've probably faced this scenario: you've built an amazing bot, API, or data processing script, but keeping it running 24/7 is a nightmare. SSH sessions disconnect, scripts crash, servers reboot, and suddenly your carefully crafted automation is offline.

In this comprehensive guide, we'll explore different approaches to running Python applications in the background, from simple development solutions to production-ready process management.

The Problem with "Just Running Python"

When you develop locally, you run scripts like this:

python my_awesome_bot.py
Enter fullscreen mode Exit fullscreen mode

But in production, this approach fails because:

  • SSH disconnections kill your process
  • Script crashes require manual intervention
  • Server reboots lose all running processes
  • Managing multiple bots becomes chaotic
  • No visibility into what's actually running

Let's fix this with proper process management.

Method 1: Screen - The Development Solution

Screen is perfect for development and testing. It creates persistent terminal sessions that survive SSH disconnections.

Getting Started with Screen 

# Install screen
sudo apt install screen  # Ubuntu/Debian
sudo yum install screen  # CentOS/RHEL

# Start a new session
screen

# Activate your virtual environment
source ./venv/bin/activate

# Run your script
python my_bot.py

# Detach (Ctrl+A, then D)
# Your script keeps running!
Enter fullscreen mode Exit fullscreen mode

Managing Screen Sessions 

# List all sessions
screen -ls

# Reattach to a session
screen -r session_name

# Kill a session (from within screen)
# Ctrl+A, then K
Enter fullscreen mode Exit fullscreen mode

When to use Screen:

  • ✅ Development and testing
  • ✅ Quick one-off scripts
  • ✅ When you need terminal access

Limitations:

  • ❌ No automatic restart on crashes
  • ❌ Manual process management
  • ❌ Sessions can be lost on server reboot
  • ❌ No built-in monitoring

Method 2: Systemd - The System Service Approach

Systemd is Linux's service manager, perfect for production environments where you need guaranteed startup and restart capabilities.

Creating a Systemd Service

Create a service file:

sudo nano /etc/systemd/system/mybot.service
Enter fullscreen mode Exit fullscreen mode

Add this configuration:

[Unit]
Description=My Python Bot Service
After=network.target

[Service]
Type=idle
Restart=always
RestartSec=3
User=myuser
WorkingDirectory=/home/myuser/bot
ExecStart=/home/myuser/bot/venv/bin/python /home/myuser/bot/my_bot.py
Environment=PATH=/home/myuser/bot/venv/bin

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Managing the Service 

# Reload systemd configuration
sudo systemctl daemon-reload

# Enable auto-start on boot
sudo systemctl enable mybot.service

# Start the service
sudo systemctl start mybot.service

# Check status
sudo systemctl status mybot.service

# View logs
sudo journalctl -u mybot.service -f

# Stop the service
sudo systemctl stop mybot.service
Enter fullscreen mode Exit fullscreen mode

When to use Systemd:

  • ✅ Production servers
  • ✅ Critical services that must survive reboots
  • ✅ System-level integration
  • ✅ When you need fine-grained control

Limitations:

  • ❌ Requires root access
  • ❌ Complex configuration
  • ❌ System-level management
  • ❌ No built-in process monitoring

Method 3: Modern Process Managers - The Python Way

For Python developers, there's a better way: Pyker - a process manager built specifically for Python applications.

Why Pyker?

Pyker combines the simplicity of screen with the power of systemd, but designed specifically for Python workflows:

  • Python-native: Built for Python developers, by Python developers
  • Virtual environment support: Native venv/conda integration
  • Zero configuration: Works out of the box
  • Beautiful monitoring: Real-time process tables
  • User-space installation: No root access required

Installation 

# One-line installation
curl -fsSL https://raw.githubusercontent.com/mrvi0/pyker/main/install.sh | bash

# Or with Python installer
curl -fsSL https://raw.githubusercontent.com/mrvi0/pyker/main/install.py | python3

# Manual installation
git clone https://github.com/mrvi0/pyker.git
cd pyker
python3 install.py
Enter fullscreen mode Exit fullscreen mode

Basic Usage 

# Start a simple script
pyker start mybot my_bot.py

# Start with virtual environment
pyker start webapp app.py --venv ./venv

# List all processes
pyker list

# View logs in real-time
pyker logs mybot -f

# Get detailed process information
pyker info mybot

# Restart a process
pyker restart mybot

# Stop a process
pyker stop mybot
Enter fullscreen mode Exit fullscreen mode

Advanced Features

Auto-restart on failure:

pyker start critical-service app.py --auto-restart
Enter fullscreen mode Exit fullscreen mode

Virtual environment support:

# Works with any Python environment
pyker start ml-worker train.py --venv /path/to/conda/envs/pytorch
pyker start webapp app.py --venv ./venv
pyker start data-processor process.py --venv /home/user/projects/venv
Enter fullscreen mode Exit fullscreen mode

Process monitoring:

pyker list
Enter fullscreen mode Exit fullscreen mode

Shows a beautiful, adaptive table with:

  • Process status (✓ running, ✗ stopped, ⚠ error)
  • Real-time CPU and memory usage
  • Start/stop times
  • Virtual environment information
  • Script paths

Real-time logs:

pyker logs mybot -f        # Follow logs in real-time
pyker logs mybot -n 100    # Show last 100 lines
Enter fullscreen mode Exit fullscreen mode

Detailed process information:

pyker info mybot
Enter fullscreen mode Exit fullscreen mode

Shows comprehensive information including:

  • Virtual environment path and Python executable
  • Resource usage (CPU, memory)
  • Runtime information
  • Log file location
  • Auto-restart status

Tab completion:

pyker <TAB>                # Shows available commands
pyker stop <TAB>           # Completes with running process names
pyker start app script.py --<TAB>  # Shows --venv, --auto-restart
Enter fullscreen mode Exit fullscreen mode

Real-World Examples

Discord Bot Management 

# Traditional approach with screen
screen
source ./venv/bin/activate
python discord_bot.py
# Ctrl+A, D

# With Pyker
pyker start discord-bot discord_bot.py --venv ./venv --auto-restart
pyker logs discord-bot -f
Enter fullscreen mode Exit fullscreen mode

Web API Deployment 

# With systemd (complex setup)
sudo nano /etc/systemd/system/api.service
# ... complex configuration ...
sudo systemctl enable api.service

# With Pyker (simple)
pyker start api app.py --venv ./venv --auto-restart
Enter fullscreen mode Exit fullscreen mode

Data Processing Pipeline 

# Multiple related processes
pyker start scraper scraper.py --venv ./data-env
pyker start processor process_data.py --venv ./data-env  
pyker start uploader upload_to_s3.py --venv ./data-env

# Monitor all processes
pyker list

# Check specific process
pyker info scraper
Enter fullscreen mode Exit fullscreen mode

Machine Learning Workflows 

# Training with proper environment
pyker start training train_model.py --venv /opt/conda/envs/pytorch --auto-restart

# Inference service
pyker start inference inference_server.py --venv ./ml-env

# Data preprocessing
pyker start preprocess preprocess_data.py --venv ./data-env
Enter fullscreen mode Exit fullscreen mode

Comparison: Choosing the Right Method

Method Complexity Auto-restart Virtual Env Monitoring Best For
Screen Low ❌ Manual Basic Development, testing
Systemd High ✅ Manual System logs Production servers
Pyker Low ✅ Native Built-in Python applications

Best Practices for Production

1. Always Use Virtual Environments 

# Create and activate environment
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

# Run with Pyker
pyker start mybot my_bot.py --venv ./venv
Enter fullscreen mode Exit fullscreen mode

2. Enable Auto-restart for Critical Services 

pyker start critical-service app.py --auto-restart
Enter fullscreen mode Exit fullscreen mode

3. Implement Proper Logging 

import logging
import sys

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    handlers=[
        logging.StreamHandler(sys.stdout),
        logging.FileHandler('app.log')
    ]
)

logger = logging.getLogger(__name__)

def main():
    logger.info("Application started")
    try:
        # Your application logic
        pass
    except Exception as e:
        logger.error(f"Application error: {e}")
        raise

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

4. Handle Graceful Shutdowns 

import signal
import sys
import logging

logger = logging.getLogger(__name__)

def signal_handler(sig, frame):
    logger.info("Received shutdown signal, cleaning up...")
    # Cleanup code here
    sys.exit(0)

# Register signal handlers
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
Enter fullscreen mode Exit fullscreen mode

5. Monitor Your Processes 

# Regular status checks
pyker list

# Monitor logs
pyker logs mybot -f

# Check system resources
htop
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Common Issues

Process Won't Start 

# Test the script manually first
python my_bot.py

# Check Pyker logs
pyker logs mybot

# Verify process information
pyker info mybot
Enter fullscreen mode Exit fullscreen mode

Virtual Environment Issues 

# Verify venv path exists
ls -la ./venv/bin/python

# Test with absolute path
pyker start mybot my_bot.py --venv /full/path/to/venv

# Check Python executable
./venv/bin/python --version
Enter fullscreen mode Exit fullscreen mode

High Resource Usage 

# Monitor resource usage
pyker list  # Shows CPU and memory columns

# Check system resources
htop
top

# Kill problematic processes
pyker stop process_name
Enter fullscreen mode Exit fullscreen mode

Log Management 

# View log files directly
ls -la ~/.pyker/logs/

# Clean old logs
rm ~/.pyker/logs/*.log.*

# Configure log rotation
nano ~/.pyker/config.json
Enter fullscreen mode Exit fullscreen mode

Security Considerations

User Permissions 

# Run as non-root user
pyker start mybot my_bot.py --venv ./venv

# Check process ownership
ps aux | grep mybot
Enter fullscreen mode Exit fullscreen mode

Environment Variables 

# Set environment variables
export API_KEY="your-secret-key"
pyker start mybot my_bot.py --venv ./venv
Enter fullscreen mode Exit fullscreen mode

Network Security 

# Bind to localhost only
pyker start api app.py --venv ./venv
# Configure your app to bind to 127.0.0.1:8000
Enter fullscreen mode Exit fullscreen mode

Conclusion

Running Python applications in production doesn't have to be complicated. The key is choosing the right tool for your needs:

  • Development/Testing: Use Screen for quick testing
  • System Services: Use Systemd for critical system services
  • Python Applications: Use Pyker for modern Python process management

Pyker offers the perfect balance of simplicity and power for Python developers. It's designed specifically for Python workflows, supports virtual environments natively, and provides beautiful monitoring without the complexity of systemd.

The best part? You can start simple and scale up. Begin with Screen for development, then move to Pyker for production, and only use Systemd when you need deep system integration.

Your Python applications deserve better than manual restarts and SSH sessions. Choose the right tool, and let your code run reliably 24/7.

What's your experience with running Python applications in production? Have you tried any of these approaches? Share your thoughts in the comments below!

python #devops #processmanagement #automation #production #linux #systemd #backgroundprocesses

Top comments (0)