Akhilesh Mishra

Posted on Jun 26 • Originally published at livingdevops.com

How to Auto-Generate requirements.txt for Python Projects: Complete Guide to pipreqs and pigar

#programming #python

Stop manually hunting for Python dependencies - automate requirements.txt generation with these powerful tools

As a DevOps engineer, I frequently collaborate with multiple teams, diving into various projects and repositories. One common challenge I face is encountering Python code without a requirements.txt file.

Every time I run such code, I need to figure out and manually install the necessary dependencies. I found it a waste of time and effort. I am sure you guys also feel the same way.

So I dug around, did some Google searches, experimented with a few tools, and found many good ones. My favorites are pipreqs and pigar.

In this comprehensive guide, I'll show you how to use these Python dependency management tools to generate requirements.txt files automatically for any Python project.

Why Not Use pip freeze?
Method 1: Using pipreqs
Method 2: Using pigar
Comparison and Best Practices

Why Not Use pip freeze?

Before diving into the solutions, let's address why pip freeze isn't always the best choice for generating requirements.txt:

❌ pip freeze problems:

Only saves packages installed with pip install in your current environment
Includes ALL packages in the environment, even those not used in your project (if you don't use virtualenv)
Sometimes you need to create requirements.txt for a new project without installing modules first
Can include system-wide packages that aren't relevant to your project

✅ What we need instead:

Tools that analyze your actual Python code
Generate requirements based on import statements
Create clean, project-specific dependency lists

Method 1: Using pipreqs - The Import-Based Analyzer

What is pipreqs?

pipreqs is a Python tool that analyzes your Python files and generates requirements.txt based on the actual imports in your code. It's smart, fast, and doesn't require installing packages beforehand.

Setting Up Your Environment

I recommend working with isolated Python virtual environments for better dependency management:

# Create a project directory
mkdir python-requirements-demo
cd python-requirements-demo

# Create and activate virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Installing pipreqs

pip install pipreqs

Practical Example: Word Cloud Generator

Let's create a real Python project to demonstrate pipreqs. Create a directory and file:

mkdir project
cd project

Create word_cloud_generator.py with the following code:

import matplotlib.pyplot as plt
from wordcloud import WordCloud
from collections import Counter
import requests
from PIL import Image
import numpy as np
import argparse
from io import BytesIO

def get_mask_image(url):
    """Download and process mask image from URL"""
    response = requests.get(url)
    img = Image.open(BytesIO(response.content))
    return np.array(img)

def create_word_cloud(text, mask_url=None):
    """Generate word cloud from text with optional mask"""
    if mask_url:
        mask = get_mask_image(mask_url)
    else:
        mask = None

    wordcloud = WordCloud(
        width=800, 
        height=400, 
        background_color='white', 
        mask=mask, 
        contour_width=1, 
        contour_color='black'
    ).generate(text)

    plt.figure(figsize=(10, 5))
    plt.imshow(wordcloud, interpolation='bilinear')
    plt.axis('off')
    plt.show()

def parse_arguments():
    """Parse command line arguments"""
    parser = argparse.ArgumentParser(description="Generate a word cloud from input text")
    parser.add_argument('--text', '-t', required=True, help="Input text for word cloud")
    parser.add_argument('--mask_url', '-m', help="URL of mask image for word cloud shape")
    return parser.parse_args()

if __name__ == "__main__":
    args = parse_arguments()
    create_word_cloud(args.text, args.mask_url)

Generating requirements.txt with pipreqs

Now comes the magic! Instead of manually figuring out dependencies, run:

# Generate requirements.txt for the project directory
pipreqs project

Output in requirements.txt:

matplotlib==3.9.1
numpy==2.0.0
Pillow==10.4.0
Requests==2.32.3
wordcloud==1.9.3

Installing and Testing Dependencies

# Install all dependencies at once
pip install -r project/requirements.txt

# Test the script
python project/word_cloud_generator.py --text "Python dependency management made easy"

# With mask image
python project/word_cloud_generator.py --text "pipreqs is awesome" --mask_url "https://example.com/mask.jpg"

Advanced pipreqs Options

# Generate requirements for specific directory
pipreqs /path/to/your/project

# Specify output file name
pipreqs . --savepath requirements-dev.txt

# Force overwrite existing requirements.txt
pipreqs . --force

# Use local database for package names
pipreqs . --use-local

# Debug mode for troubleshooting
pipreqs . --debug

Method 2: Using pigar - The Advanced Dependency Scanner

What is pigar?

pigar is another excellent tool for generating Python requirements. It offers more advanced features and better handling of complex projects.

Installing pigar

# Using pip
pip install pigar

# From source (latest features)
pip install git+https://github.com/damnever/pigar.git@main --upgrade

# Using conda
conda install -c conda-forge pigar

Real-World Example: Flask Application

Let's test pigar with a Flask application:

# Clone a sample Flask project
git clone https://github.com/akhileshmishrabiz/Devops-zero-to-hero
cd Devops-zero-to-hero/project5

# Remove existing requirements.txt for demo
rm requirements.txt

Generating requirements with pigar

# Generate requirements.txt for current directory
pigar generate

# Check the generated file
cat requirements.txt

Output:

# Automatically generated by https://github.com/damnever/pigar.

Flask==3.0.3
Flask-SQLAlchemy==3.1.1

Advanced pigar Usage

# Generate requirements for specific directory with custom filename
pigar gen -f custom-requirements.txt /path/to/project

# Example with Python Lambda project
pigar gen -f python-lambda-requirements.txt python-for-devops/python-lambda-runtime/

Output for Lambda project:

# Automatically generated by https://github.com/damnever/pigar.

boto3==1.34.142
colorlog==6.8.2
packaging==24.1

pigar Advanced Features

# Check for outdated packages
pigar check

# Search for packages
pigar search numpy

# Show package information
pigar show requests

Comparison: pipreqs vs pigar

Feature	pipreqs	pigar
Speed	⚡ Very Fast	🔄 Moderate
Accuracy	✅ High	✅ Very High
Local imports	❌ Limited	✅ Excellent
Complex projects	✅ Good	✅ Excellent
Additional features	❌ Basic	✅ Package search, outdated checks
File size	📦 Lightweight	📦 Slightly larger

Best Practices for Python Dependency Management

1. Use Virtual Environments

# Always create isolated environments
python -m venv project-env
source project-env/bin/activate

2. Pin Exact Versions

# Good - specific versions
requests==2.32.3
numpy==2.0.0

# Avoid - unpinned versions
requests
numpy

3. Separate Development Dependencies

# Create different requirement files
pigar gen -f requirements.txt .           # Production
pigar gen -f requirements-dev.txt .       # Development

4. Regular Dependency Audits

# Check for security vulnerabilities
pip-audit -r requirements.txt

# Update outdated packages
pip list --outdated

Troubleshooting Common Issues

pipreqs Not Finding All Imports

# Use debug mode
pipreqs . --debug

# Force scan all files
pipreqs . --scan-notebooks

pigar Missing Local Modules

# Include local packages
pigar gen --with-referenced-comments

Version Conflicts

# Generate without version pins
pipreqs . --no-version

# Then manually specify versions

Automation with CI/CD

GitHub Actions Example

name: Check Requirements
on: [push, pull_request]

jobs:
  check-deps:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Generate requirements
      run: |
        pip install pipreqs
        pipreqs . --force
        git diff --exit-code requirements.txt

Conclusion

Both pipreqs and pigar are excellent tools for automating Python dependency management:

Choose pipreqs for: Simple projects, fast generation, lightweight solution
Choose pigar for: Complex projects, advanced features, comprehensive analysis

Stop wasting time manually managing Python dependencies. These tools will save you hours and reduce deployment errors significantly.

Next Steps

Try both tools on your existing Python projects
Integrate dependency generation into your development workflow
Set up automated checks in your CI/CD pipeline
Share this guide with your team to standardize dependency management

DEV Community