The litellm Supply Chain Attack: How I Audited My Dependencies and What You Should Do Now

The Moment Your System Gets Compromised Without You Knowing

I was having a regular Wednesday afternoon when a colleague sent me a link to a thread about litellm. My stomach dropped as I read the details. This wasn't a zero-day vulnerability that required specific code paths to trigger. This wasn't a bug you could patch. This was a deliberate supply chain attack that executed the moment you ran pip install—before you even imported the library.

Two poisoned versions of litellm (1.82.7 and 1.82.8) were in the wild, exfiltrating SSH keys, AWS credentials, Kubernetes secrets, environment variables, and anything else it could find. The attackers had done something I'd never seriously prepared for: they compromised the tooling used to secure the tooling.

What shocked me most wasn't that the attack happened. What shocked me was that I had no idea how many of my projects were affected. I had three packages pulling litellm in transitively, buried deep in my dependency tree like a ticking time bomb.

Understanding the Attack Surface

Before diving into solutions, let's understand what happened and why it's so terrifying for the Python ecosystem.

The Attack Chain

The attackers didn't actually compromise litellm's GitHub repository. Instead, they compromised Trivy—a security scanning tool used to find vulnerabilities. From there, they stole the PyPI publish token for litellm and uploaded malicious versions directly to PyPI.

This is the scariest part: a tool designed to protect you became the vector of attack. It's like a security guard handing over the keys to the building they're supposed to protect.

Why Installation-Time Execution is Catastrophic

Most Python supply chain attacks require the malicious code to be imported and executed. This one didn't care about that requirement. Here's what happens when you install a Python package:

1. PyPI serves the package
2. pip downloads and extracts it
3. setup.py or pyproject.toml scripts run
4. The package is installed
5. Only then can you import it

Malicious actors can inject code in step 3—the setup scripts—and it executes with whatever privileges your pip process has. On a development machine, that's probably your user account. On a server, that might be root or a service account with broad permissions.

The attackers' code was looking for every credential it could find and exfiltrating them. They had the keys to the kingdom before you even typed import litellm.

The Irony of the Fork Bomb

Here's what actually saved us: the malicious code had a bug. It spawned processes in an infinite loop and crashed the infected systems before exfiltration completed. If it had worked cleanly, this could have gone undetected for weeks.

That's a sobering thought.

My Audit Process: From Panic to Action

After seeing the alert, I did what you should do: immediately audit every project. Here's my step-by-step process and the tools I used.

Step 1: Finding All Affected Packages

First, I needed to understand the dependency tree. Python's pip command gives you the information, but you need to query it properly.

# List all installed packages with their versions
pip list

# Show dependency tree (install pipdeptree if you don't have it)
pip install pipdeptree
pipdeptree

# Find all packages that depend on litellm
pipdeptree | grep -i litellm
pipdeptree | grep -A 5 litellm

# Check specific project for vulnerabilities
pip check

# Generate a comprehensive dependency report
pip freeze > requirements.txt
pipdeptree --graph-output png > dependency_tree.png

I ran this across every project and environment I managed. It was enlightening (and terrifying) to see how deep litellm's hooks went into my stack.

Step 2: Checking Installed Versions

The vulnerable versions were 1.82.7 and 1.82.8. I needed to check which machines actually had these versions installed.

# Check litellm version on current system
pip show litellm

# Get version across all environments
for env in $(find . -name "venv" -o -name ".venv"); do
    echo "Checking $env:"
    $env/bin/pip show litellm
done

# Check in a requirements file what version is pinned
grep -r "litellm" requirements*.txt setup.py pyproject.toml

# Check Docker images you've built
docker run --rm your-image:latest pip show litellm

I found version 1.82.7 on our staging server. My heart rate spiked.

Step 3: Creating an Automated Audit Script

Rather than doing this manually across all projects, I wrote a script to automate the process:

#!/usr/bin/env python3
"""
Audit script to find litellm and other vulnerable dependencies
across multiple Python environments.
"""

import subprocess
import json
import sys
from pathlib import Path
from packaging import version

VULNERABLE_PACKAGES = {
    "litellm": ["1.82.7", "1.82.8"],
    # Add other known vulnerable packages here
}

def get_installed_packages(python_executable="python"):
    """Get all installed packages and versions."""
    try:
        output = subprocess.run(
            [python_executable, "-m", "pip", "show", "--all", "--format", "json"],
            capture_output=True,
            text=True,
            check=True
        )
        # pip show doesn't support --all with json in all versions
        # Fall back to parsing pip list
        output = subprocess.run(
            [python_executable, "-m", "pip", "list", "--format", "json"],
            capture_output=True,
            text=True,
            check=True
        )
        return json.loads(output.stdout)
    except Exception as e:
        print(f"Error getting packages from {python_executable}: {e}")
        return []

def check_vulnerabilities(packages):
    """Check if any installed packages are vulnerable."""
    vulnerabilities = []

    for package in packages:
        name = package["name"].lower()
        installed_version = package["version"]

        for vuln_name, vuln_versions in VULNERABLE_PACKAGES.items():
            if name == vuln_name.lower():
                if installed_version in vuln_versions:
                    vulnerabilities.append({
                        "package": name,
                        "installed_version": installed_version,
                        "vulnerable_versions": vuln_versions,
                        "severity": "CRITICAL"
                    })

    return vulnerabilities

def audit_environment(env_path=None):
    """Audit a specific Python environment."""
    python_exe = "python"

    if env_path:
        python_exe = str(Path(env_path) / "bin" / "python")
        if not Path(python_exe).exists():
            python_exe = str(Path(env_path) / "Scripts" / "python.exe")

    print(f"\n🔍 Auditing environment: {python_exe}")

    packages = get_installed_packages(python_exe)
    vulnerabilities = check_vulnerabilities(packages)

    if vulnerabilities:
        print(f"⚠️  CRITICAL: Found {len(vulnerabilities)} vulnerable package(s)!")
        for vuln in vulnerabilities:
            print(f"   - {vuln['package']} {vuln['installed_version']} (vulnerable versions: {vuln['vulnerable_versions']})")
        return True
    else:
        print("✅ No known vulnerabilities found in this environment")
        return False

def main():
    """Main audit function."""
    print("🛡️  Python Dependency Security Audit")
    print("=" * 50)

    # Audit current environment
    has_vulns = audit_environment()

    # Audit virtual environments in common locations
    venv_locations = [
        Path(".venv"),
        Path("venv"),
        Path(".env"),
    ]

    for location in venv_locations:
        if location.exists() and location.is_dir():
            has_vulns = audit_environment(str(location)) or has_vulns

    print("\n" + "=" * 50)
    if has_vulns:
        print("❌ VULNERABLE PACKAGES DETECTED!")
        print("\nIMMEDIATE ACTIONS REQUIRED:")
        print("1. Rotate all credentials (AWS, API keys, secrets)")
        print("2. Update packages: pip install --upgrade litellm")
        print("3. Audit logs for unauthorized access")
        print("4. Review what credentials were on this system")
        sys.exit(1)
    else:
        print("✅ All audited environments appear safe")
        sys.exit(0)

if __name__ == "__main__":
    main()

This script saved me hours. I could run it across my entire infrastructure and immediately see which systems were at risk.

The Response: Containing the Damage

Once I found the vulnerable version, panic turned to action.

Immediate Steps (Within 1 Hour)

1. Rotated every credential that was on the staging server

   • AWS access keys
   • Database passwords
   • API tokens for OpenAI, Anthropic, and other services
   • GitHub tokens
   • Docker registry credentials

2. Updated the package immediately
   

   pip install --upgrade litellm
   # or specify a safe version
   pip install litellm==1.83.0  # or whatever the latest patched version is

1. Checked for suspicious activity

   # Check AWS CloudTrail for unauthorized API calls
   # Check application logs for strange behavior
   # Check system logs for suspicious process execution
   # Review SSH logs for unauthorized access

Medium-Term (That Day and Next Day)

1. Updated all requirements files to exclude vulnerable versions:

   litellm>=1.82.9
   # Or be more specific
   litellm==1.83.5

1. Added dependency pinning for critical packages in pyproject.toml:

   [project]
   dependencies = [
       "litellm>=1.83.0,!=1.82.7,!=1.82.8",
       "requests>=2.31.0",
   ]

1. Rebuilt staging and production from fresh base images

Long-Term Prevention

This taught me that I needed better processes:

1. Implement a Software Bill of Materials (SBOM)

   • Track exactly what's installed in production
   • Use tools like cyclonedx to generate SBOM

2. Set up vulnerability scanning
   

   # Install safety to check for known vulnerabilities
   pip install safety
   safety check --json > vulnerabilities.json

1. Pin versions more aggressively for critical dependencies
2. Use hash verification for packages
3. Run pip in a restricted environment (containerized, limited permissions)

Common Pitfalls and Edge Cases

Pitfall 1: Hidden Transitive Dependencies

You might not have litellm directly in your requirements. Check indirect dependencies:

# Find what's pulling in litellm
pip show litellm | grep -i "required-by"

# For a more detailed view
pipdeptree | grep litellm

I found three packages I'd added months ago that indirectly pulled in litellm. One was a langchain integration I'd completely forgotten about.

Pitfall 2: Multiple Python Environments

Don't just check your current Python. Check:

• Virtual environments
• Docker images
• CI/CD runners
• Production servers
• Development machines

Each could have different versions installed.

Pitfall 3: Cached pip Wheels

If you're using pip cache or local wheels, the old versions might still be there:

pip cache purge
# or be selective
pip cache remove litellm

Pitfall 4: Requirements in Multiple Files

Check all possible locations:

• requirements.txt
• requirements-dev.txt
• requirements-prod.txt
• setup.py
• setup.cfg
• pyproject.toml
• Pipfile
• poetry.lock

Summary and Next Steps