DEV Community: Dale Wright

LiteLLM PyPI Supply Chain Compromise: How a Popular LLM Proxy Became a Credential-Stealing Backdoor

Dale Wright — Wed, 25 Mar 2026 13:26:13 +0000

On March 24, 2026, the AI developer community received a stark reminder of how fragile software supply chains have become. Two versions of litellm — a widely used Python library that serves as a unified proxy for over 100 LLM providers (OpenAI, Anthropic, AWS Bedrock, Google Vertex, and many more) — were compromised on PyPI.

Versions 1.82.7 and 1.82.8 contained malicious code that turned the package into an aggressive credential stealer and Kubernetes lateral-movement tool. The attack was short-lived (the malicious releases were available for roughly 2–5 hours), but given litellm’s massive adoption — millions of daily downloads and heavy use in AI agent frameworks, MCP servers, orchestration tools, and production LLM pipelines — the potential impact is enormous.

This wasn’t typo-squatting or a fake package. It was a direct compromise of the legitimate litellm project on PyPI, attributed to the threat actor TeamPCP (the same group behind recent attacks on Trivy, Checkmarx/KICS, and other security tooling).

What Is LiteLLM and Why Does This Matter?

LiteLLM acts as the “universal API gateway” for LLMs. Developers import it once and can call any model provider with the same OpenAI-compatible interface. Because it often centralizes API keys and authentication tokens, it frequently runs with broad access to secrets — making it a high-value target.

When the malicious versions were installed (via pip install litellm or as a transitive dependency), the backdoor activated silently. Version 1.82.8 was especially dangerous: it shipped with a file called litellm_init.pth.

How the Attack Worked: Technical Breakdown

Python automatically executes any .pth file found in site-packages/ at interpreter startup — no import litellm required. The attackers abused this built-in mechanism perfectly.

Execution trigger: litellm_init.pth (approximately 34 KB, heavily obfuscated with double base64 encoding) launched a child Python process containing the real payload.
Credential harvesting: The malware scanned for and exfiltrated:
- SSH keys and configs (~/.ssh/)
- Cloud credentials (AWS, GCP, Azure)
- Kubernetes configs and service-account tokens (~/.kube/)
- LLM API keys (the very keys litellm was proxying)
- .env files, Git credentials, shell histories, database passwords, and crypto wallets (Bitcoin, Ethereum, Solana, etc.)
- Environment variables and cloud metadata endpoints (IMDS)
Exfiltration: Data was bundled into a tar archive, encrypted with AES-256-CBC (session key wrapped in a hardcoded RSA public key), and POSTed to https://models.litellm.cloud/ (an attacker-controlled domain).
Kubernetes worm behavior (if a service-account token was present):
- Read all secrets across all namespaces
- Deploy privileged alpine:latest pods (named node-setup-*) on every node
- Install a persistent backdoor (~/.config/sysmon/sysmon.py) via systemd user service
Version 1.82.7 injected its payload into litellm/proxy/proxy_server.py but delivered the same stealer functionality.

A bug in the malware sometimes caused an exponential fork bomb (the child process re-triggered the .pth file), which ironically helped some researchers notice the issue faster when their systems became unresponsive.

Discovery and Response Timeline

~10:52 UTC — Version 1.82.8 published to PyPI (no corresponding GitHub tag or release).
Shortly after — Version 1.82.7 also confirmed malicious.
~12:30 UTC — FutureSearch researchers (via a transitive dependency in Cursor’s MCP plugin) raised the alarm publicly.
PyPI admins quarantined the project and removed the malicious wheels.
BerriAI (maintainers) confirmed the breach, linking it to stolen PyPI credentials likely obtained through their use of the previously compromised Trivy scanner in CI/CD.

By the afternoon, the malicious versions were gone and quarantine was lifted.

Who Was Affected?

Anyone who installed or upgraded to 1.82.7 or 1.82.8 on March 24, 2026 — including:

Developer laptops
CI/CD runners
Docker containers and production servers
Any project using litellm as a transitive dependency

If you run litellm in Kubernetes or handle LLM API keys, treat the environment as potentially fully compromised.

Immediate Remediation Steps (Do This Now)

Check your installations

   pip show litellm
   # or, if using uv:
   find ~/.cache/uv -name "litellm_init.pth"

Pin to a safe versiontxtlitellm<=1.82.6
Rotate EVERY credential that existed on any affected system:
All LLM API keys (OpenAI, Anthropic, Bedrock, etc.)
Cloud provider keys/tokens
SSH keys
Kubernetes secrets
Database passwords
Anything stored in .env files or shell history

Hunt for persistence:
Check for ~/.config/sysmon/sysmon.py and the corresponding systemd service
In Kubernetes, audit the kube-system namespace for node-setup-* pods and review secret access logs

Purge cachesBashpip cache purge

or

rm -rf ~/.cache/pip ~/.cache/uv

Broader Lessons for the AI Supply Chain
This incident is the latest in TeamPCP’s campaign targeting security tooling and AI infrastructure. It highlights three uncomfortable truths:

CI/CD dependencies are attack surfaces — LiteLLM’s reliance on Trivy gave attackers the publishing credentials they needed.
.pth files are stealthy — They execute before your code even imports the package. Many static analysis tools miss them.
AI tooling moves fast — With millions of downloads and heavy transitive usage, a single compromised release can reach thousands of production environments in minutes.

Supply-chain attacks on AI libraries are no longer theoretical — they target the exact layer where your most sensitive secrets live.
Final Advice
Pin your dependencies aggressively. Review your CI/CD pipelines for external scanners and tools. Assume any popular AI package could be next. And if you installed litellm 1.82.7 or 1.82.8 yesterday — rotate those keys immediately.
The packages have been removed, but any secrets they stole may already be in the attackers’ hands.
Stay vigilant. The AI supply chain just became significantly more dangerous.

Visit pypistats.com for PyPI pacakge analytics and AI insights

This post is based on public disclosures from BerriAI, FutureSearch, Sonatype, Endor Labs, Snyk, ARMO, and other security researchers.

PyPI Stats: Turning Raw Download Numbers into Actionable Insights for Python Package Maintainers

Dale Wright — Thu, 05 Mar 2026 15:26:09 +0000

Hey dev community 👋

If you've ever published a Python package on PyPI, you've probably stared at the basic download stats page wondering:

Is usage actually growing... or just fluctuating noise?
Which versions are people really installing?
Did that blog post / Reddit thread / vulnerability actually cause a spike?
How does my package stack up against competitors?

The official PyPI stats are great raw data (thanks BigQuery!), but they're not exactly maintainer-friendly. No easy trends, no alerts for drops, no quick comparisons, and definitely no AI to explain "hey, your downloads just tanked 40%—maybe check your last release?"

After dealing with this frustration on my own projects, I decided to build something better.

Enter pypistats.com — The Developer's Dashboard for PyPI.

What It Does (Core Features)

Live, Real-Time Package Dashboards Just go to https://pypistats.com/packages/your-package-name and get instant visuals:
- Daily/weekly/monthly download trends
- Breakdowns by Python version, implementation (CPython/PyPy/etc.), OS, country
- Top versions over time Data pulled fresh from PyPI's public BigQuery exports.

Trend Detection & Smart Alerts

Spot anomalies automatically — sudden spikes from viral mentions or drops from deprecation.

Set up email alerts (Pro/Enterprise) so you're notified before a trend becomes a problem.
AI-Powered Insights

Powered by Claude, it generates human-readable summaries like:

"Your package saw a 150% week-over-week increase after that Hacker News post — version 2.1 is dominating adoption."

Helps you understand why numbers are moving without manual analysis.
Embeddable Badges for Your README

Show off your stats with zero effort:

   [![PyPI Downloads](https://pypistats.com/api/badges/your-package)](https://pypistats.com/packages/your-package)

Turns metrics into social proof for contributors and users. Downloads badge in this example.

Weekly Digests & Comparisons Get a curated email recap of your package's week. Compare against similar packages to see if you're gaining/losing ground.

Tech Behind It (Quick Overview)

Backend: Python (FastAPI for the API, processing BigQuery data)
Frontend: Modern React/Next.js for smooth dashboards
Data: Sourced from PyPI's public BigQuery dataset (shoutout to the PSF!)
AI: Claude for natural-language commentary
Free tier covers core stats + badges; paid unlocks AI, alerts, digests

Full disclosure: Yes, I'm the creator (Dale / @pypistats on X). I built this for myself and other maintainers first — it's not trying to be a massive enterprise thing, just a helpful tool in the Python ecosystem.

Why This Matters in 2026

Python's ecosystem is massive (millions of packages, billions of downloads), but most maintainers fly blind on adoption. Better visibility means:

Prioritizing features users actually want
Catching issues early (security, compatibility breaks)
Attracting sponsors/contributors with real proof of impact
Making data-driven decisions instead of gut feels

Try It Out & Give Feedback!

Head over to pypistats.com, plug in your favorite package (or yours!), and see what the trends reveal. This has very recently launched, so I'm still backfilling package data. If your package doesn't show up, a search will trigger a download, so be patient.

What do you think?

Does this solve a real pain point for you?
Missing metrics (e.g., more granular geo, dependency graphs)?
Better ways to visualize or alert?

Drop a comment below — I'd love to hear your thoughts, suggestions, or even roast the UI if it's ugly 😅. If enough people find it useful, I'll keep iterating.

Thanks for reading!

Dale

Creator of pypistats.com

@pypistats on X

json-key-parser vs jsonpath-ng: Simplicity Wins for Messy JSON, Power for Complex Queries

Dale Wright — Wed, 04 Mar 2026 18:16:40 +0000

Some of you may know that json-key-parser (v0.1.0, released Feb 18, 2026) is the tiny, zero-dependency hero that lets you yank specific keys out of deeply nested JSON with almost zero effort.

But the Python ecosystem already has a heavyweight champion: jsonpath-ng (v1.8.0, released Feb 24, 2026). It’s the full-featured JSONPath implementation that’s been battle-tested for years.

So which one should you actually reach for? Let’s compare them head-to-head on the things that matter.

1. Philosophy & API

Aspect	json-key-parser	jsonpath-ng
Core idea	“Just give me these keys, wherever they are”	“Write a precise path query like XPath for JSON”
Syntax	Simple list of strings + `*` wildcard	Full JSONPath expressions (`$..key`, `[*]`, `?(@.price>10)`)
Learning curve	30 seconds	15–30 minutes + docs
Typical line count	1–2 lines	3–10+ lines (parse + find + process)

json-key-parser example (the one-liner magic):

from json_parser import JsonParser

result = JsonParser(data, ["first_name", "street", "address*"]).get_data()

Done. It recursively hunts every level, handles lists, merges duplicate keys into lists automatically.

jsonpath-ng equivalent (you need multiple expressions or clever wildcards):

from jsonpath_ng import parse

first_names = parse('$..first_name').find(data)
streets = parse('$..street').find(data)
addresses = parse('$..address*').find(data)  # Note: * works but not exactly the same

You then have to extract .value from Match objects and handle merging yourself.

Winner for quick-and-dirty extraction: json-key-parser.

2. Power & Features

json-key-parser shines when:

JSON structure is unpredictable or changes often (third-party APIs, logs, scraped data)
You just want a flat dict/list of the fields you care about
Duplicate keys appear at different nesting levels (it merges them intelligently)
You hate writing defensive data.get("a", {}).get("b", []) chains

jsonpath-ng dominates when:

You need filters: $.books[?(@.price > 20)]
You want to update or delete values in place: expr.update(data, new_value)
You need arithmetic, regex, slicing, or parent references
You’re doing metaprogramming (it exposes a clean AST)
You want full paths back: match.full_path

jsonpath-ng also supports extensions for len(), keys(), sorting, etc.

3. Dependencies & Size

Both are zero external dependencies now.

json-key-parser: pure stdlib, < 50 KB installed
jsonpath-ng: robust parser (no longer pulls in ply at runtime in recent versions), still tiny

Both install with one pip command and support Python 3.8+.

4. Performance & Maturity

jsonpath-ng is mature, heavily tested, and used in production everywhere.
json-key-parser is brand new (Beta status) but laser-focused, so its recursion is extremely fast for the 95% use case.

For massive JSON documents with thousands of nested objects, jsonpath-ng’s compiled expressions can edge it out on complex queries. For simple key extraction? json-key-parser is often faster because it doesn’t parse a query language at all.

When to Choose Which

Use json-key-parser if:

You’re writing ETL scripts, API clients, or data-cleaning notebooks
Your JSON is messy and you just want “first_name, email, total” regardless of nesting
You value readability and minimal code

Use jsonpath-ng if:

You need conditional filtering or data transformation
You’re building a library or tool that requires precise, repeatable queries
You want to modify the JSON structure in place

Pro tip: You can even use both in the same project! Parse once with jsonpath-ng for complex stuff, then hand the result off to json-key-parser for final flattening.

Bottom Line

If jsonpath-ng is the Swiss Army knife of JSON querying, json-key-parser is the precision scalpel for the single most common pain point: “I just need these damn keys.”

For 80% of real-world Python scripts dealing with APIs and configs, json-key-parser will save you more time and frustration than any other library released this year.

Try it right now:

pip install json-key-parser

Then compare it yourself with the exact same data in both libraries. I bet you’ll delete a bunch of boilerplate within five minutes.

What’s your biggest JSON-parsing headache right now? Drop it in the comments — I’ll show you which tool slays it faster.