I found bugs in LiteLLM and Prism by reading their git history

Zain Ali — Tue, 20 Oct 2026 21:31:23 +0000

A few weeks ago I was reading the source of Prism, the Laravel package for talking to LLM providers. It has ten provider integrations side by side: OpenAI, Anthropic, Groq, Mistral, DeepSeek and so on. Each one has a ToolMap.php that turns your tool definitions into whatever that provider's API expects.

I noticed that a commit called 9447c71 had fixed a crash in the DeepSeek and OpenRouter tool maps. If a tool had no parameters, they used to send "properties": [], an empty JSON array, and the APIs rejected it because they want an object. The fix was a couple of lines.

Then I opened the Groq tool map. Same code, still broken. Mistral, xAI and Z too. The fix had reached two of the ten providers and stopped.

I sent a PR for the other four. But the thing that stuck with me was how ordinary this bug was. Nobody did anything wrong. Someone hit the problem with DeepSeek, fixed DeepSeek, and moved on. The code that needed the same fix was in a different folder.

The pattern

Most codebases have siblings. Provider adapters, database drivers, cache stores, mail transports, payment gateways, platform ports. They all do the same job their own way, and they're usually written by copying the previous one.

When a bug turns up in one sibling, there's a good chance the others have it too. And the fix is already written down in your git history. Someone just has to go and look, and that's the part I wanted a tool for.

half-fixed

half-fixed reads your git history and finds changes that reached some siblings but not the others.

First it groups files that sit at the same path in parallel folders, like src/Providers/*/Maps/ToolMap.php. Then it walks the history for commits that touched some members of a group and pulls out what each one added or removed. For every sibling a commit didn't touch, it looks for the old code. If the old code is still there, that's a finding.

For Prism it reports this:

hf-5505660  high  fixed in 2 of 10, still old in 4
  fixed    DeepSeek, OpenRouter 9447c71 2025-10-14 fix(DeepSeek|OpenRouter): ...
  missing  Groq:21, Mistral:21, XAI:21, Z:21

It only runs git, with no network calls and no dependencies, and there's no AI in the scanner itself. It read 5,000 LiteLLM commits in 69 seconds on my laptop. Try it on any repo:

npx half-fixed scan

What it found

I ran it on six projects that keep implementations side by side: Prism, Laravel, Django, Rails, LiteLLM and Knex. The full results are here, with a link to the exact commit and line for every claim.

LiteLLM was the interesting one. It has over 100 LLM providers and 50 guardrails, which is a lot of siblings. The scan gave 66 high-confidence findings, and I went through every one by hand: read the original commit, read the sibling, check the provider's docs where it mattered.

Verdict	Findings
Real gap	13
Obsolete	29
Handled elsewhere	11
Intentional	12
Unsure	1

The 13 gaps come down to 11 distinct bugs. Two of them I confirmed end to end on a running proxy and sent as PRs:

#44925: an earlier change made disable_custom_api_keys read from config.yaml. The setting right next to it, scope_user_search_to_org, didn't get the same change. If you set it in config.yaml, the settings API said it was on, but internal users could still search every user on the proxy.
#44936: an earlier fix made the Azure passthrough check the value of stream. The vLLM passthrough still only checked that the key existed, so a request with "stream": false got a streaming wrapper and the proxy answered with a 500.

Most findings aren't bugs

Look at that table again. 13 real gaps out of 66. In Laravel, Django and Rails, every finding I checked was either deliberate or harmless: the deprecated Redis cache store in Rails uses a different gem, the old SES transport in Laravel calls an API where the parameter really is called Tags.

In mature, well-reviewed projects, most differences between siblings are on purpose. That's why half-fixed doesn't fix anything automatically. The scanner turns thousands of commits into a short list worth reading. Reading it is the part that needs judgement.

For that part, half-fixed also ships as an Agent Skill for Claude Code and Codex. The skill tells the agent how to triage each finding: read the original commit, read the sibling, decide whether the change applies, and back the verdict with evidence. If it's a real gap, it writes the fix with a test. A second skill writes the pull request the way that project's maintainers expect, by reading their recent merged PRs first.

# Claude Code
/plugin marketplace add heyzainali/half-fixed
/plugin install half-fixed@half-fixed

# Codex
codex plugin marketplace add heyzainali/half-fixed
codex plugin add half-fixed@half-fixed

You can also run it on your own branch before you open a PR. npx half-fixed scan --base origin/main only looks at your commits, so it catches the case where you fixed two siblings and forgot the third.

What doesn't work yet

It groups siblings by file path, so it only works when they live in parallel folders. Knex names its dialect files mysql-querycompiler.js and pg-querycompiler.js in the same folder, and half-fixed finds almost nothing there. It also compares lines of text, so if a sibling was refactored heavily it can miss the match.

Try it

The code is MIT licensed: github.com/heyzainali/half-fixed

If you run it on your own code, tell me what it gets wrong. False positives and missed gaps are the most useful thing I can hear about right now.