If your app talks to models through an OpenAI-compatible gateway (LiteLLM or one of its many peers), that gateway is doing a lot on every request. It re-streams the response chunk by chunk, recomputes token usage, works out what the request cost, and has to pass upstream failures back to you honestly.
That is stateful, timing-sensitive code sitting in the hot path of every call. The bugs it produces share one nasty property: they look like success.
The five failure modes
1. A dropped or reordered stream chunk. The answer arrives one word short. Nobody notices in a manual test, because you'd have to know what the full answer should have been.
2. Wrong token usage. prompt_tokens, completion_tokens or total_tokens is off, either in the final streamed chunk or the non-streamed body. Your usage dashboards are now fiction.
3. Wrong cost. The gateway still returns a number. You only find out it was wrong when the bill doesn't match the usage.
4. Cross-stream leakage. Under concurrency, content from one request shows up in another's stream. This is a real bug class in gateways that multiplex streams over shared internal state, and it is also a data-exposure problem.
5. A swallowed upstream error. The upstream fails mid-stream and the gateway reports finish_reason: "stop". To your application, that is indistinguishable from a complete, successful answer. This is the worst one.
Why you can't test this against a real model
The obvious approach is to point a test at the live gateway and check the output. The problem is ground truth: you don't know in advance what a real model will say, how many tokens it will use, or exactly when it will stream each chunk. So you can't tell whether the gateway changed anything on the way through.
The fix is to replace the model with something scripted.
The pattern: a deterministic mock upstream
- Start a mock upstream that speaks
/v1/chat/completionslike a real provider, but every response is scripted: exact content, exact usage, exact timing, and, when you want it, an exact failure. - Configure the gateway under test to use that mock as its upstream, the same way you'd normally point it at a provider.
- Send requests through the gateway and compare what comes back with what the mock was told to send.
checks ──requests──> [ gateway under test ] ──requests──> mock upstream (ground truth)
<──responses── <──responses──
Each failure mode becomes a check with a precise pass condition:
| Check | Passes when |
|---|---|
| streaming fidelity | the reassembled stream equals the non-streamed response to the same request |
| usage accuracy | reported token counts equal the mock's scripted usage |
| cost accuracy | the reported cost (e.g. LiteLLM's x-litellm-response-cost header) equals usage × your pricing table |
| concurrent isolation | parallel streams never contain each other's content |
| error propagation | a scripted mid-stream failure surfaces as an error, never as finish_reason: "stop"
|
Keep the checks independent so a failure tells you exactly which promise broke.
Doing it with an existing tool
I packaged this pattern as an MIT-licensed TypeScript tool called gatewayproof. Configure your gateway's upstream as http://127.0.0.1:5055, start the gateway, then:
npx gatewayproof run --target http://localhost:4000 --mock-port 5055
A run against a gateway with a cost bug looks like this:
PASS streaming-fidelity -- Streamed content matches non-streamed content, with no drops, dupes, or reordering
PASS usage-accuracy -- Reported token usage matches the upstream's actual usage
FAIL cost-accuracy -- Reported cost matches usage x the supplied pricing table
expected cost 0.00019999999999999998, got 0.000001
PASS concurrent-isolation -- Concurrent streamed requests never leak content across streams
PASS error-propagation -- A mid-stream upstream failure surfaces as an error, never a silently-truncated success
4/5 checks passed.
It exits 1 on any failure, so it works as a CI gate. There is also a GitHub Action:
- name: Start your gateway
run: |
my-gateway serve --port 4000 & # upstream = http://127.0.0.1:5055
sleep 2
- name: Run gatewayproof against it
uses: jayblast-spec/gatewayproof@main
with:
target: http://localhost:4000
mock-port: '5055'
If you are building a gateway yourself, use it as a library inside your own test suite:
import { startMockUpstream, runChecks } from "gatewayproof";
const mock = await startMockUpstream(5055);
const gateway = await startMyGateway({ upstreamUrl: mock.url });
const report = await runChecks({ targetBaseUrl: gateway.url, mock });
console.log(report.passed, report.results);
await gateway.stop();
await mock.stop();
What this does not cover
It is not a load test or a benchmark, and it is not a fuzzer: every check uses a fixed script. It also doesn't yet cover provider-specific features such as function calling or vision inputs. It covers the chat-completions streaming, usage and cost surface every OpenAI-compatible gateway shares, which is where the most expensive silent bugs tend to live.
Takeaway
Whatever tool you use, the principle carries over: don't test a gateway against a real model. Script the upstream, push requests through the gateway, and assert on the difference. That turns "it seemed fine when I tried it" into a pass/fail you can put in CI.
I wrote gatewayproof, so I'm biased; issues and PRs are welcome on GitHub. This article was drafted with AI assistance and checked against the tool's source and README by me.
Top comments (3)
The ground-truth problem is exactly why these need a stub upstream rather than a live model: return a fixed chunk sequence, deterministic usage/cost, and injected mid-stream failures and concurrency interleavings, then assert the gateway preserves chunk order and count, recomputes usage and cost consistently, and maps upstream failures to an honest finish_reason instead of stop. Cross-stream leakage is the concurrency/shared-state isolation bug class, and the swallowed-error case hurts most because the app treats a 200 and finish_reason stop as a complete answer. The same caution applies to any multiplexed IPC stream on the desktop side: key every message by request/session id so concurrent channels can't bleed into each other, and propagate failure events verbatim instead of marking the stream cleanly closed.
Wrong token counts are the bug I'd check first, because the cost math is built on them. Compare the gateway's count with the usage block the provider returns for the same call. If those two disagree on one call, every number downstream of it is suspect.
One variant of the cost bug that shows up in coding-agent gateways is cached token pricing. Providers bill uncached inputs, cache writes, and cache hits at separate rates, so a cost check built only on prompt_tokens and completion_tokens can pass synthetic tests while overstating real agent bills by an order of magnitude. Including prompt_tokens_details.cached_tokens in the mock payload catches that math before production.