DEV Community

ArkNet Digital
ArkNet Digital

Posted on

Five LLM gateway bugs that look like success (and how to test for them)

If your app talks to models through an OpenAI-compatible gateway (LiteLLM or one of its many peers), that gateway is doing a lot on every request. It re-streams the response chunk by chunk, recomputes token usage, works out what the request cost, and has to pass upstream failures back to you honestly.

That is stateful, timing-sensitive code sitting in the hot path of every call. The bugs it produces share one nasty property: they look like success.

The five failure modes

1. A dropped or reordered stream chunk. The answer arrives one word short. Nobody notices in a manual test, because you'd have to know what the full answer should have been.

2. Wrong token usage. prompt_tokens, completion_tokens or total_tokens is off, either in the final streamed chunk or the non-streamed body. Your usage dashboards are now fiction.

3. Wrong cost. The gateway still returns a number. You only find out it was wrong when the bill doesn't match the usage.

4. Cross-stream leakage. Under concurrency, content from one request shows up in another's stream. This is a real bug class in gateways that multiplex streams over shared internal state, and it is also a data-exposure problem.

5. A swallowed upstream error. The upstream fails mid-stream and the gateway reports finish_reason: "stop". To your application, that is indistinguishable from a complete, successful answer. This is the worst one.

Why you can't test this against a real model

The obvious approach is to point a test at the live gateway and check the output. The problem is ground truth: you don't know in advance what a real model will say, how many tokens it will use, or exactly when it will stream each chunk. So you can't tell whether the gateway changed anything on the way through.

The fix is to replace the model with something scripted.

The pattern: a deterministic mock upstream

  1. Start a mock upstream that speaks /v1/chat/completions like a real provider, but every response is scripted: exact content, exact usage, exact timing, and, when you want it, an exact failure.
  2. Configure the gateway under test to use that mock as its upstream, the same way you'd normally point it at a provider.
  3. Send requests through the gateway and compare what comes back with what the mock was told to send.
  checks  ──requests──>  [ gateway under test ]  ──requests──>  mock upstream (ground truth)
          <──responses──                         <──responses──
Enter fullscreen mode Exit fullscreen mode

Each failure mode becomes a check with a precise pass condition:

Check Passes when
streaming fidelity the reassembled stream equals the non-streamed response to the same request
usage accuracy reported token counts equal the mock's scripted usage
cost accuracy the reported cost (e.g. LiteLLM's x-litellm-response-cost header) equals usage × your pricing table
concurrent isolation parallel streams never contain each other's content
error propagation a scripted mid-stream failure surfaces as an error, never as finish_reason: "stop"

Keep the checks independent so a failure tells you exactly which promise broke.

Doing it with an existing tool

I packaged this pattern as an MIT-licensed TypeScript tool called gatewayproof. Configure your gateway's upstream as http://127.0.0.1:5055, start the gateway, then:

npx gatewayproof run --target http://localhost:4000 --mock-port 5055
Enter fullscreen mode Exit fullscreen mode

A run against a gateway with a cost bug looks like this:

PASS  streaming-fidelity -- Streamed content matches non-streamed content, with no drops, dupes, or reordering
PASS  usage-accuracy -- Reported token usage matches the upstream's actual usage
FAIL  cost-accuracy -- Reported cost matches usage x the supplied pricing table
      expected cost 0.00019999999999999998, got 0.000001
PASS  concurrent-isolation -- Concurrent streamed requests never leak content across streams
PASS  error-propagation -- A mid-stream upstream failure surfaces as an error, never a silently-truncated success

4/5 checks passed.
Enter fullscreen mode Exit fullscreen mode

It exits 1 on any failure, so it works as a CI gate. There is also a GitHub Action:

- name: Start your gateway
  run: |
    my-gateway serve --port 4000 &   # upstream = http://127.0.0.1:5055
    sleep 2

- name: Run gatewayproof against it
  uses: jayblast-spec/gatewayproof@main
  with:
    target: http://localhost:4000
    mock-port: '5055'
Enter fullscreen mode Exit fullscreen mode

If you are building a gateway yourself, use it as a library inside your own test suite:

import { startMockUpstream, runChecks } from "gatewayproof";

const mock = await startMockUpstream(5055);
const gateway = await startMyGateway({ upstreamUrl: mock.url });

const report = await runChecks({ targetBaseUrl: gateway.url, mock });
console.log(report.passed, report.results);

await gateway.stop();
await mock.stop();
Enter fullscreen mode Exit fullscreen mode

What this does not cover

It is not a load test or a benchmark, and it is not a fuzzer: every check uses a fixed script. It also doesn't yet cover provider-specific features such as function calling or vision inputs. It covers the chat-completions streaming, usage and cost surface every OpenAI-compatible gateway shares, which is where the most expensive silent bugs tend to live.

Takeaway

Whatever tool you use, the principle carries over: don't test a gateway against a real model. Script the upstream, push requests through the gateway, and assert on the difference. That turns "it seemed fine when I tried it" into a pass/fail you can put in CI.


I wrote gatewayproof, so I'm biased; issues and PRs are welcome on GitHub. This article was drafted with AI assistance and checked against the tool's source and README by me.

Top comments (3)

Collapse
 
mp3sss31 profile image
mp3sss •

The ground-truth problem is exactly why these need a stub upstream rather than a live model: return a fixed chunk sequence, deterministic usage/cost, and injected mid-stream failures and concurrency interleavings, then assert the gateway preserves chunk order and count, recomputes usage and cost consistently, and maps upstream failures to an honest finish_reason instead of stop. Cross-stream leakage is the concurrency/shared-state isolation bug class, and the swallowed-error case hurts most because the app treats a 200 and finish_reason stop as a complete answer. The same caution applies to any multiplexed IPC stream on the desktop side: key every message by request/session id so concurrent channels can't bleed into each other, and propagate failure events verbatim instead of marking the stream cleanly closed.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

Wrong token counts are the bug I'd check first, because the cost math is built on them. Compare the gateway's count with the usage block the provider returns for the same call. If those two disagree on one call, every number downstream of it is suspect.

Collapse
 
reidmarlow profile image
Reid Marlow •

One variant of the cost bug that shows up in coding-agent gateways is cached token pricing. Providers bill uncached inputs, cache writes, and cache hits at separate rates, so a cost check built only on prompt_tokens and completion_tokens can pass synthetic tests while overstating real agent bills by an order of magnitude. Including prompt_tokens_details.cached_tokens in the mock payload catches that math before production.