If you run a proxy in front of one or more model providers, the interesting failures are not in the model. They are in your layer: a header you forwarded that you should have stripped, an upstream 429 you turned into a 500, a stream you accidentally buffered.
The layer under test
A gateway sits between your applications and one or more providers, and it promises two things at once. Downstream, it promises to speak the OpenAI-compatible contract so that ordinary SDKs work against it. Upstream, it promises to translate faithfully into whatever each provider actually wants. Almost every bug that costs you an incident lives in one of those two promises rather than in the inference.
That means the suite you want is not the one from evaluating a candidate provider. That suite asks whether a remote service is shaped correctly. This one asks whether your code preserves shape, and it should run without touching a provider at all: fast, deterministic, and on every pull request. The distinction matters because a suite that needs a real model is a suite that runs nightly at best, and a proxy bug found nightly has already shipped.
List the promises explicitly before writing a line. A typical set: the response envelope is preserved; the model alias resolves to the documented concrete model; the upstream key never appears downstream and the downstream key never appears upstream; usage is reported and equals what the upstream reported; an upstream error becomes a documented downstream error rather than a stack trace; and a streamed request produces a stream, not a buffered response delivered at the end.
Stub the upstream, not the gateway
The direction of the stub is the design decision. If you mock your own gateway you are testing nothing. What you want is a fake provider: a local HTTP server that returns whatever fixture the test needs, including malformed and hostile fixtures a real provider will not produce on demand. Mock Service Worker’s Node interceptor is the usual choice in TypeScript, and it works at the request level so your gateway’s real HTTP client, real timeouts and real retry logic all execute.
// gateway/contract/upstream.ts
import { setupServer } from "msw/node";
import { http, HttpResponse } from "msw";
export const upstream = setupServer();
const FIXTURE_ID = "chatcmpl-fixture-1";
export const completionFixture = (over = {}) => ({
id: FIXTURE_ID,
object: "chat.completion",
created: 1754870400,
model: "provider-model-v2",
choices: [{
index: 0,
message: { role: "assistant", content: "ok" },
finish_reason: "stop",
}],
usage: { prompt_tokens: 11, completion_tokens: 2, total_tokens: 13 },
...over,
});
export const respondsWith = (status: number, body: unknown) =>
http.post("https://upstream.test/v1/chat/completions", () =>
HttpResponse.json(body as never, { status }));
Two properties make this worth the setup. The fixture is a value, so a test can mutate one field and assert that your gateway notices. And the interceptor records the outbound request, so you can assert on what your gateway sent, which is half the contract and the half that is otherwise invisible.
Pass-through fidelity
The single most valuable assertion in a gateway suite is that unknown fields survive. A gateway that parses an upstream response into a typed struct and re-serialises it will silently drop every field its struct does not know about — and providers add fields constantly. Your users lose logprobs, or a reasoning block, or a cache-hit count, and nobody notices until one of them opens a ticket.
it("preserves fields the gateway does not model", async () => {
upstream.use(respondsWith(200, completionFixture({
system_fingerprint: "fp_abc123",
provider_specific_metadata: { cache_hit: true },
})));
const res = await fetch(gatewayURL + "/v1/chat/completions", {
method: "POST",
headers: { "content-type": "application/json", authorization: "Bearer tenant-key" },
body: JSON.stringify({ model: "alias-fast", messages: [{ role: "user", content: "hi" }] }),
});
const body = await res.json();
expect(body.system_fingerprint).toBe("fp_abc123");
expect(body.provider_specific_metadata).toEqual({ cache_hit: true });
expect(body.usage.total_tokens).toBe(13);
});
The companion assertion runs in the other direction: capture the request your gateway made upstream and check that the client’s parameters arrived intact. A gateway that rebuilds the request body from a known-parameter list drops the one parameter a client added last week, and the client sees a model that ignores their response_format with no error anywhere.
Error mapping and header hygiene
Error mapping is a contract in both directions and it is the part teams write once and never test. Assert each upstream status your gateway can receive against the downstream status it should produce: a 429 must stay a 429 or your clients’ backoff never engages; a Retry-After header must survive, since it carries the only number worth obeying; a 400 from the provider about an invalid parameter must not become a 502, because a 502 tells the client to retry a request that will fail identically forever.
- Upstream 429 with
Retry-After: 3→ downstream 429 with the same header. Assert the header, not only the status. - Upstream 401 → downstream 502 or 500, never 401. A 401 tells your tenant their key is bad, and it is not — yours is.
- Upstream timeout → downstream 504 within your own documented deadline. Assert the elapsed time is bounded, or the test passes while the behaviour is unbounded.
- Upstream returns HTML — a CDN error page rather than JSON — → a structured error, not a JSON parse exception. This is a real production shape and worth a fixture.
Header hygiene deserves its own explicit test because the failure is a security failure rather than an availability one. Assert that the downstream Authorization value never appears in the captured upstream request, and that no upstream provider header that identifies your account is echoed downstream. Both are one-line assertions on captured objects and neither is caught by any other kind of test.
Streaming is a separate contract
A gateway can pass every buffered test and still be broken for streaming, because the two are usually different code paths and the streaming one has a failure mode the buffered one cannot have: it can be correct in content and wrong in time. A proxy that reads the whole upstream stream, assembles it, and writes it out as SSE at the end produces byte-identical output and destroys the only reason anyone streams.
So assert on timing, not only on bytes. With a stub upstream you control the delay between chunks, which makes this deterministic rather than flaky: emit chunk one, wait, emit chunk two, and assert that your gateway delivered the first chunk before the second was sent. Record the wall-clock offset of the first downstream chunk and require it to be below the upstream’s inter-chunk delay. The detail of the wire format — the data: prefix, the blank line separator, the terminating sentinel — is covered in contract tests for streaming chunk format, and a gateway suite should reuse the same parser its clients use rather than writing a second one.
Most of the work in operating a gateway is exactly this suite: the error mapping, the header hygiene, the pass-through fidelity, and the streaming path, maintained across every provider you add. If that layer is not itself the product you are building, Multigrid is that layer as a service — one API and one key, with the provider differences absorbed behind it.
Top comments (0)