What 79 Production Probes Taught Us About OpenAI-Compatible APIs

stelkey — Wed, 26 Aug 2026 12:51:06 +0000

Disclosure: AI assistance was used to organize and edit this article. I manually
checked the technical claims against the underlying test evidence and take
responsibility for the final text.

An endpoint returning HTTP 200 is not enough to call it OpenAI-compatible.

Real clients depend on request validation, response shapes, public model identity,
event-stream framing, termination, usage accounting, authentication errors, and
repeatable behavior across every route a gateway advertises. A one-line curl command
can succeed while an SDK still hangs, a wallet is charged twice, or an upstream model
name leaks through the public boundary.

This article describes a production canary run performed on August 15, 2026. The
frozen catalog contained 40 public routes:

32 Chat routes, each tested in JSON and SSE modes;
7 Responses routes, each tested in JSON and SSE modes;
1 Embeddings route, tested in JSON mode.

That produced 79 protocol probes in total. All 79 completed successfully after the
release gate caught and corrected two Embeddings compatibility defects. The method is
vendor-neutral and can be applied to any OpenAI-compatible gateway.

1. Freeze the public contract before sending traffic

Fetch the public model catalog once, store the exact set, and calculate an immutable
digest. Do not rediscover the catalog halfway through a test run.

For each public ID, record:

the model ID that clients are expected to send;
the exact supported protocol;
the endpoint and response modes covered by the release;
any public limits or prices that have a verified source;
the catalog digest used by the test runner.

“Configured upstream,” “present in a database,” “listed publicly,” and “proven in the
customer path” are four different states. The test matrix should be generated from the
public state, not from an internal provider inventory.

2. Generate probes from the protocol declaration

A useful test count is derived, not hand-written:

Chat:       32 routes × (JSON + SSE) = 64 probes
Responses:   7 routes × (JSON + SSE) = 14 probes
Embeddings:  1 route  × JSON         =  1 probe
                                      ---------
                                            79

This prevents two common mistakes: sending an SSE test to an endpoint that does not
stream, or counting one successful route as evidence for a different protocol.

3. Use a disposable identity and a hard cost ceiling

The canary should not reuse a customer account or key. Create a short-lived synthetic
identity with:

a model-scoped or release-scoped API key;
a small explicit wallet ceiling;
an expiry measured in minutes;
a unique traffic label excluded from customer analytics;
cleanup logic prepared before the first request.

Run probes serially when upstream limits, ledger order, or settlement workers matter.
If a request fails, fail the release gate closed instead of silently removing that
model from the result.

4. Assert the JSON contract, not only the body text

For Chat and Responses, assert at least:

HTTP 200 and the expected content type;
a stable response object and non-empty request ID;
the requested public model identity or a documented public alias;
usable output content;
a meaningful completion status;
finite, non-negative usage values;
a stable sanitized error shape for deliberately invalid requests.

For Embeddings, also assert:

a non-empty numeric vector;
a stable vector length for the tested route;
the correct public model ID in the response;
usable input and total token accounting.

The public-model assertion matters. In the first Embeddings canary, the request worked,
but the response exposed the private upstream model name. That is a boundary failure
even when the vector itself is correct.

5. Treat SSE as a protocol

For every route that declares streaming, parse frames incrementally and require:

a valid event-stream content type;
valid data: records;
independently parseable JSON frames;
ordered output deltas;
an error frame never being counted as a success;
exactly one unambiguous terminal condition;
bounded connection close;
attributable usage when the gateway promises it.

“Some text arrived” is not enough. Duplicate terminal markers, truncated UTF-8, a
missing final event, or a socket that never closes can all break otherwise standard
clients.

6. Verify status-code semantics

Compatibility includes the HTTP layer. The first successful Embeddings response in
this release returned 201 because of a framework default. The payload was valid, but
the public contract required 200.

That defect was only visible because the canary asserted the exact status code rather
than accepting every 2xx response. After the route declared 200 explicitly, the test
was rerun from the customer boundary.

7. Reconcile usage, billing, and wallet movement

After each request, the authoritative records should satisfy one invariant:

request usage
  -> one settled charge
  -> one wallet movement
  -> no pending liability
  -> no second settlement during retry or readback

Use the same decimal scale and rounding contract as the production ledger. Missing,
negative, infinite, or contradictory usage must fail closed before a customer is
charged.

Also test idempotency. Re-reading a request, receiving a duplicate callback, or
retrying a settlement worker must not debit the wallet twice.

8. Make revocation part of the release gate

After all probes finish:

revoke the disposable key;
repeat one minimal request with that exact key;
require HTTP 401 with a sanitized error body;
confirm the request did not reach upstream routing or billing.

A dashboard saying “revoked” is not proof until the API boundary enforces it.

9. Cleanup is part of the result

The run is not complete when the final model returns output. It is complete when the
temporary state is gone.

Check for zero release-specific residue across:

active synthetic users and keys;
in-flight requests;
unsettled usage and pending liabilities;
spend-limit leases and retry markers;
temporary wallet and ledger rows;
test-only sessions and cached credentials.

Keep immutable evidence for the catalog, runner, runtime image, accounting summary,
and final readback. Never store API keys, cookies, provider secrets, payment
credentials, or private prompts in the report.

10. Publish the limitations next to the pass count

The 79/79 result proves only the matrix that was actually executed: Chat JSON/SSE,
Responses JSON/SSE, and Embeddings JSON for the exact frozen routes.

It does not automatically prove:

tool calling;
vision or audio;
computer control;
Anthropic Messages compatibility;
every optional parameter;
model quality or long-term availability.

Each extension needs its own capability matrix. A model name containing vl,
reasoner, or computer-use is not a substitute for a protocol-level canary.

Reusable release checklist

[ ] Freeze and hash the public catalog.
[ ] Generate probes from each model's declared protocol.
[ ] Use a disposable identity, limited key, expiry, and hard cost ceiling.
[ ] Assert exact JSON fields, public model identity, and HTTP status.
[ ] Assert SSE framing, ordered deltas, termination, and bounded close.
[ ] Validate Embeddings vector shape and public model normalization.
[ ] Reconcile usage, one settled charge, wallet movement, and pending liability.
[ ] Retry readback to detect duplicate settlement.
[ ] Revoke the key and require 401 without upstream work.
[ ] Remove release-specific synthetic state and prove cleanup.
[ ] State explicitly what the matrix did not test.

Compatibility is a system property. The useful question is not “Did one request return
200?” It is “Can every advertised route complete its declared contract, settle once,
fail safely, and leave no release-specific test state behind?”

DEV Community: stelkey