Many AI providers describe an endpoint as OpenAI-compatible. Usually that means an existing client can send a familiar request and receive a familiar-looking response. That is useful, but it is not a complete compatibility claim.
A coding agent exercises much more of an API than a one-shot chat demo. It discovers models, streams partial output, emits tool calls, retries failures, preserves session state, and needs usage data to explain what happened. A provider can return HTTP 200 for a simple request and still fail as soon as an agent uses one of those paths.
What “OpenAI-compatible” usually means
At minimum, the provider exposes a documented base URL, accepts a familiar authentication format, understands a request with a model and messages, and returns text in a recognizable response shape. Some providers also expose /v1/models, streaming, tools, or the newer Responses API.
The important word is “some”. Compatibility is a matrix, not a yes/no label. Clients may depend on fields that are optional in a simplified implementation. They may also use a different endpoint, model identifier, or retry policy than the example in the provider documentation.
Why coding agents expose more problems
A coding agent turns one user prompt into a sequence of model calls. It may ask the model to inspect files, call a tool, interpret the result, revise a plan, and continue streaming. That creates several points where a provider can diverge from the client’s assumptions:
- model discovery returns an ID that cannot be used for inference;
- a custom base URL is rewritten or joined incorrectly;
- a streamed tool call is split into deltas that the client cannot reconstruct;
- a Responses request is silently downgraded to Chat Completions;
- a retry occurs after partial output and duplicates text or tool execution;
- usage is omitted, delayed, or reported under a different model ID;
- a 429 response has no usable retry information.
The cure is a small, repeatable smoke test rather than a longer model list.
1. Test /v1/models
Start with discovery. Use the exact base URL and authentication method that the agent will use:
curl "$BASE_URL/v1/models" \
-H "Authorization: Bearer $API_KEY"
Record the status, response time, and returned IDs. Pick one ID from the response and use it in the next test. If the provider requires a prefix, an account-specific alias, or a different spelling, that is part of the compatibility contract.
Do not assume that a model displayed on a marketing page is callable by the API. Compare the requested model, upstream model, logged model, and billed model when the request is complete.
2. Chat Completions versus Responses
Send one minimal Chat Completions request and confirm the roles, content, finish reason, and usage fields. Then test /v1/responses separately. A successful Chat Completions request does not prove Responses support.
curl "$BASE_URL/v1/responses" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"MODEL_ID","input":"Reply with OK"}'
For a real agent, compare the fields the client reads rather than only checking that text appears somewhere in the JSON. Response IDs, output item types, status fields, and usage placement can affect conversation state and billing.
3. Streaming and SSE
Streaming changes the failure model. Capture every SSE event, verify ordering, and confirm the stream ends cleanly:
curl -N "$BASE_URL/v1/chat/completions" \
-H "Authorization: Bearer $API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"MODEL_ID","stream":true,"messages":[{"role":"user","content":"Count to three"}]}'
Test a slow response and a downstream disconnect. Ask: did the provider continue generating, did final usage arrive, and did the gateway settle the request consistently? A retry is not automatically safe after the client has received text or a partial tool call.
4. Tool calls
Send one deterministic tool schema. Check the tool-call ID, index, name, and argument fragments in both streamed and non-streamed responses. The test passes only when the client can reconstruct one complete call without duplicate fragments or reordered arguments.
Then send the tool result back and continue the conversation. Some endpoints accept tool calls in Chat Completions but use a different structure in Responses. That difference matters to coding agents.
5. Model IDs and custom base URLs
Use the exact model ID returned by discovery. Test case sensitivity, prefixes, aliases, and unsupported models. A clear error is safer than silently falling back to another model.
Also test the configured base URL with and without a /v1 suffix. Verify path joining, redirects, and credential boundaries. A client that works against a first-party URL may still mishandle a self-hosted or gateway URL.
6. Errors, 429s, and retries
Record behavior for invalid requests, missing credentials, unknown models, rate limits, timeouts, and upstream failures. Capture the status, request ID, and Retry-After value when present.
The retry rule should use request state, not only the HTTP status. Before output, a retry may be safe. After partial text, a tool-call delta, or an upstream charge, retrying can duplicate output, execution, or billing. A gateway should record the selected route, attempt number, downstream output state, cancellation state, and final usage source.
7. Usage and billing
Record input, output, cache-read, cache-write, latency, retries, and final charge when the provider exposes them. Compare those values with the request log. Missing usage is not proof of zero usage, and a price shown on a public page is not enough to explain a particular request without the model ID and billing unit.
For a coding agent, compare cost per completed task as well as cost per million tokens. Extra correction passes, retries, or tool loops can change the result.
8. Minimal test matrix
| Capability | Why it matters | Basic test |
|---|---|---|
/v1/models |
Agent discovers models | list models |
| Chat Completions | Legacy/current clients | simple request |
| Responses | Newer agent workflows | text response |
| Streaming | Interactive coding agents | SSE stream |
| Tool calls | Agent actions | one tool schema |
| Model IDs | Client selection | exact model name |
| Base URL | Custom provider routing | override provider endpoint |
| Authentication | Credential handling | valid and invalid key |
| Errors | Fallback logic | controlled invalid request |
| 429 | Rate-limit behavior | observe structured response |
| Usage | Cost visibility | verify usage metadata/logs |
The final classification should be explicit:
- Basic compatible: discovery and a minimal request work.
- Streaming compatible: SSE ordering and termination are correct.
- Tool compatible: tool-call deltas reconstruct correctly.
- Agent compatible: the real workflow survives tools, retries, cancellation, sessions, and usage settlement.
Do not mark the last category based on the first one.
I’m using the same compatibility checklist while building Your Model, an OpenAI-compatible multi-model API.
Disclosure: I’m building Your Model.
Top comments (0)