Five Tests an AI Gateway Should Pass Before You Connect a Coding Agent

An API can be OpenAI-compatible and still fail in a real coding-agent workflow.

A successful non-streaming chat request proves that the endpoint understands a basic request and can return a basic response. It does not prove that the gateway handles the stateful parts of agent traffic correctly.

Here are five tests I would run before connecting a coding agent to a custom provider or gateway.

1. Streaming tool calls

Tool calls are often emitted as multiple deltas.

The client needs to reconstruct the tool name, call ID and JSON arguments in the correct order. A weak implementation may duplicate argument fragments, mix two calls together or finish the stream before the arguments are complete.

Test both a single tool call and multiple calls in one response.

2. Responses API compatibility

Many services support /v1/chat/completions but do not fully support newer response formats.

Check whether the gateway correctly handles:

• response objects
• output items
• tool-call events
• usage fields
• streaming event types
• error responses

Protocol conversion should preserve meaning, not merely change field names.

3. Retry without duplicate output or billing

A retry policy needs to know whether the downstream client has already received output.

Retrying before any output may be safe. Retrying after partial output can duplicate content, tool calls or billing.

Useful test cases include:

• upstream timeout before output
• upstream failure after partial output
• downstream disconnect
• cancellation
• final usage arriving after a partial stream

4. Observable usage and errors

A production gateway should make a request explainable.

For each request, you should be able to determine:

• selected model
• selected route or provider
• input and output usage
• cache usage
• retry count
• final error
• billing multiplier
• final charge

Without this information, cost and reliability problems become difficult to distinguish.

5. Session and cancellation behavior

Coding agents often maintain longer workflows than a normal chat client.

Check what happens when:

• the client reconnects
• the process restarts
• a request is cancelled
• the downstream connection closes
• a tool call takes a long time
• the same task status is polled repeatedly

These cases expose state and concurrency problems that a basic smoke test will never find.

A practical minimum test set

Before using a gateway for an agent workflow, I would test:

1. Basic Chat Completions
2. Streaming text
3. Streaming tool calls
4. Responses API
5. Cancellation
6. Retry behavior
7. Usage reconciliation
8. Session recovery

A long model list is useful, but protocol behavior, billing semantics and observability usually matter more once the gateway is connected to a real agent.