OpenAI-Compatible Is Not Agent-Compatible: A Practical Smoke Test

Most providers can pass a basic POST /chat/completions request.
That is useful, but it is not enough if your real workload is a coding agent.

Agent loops stress more than transport:

• tool-call formatting
• streaming reliability
• retries after partial failure
• session handoff
• usage reporting
• timeout behavior
• billing traceability

This post is a practical smoke test you can run against any OpenAI-compatible API before you trust it for a coding agent.

1) Why /chat/completions alone is not enough

A request can succeed and still fail the job you actually care about.

For example:

• the SDK call works, but tool calls are malformed
• streaming starts, but the stream breaks mid-response
• the provider returns content, but usage is missing
• retries duplicate a tool call or change the answer
• the API accepts the model name, but the agent loop cannot continue cleanly

If your app uses Codex, OpenCode, Cursor, Claude Code, or a similar coding agent, you need to test the whole loop, not just one happy-path request.

2) Basic authentication and model list check

Start by verifying the base URL and API key.

export BASE_URL="https://example.com/v1"
export API_KEY="sk-your-key"

curl -sS "$BASE_URL/models" \
  -H "Authorization: Bearer $API_KEY" | jq .

What to confirm:

• the request returns JSON
• the model list is readable
• the model IDs are the ones your SDK expects
• the response does not expose private metadata you did not ask for

If /models fails, stop there.

3) Ordinary chat completions test

Use one small prompt and one known model.

curl -sS "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-id",
    "messages": [
      { "role": "user", "content": "Say hello in one short sentence." }
    ],
    "temperature": 0
  }' | jq .

Check:

• HTTP status is 200
• content is non-empty
• the response format matches your client
• usage is present when expected

4) Streaming test

Streaming is where many providers diverge.

curl -N "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-id",
    "messages": [
      { "role": "user", "content": "Count from 1 to 5 slowly." }
    ],
    "stream": true
  }'

Check:

• the first chunk arrives promptly
• chunks keep flowing
• the stream ends cleanly
• the client does not hang waiting for the final marker

5) Tool call test

Agent compatibility usually depends on tool calls.

Example request:

curl -sS "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-id",
    "messages": [
      {
        "role": "user",
        "content": "If you need a calculator, call the tool and return the result."
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "calculator",
          "description": "Add two numbers",
          "parameters": {
            "type": "object",
            "properties": {
              "a": { "type": "number" },
              "b": { "type": "number" }
            },
            "required": ["a", "b"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }' | jq .

Check:

• the tool call is syntactically valid
• arguments are complete
• the tool call can be parsed by your client
• a retry does not duplicate the tool call unexpectedly

6) Responses API compatibility check

If your client uses the Responses API, test that path too.

curl -sS "$BASE_URL/responses" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-id",
    "input": "Give me a one-line summary of why agent testing is different from chat testing."
  }' | jq .

Check:

• the route exists
• the response shape is valid for your client
• usage is present or documented
• streaming works if you enable it

7) Error codes and timeout behavior

Good APIs are not only correct when happy.

Test at least one failure path:

curl -sS -o /tmp/devto-smoke-error.json -w "%{http_code}\n" "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "missing-model",
    "messages": [
      { "role": "user", "content": "Hello" }
    ]
  }'

Check:

• invalid model returns a clear error
• timeout behavior is documented or observable
• the client can distinguish retryable and non-retryable failures

8) Usage, cache, and billing logs

After a successful request, verify the accounting trail.

Look for:

• input tokens
• output tokens
• cached input or cache read tokens
• cache write tokens if supported
• request status
• model name
• request duration
• final cost

The main question is simple:

Can you explain the bill from the logs?

If not, the provider may still be usable for chat, but it is harder to trust for production agent work.

9) Copy-paste smoke test checklist

• [ ] /models returns valid JSON
• [ ] basic chat request succeeds
• [ ] streaming request completes cleanly
• [ ] tool call is emitted correctly
• [ ] Responses API works if your client uses it
• [ ] invalid model returns a readable error
• [ ] timeout and retry behavior are understandable
• [ ] usage fields are present
• [ ] billing logs match the request

10) Final judgment table

Check	Pass?
Basic compatible
Streaming compatible
Tool compatible
Agent compatible

If a provider only passes the first row, it may still be fine for chat.
If it passes all four, it is much more plausible for a coding agent workflow.

Closing note

This article is about how to test the route, not about any one vendor.
That matters because the gap between “OpenAI-compatible” and “agent-compatible” is often where the real integration risk lives.

Disclosure: I’m building Your Model, an OpenAI-compatible multi-model API. The checklist above is based on the compatibility issues I look for when testing API routes.

https://y-models.com

This article was prepared with AI-assisted research and editing, then reviewed before publication.