OpenAI-Compatible Is Not Agent-Compatible: A Practical Smoke Test

Triumph — Mon, 03 Aug 2026 07:14:08 +0000

Most providers can pass a basic POST /chat/completions request.
That is useful, but it is not enough if your real workload is a coding agent.

Agent loops stress more than transport:

tool-call formatting
streaming reliability
retries after partial failure
session handoff
usage reporting
timeout behavior
billing traceability

This post is a practical smoke test you can run against any OpenAI-compatible API before you trust it for a coding agent.

1) Why `/chat/completions` alone is not enough

A request can succeed and still fail the job you actually care about.

For example:

the SDK call works, but tool calls are malformed
streaming starts, but the stream breaks mid-response
the provider returns content, but usage is missing
retries duplicate a tool call or change the answer
the API accepts the model name, but the agent loop cannot continue cleanly

If your app uses Codex, OpenCode, Cursor, Claude Code, or a similar coding agent, you need to test the whole loop, not just one happy-path request.

2) Basic authentication and model list check

Start by verifying the base URL and API key.

export BASE_URL="https://example.com/v1"
export API_KEY="sk-your-key"

curl -sS "$BASE_URL/models" \
  -H "Authorization: Bearer $API_KEY" | jq .

What to confirm:

the request returns JSON
the model list is readable
the model IDs are the ones your SDK expects
the response does not expose private metadata you did not ask for

If /models fails, stop there.

3) Ordinary chat completions test

Use one small prompt and one known model.

curl -sS "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-id",
    "messages": [
      { "role": "user", "content": "Say hello in one short sentence." }
    ],
    "temperature": 0
  }' | jq .

Check:

HTTP status is 200
content is non-empty
the response format matches your client
usage is present when expected

4) Streaming test

Streaming is where many providers diverge.

curl -N "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-id",
    "messages": [
      { "role": "user", "content": "Count from 1 to 5 slowly." }
    ],
    "stream": true
  }'

Check:

the first chunk arrives promptly
chunks keep flowing
the stream ends cleanly
the client does not hang waiting for the final marker

5) Tool call test

Agent compatibility usually depends on tool calls.

Example request:

curl -sS "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-id",
    "messages": [
      {
        "role": "user",
        "content": "If you need a calculator, call the tool and return the result."
      }
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "calculator",
          "description": "Add two numbers",
          "parameters": {
            "type": "object",
            "properties": {
              "a": { "type": "number" },
              "b": { "type": "number" }
            },
            "required": ["a", "b"]
          }
        }
      }
    ],
    "tool_choice": "auto"
  }' | jq .

Check:

the tool call is syntactically valid
arguments are complete
the tool call can be parsed by your client
a retry does not duplicate the tool call unexpectedly

6) Responses API compatibility check

If your client uses the Responses API, test that path too.

curl -sS "$BASE_URL/responses" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model-id",
    "input": "Give me a one-line summary of why agent testing is different from chat testing."
  }' | jq .

Check:

the route exists
the response shape is valid for your client
usage is present or documented
streaming works if you enable it

7) Error codes and timeout behavior

Good APIs are not only correct when happy.

Test at least one failure path:

curl -sS -o /tmp/devto-smoke-error.json -w "%{http_code}\n" "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "missing-model",
    "messages": [
      { "role": "user", "content": "Hello" }
    ]
  }'

Check:

invalid model returns a clear error
timeout behavior is documented or observable
the client can distinguish retryable and non-retryable failures

8) Usage, cache, and billing logs

After a successful request, verify the accounting trail.

Look for:

input tokens
output tokens
cached input or cache read tokens
cache write tokens if supported
request status
model name
request duration
final cost

The main question is simple:

Can you explain the bill from the logs?

If not, the provider may still be usable for chat, but it is harder to trust for production agent work.

9) Copy-paste smoke test checklist

[ ] /models returns valid JSON
[ ] basic chat request succeeds
[ ] streaming request completes cleanly
[ ] tool call is emitted correctly
[ ] Responses API works if your client uses it
[ ] invalid model returns a readable error
[ ] timeout and retry behavior are understandable
[ ] usage fields are present
[ ] billing logs match the request

10) Final judgment table

Check	Pass?
Basic compatible
Streaming compatible
Tool compatible
Agent compatible

If a provider only passes the first row, it may still be fine for chat.
If it passes all four, it is much more plausible for a coding agent workflow.

Closing note

This article is about how to test the route, not about any one vendor.
That matters because the gap between “OpenAI-compatible” and “agent-compatible” is often where the real integration risk lives.

Disclosure: I’m building Your Model, an OpenAI-compatible multi-model API. The checklist above is based on the compatibility issues I look for when testing API routes.

https://y-models.com

This article was prepared with AI-assisted research and editing, then reviewed before publication.

DEV Community: Triumph

OpenAI-Compatible Is Not Agent-Compatible: A Practical Smoke Test

1) Why /chat/completions alone is not enough

2) Basic authentication and model list check

3) Ordinary chat completions test

4) Streaming test

5) Tool call test

6) Responses API compatibility check

7) Error codes and timeout behavior

8) Usage, cache, and billing logs

9) Copy-paste smoke test checklist

10) Final judgment table

Closing note

1) Why `/chat/completions` alone is not enough