DEV Community

Cover image for Where 25 OpenAI-Compatible APIs Disagree with OpenAI
Kuldeep Paul
Kuldeep Paul

Posted on

Where 25 OpenAI-Compatible APIs Disagree with OpenAI

#api #ai #programming #devops

Where 25 OpenAI-Compatible APIs Disagree with OpenAI

TL;DR

  • Dozens of LLM providers advertise OpenAI-compatible APIs, but strict client SDKs frequently fail due to payload mismatches.
  • Primary discrepancies center on streaming tool-call deltas, omitted chunk indices, conflicting finish_reason values, and non-standard usage accounting.
  • Reasoning models compound these fractures by splitting thought traces between vendor-specific fields like reasoning_content and raw message bodies.
  • Error handling remains fragmented, with self-hosted runtimes returning framework-level schemas instead of nested OpenAI error envelopes.
  • Dedicated routing gateways resolve these edge cases before payloads reach client applications.

More than 25 inference providers and self-hosted model engines now market drop-in OpenAI-compatible APIs, yet subtle contract divergences across streaming, tool calls, and error handling regularly break production clients. While changing a base URL and an API key allows basic text completions to succeed, production systems built on typed SDKs encounter validation errors, lost tool arguments, and broken retry loops. Bifrost, an open-source AI gateway written in Go by Maxim AI, addresses this fragmentation by translating divergent provider responses into the exact wire format expected by OpenAI SDKs. Understanding where these implementations disagree with the canonical specification is essential for maintaining resilient AI infrastructure.

The Illusion of Drop-In Compatibility

An OpenAI-compatible API is an HTTP interface that mirrors the paths, request bodies, and response structures of the OpenAI REST specification, specifically the /v1/chat/completions endpoint. When an endpoint adheres to this standard, client libraries such as the official openai Python and TypeScript packages can interact with the backend without code modifications beyond changing the base URL.

Expected OpenAI Chat Completion Architecture:

[Client Application] 
       │  (OpenAI SDK: Python / TypeScript / Go)
       ▼
[Target Endpoint: /v1/chat/completions]
       │
       ├─► Request Parsing   (Validates roles, tools, response_format)
       ├─► Model Inference   (Generates tokens, reasoning, tool arguments)
       └─► SSE Serialization (Emits spec-compliant delta chunks & usage)
Enter fullscreen mode Exit fullscreen mode

In practice, full compliance is rare. The official OpenAI client library validates response structures using strict schemas such as Pydantic in Python. When an upstream provider omits a required property, changes an enumerated string, or formats an error payload differently, the SDK raises an internal client-side error. These failures do not stem from model generation quality, but from wire-level protocol violations that occur during edge cases such as tool calling, chunked streaming, and context overflows.

How 25 "Compatible" Endpoints Were Surveyed

To map the reality of provider compatibility, 25 popular inference providers, cloud platforms, and local serving engines were evaluated against the standard OpenAI Chat Completions specification:

  1. vLLM
  2. Ollama
  3. SGLang
  4. Groq
  5. Mistral AI
  6. Together AI
  7. Fireworks AI
  8. Cerebras
  9. DeepSeek
  10. Google Gemini (OpenAI compatibility mode)
  11. Anthropic (OpenAI SDK adapter layer)
  12. Perplexity
  13. OpenRouter
  14. Cohere
  15. xAI
  16. Nebius
  17. Lepton AI
  18. Cloudflare AI Gateway
  19. SambaNova Systems
  20. FriendliAI
  21. Modal (vLLM container runtime)
  22. Lambda Labs Inference
  23. Hyperbolic
  24. Novita AI
  25. Predibase

Testing evaluated how each endpoint handles five critical operational areas: streaming tool-call accumulation, final chunk finish_reason reporting, token usage inclusion via stream_options, handling of the developer system role, and error object structure.

Provider / Runtime Tool Call Delta Index Final finish_reason Streaming Usage (stream_options) Developer Role Accepted Error Payload Envelope
OpenAI (Canonical) Explicit integer (0, 1) "tool_calls" Supported (trailing chunk) Supported Standard { "error": {...} }
vLLM Explicit integer "tool_calls" Supported (recent releases) Converted to system Standard { "error": {...} }
Ollama Sometimes missing in deltas Varies ("stop" on older tags) Ignored / Emitted unconditionally Converted to system Raw string or nested error
SGLang Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
Groq Explicit integer "tool_calls" Supported Supported Standard { "error": {...} }
Mistral AI Explicit integer "tool_calls" Supported Rejected (HTTP 400) Standard { "error": {...} }
Together AI Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
Fireworks AI Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
Cerebras Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
DeepSeek Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
Google Gemini Inconsistent across updates "tool_calls" Often omitted Rejected in multi-turn Non-standard error codes
Anthropic (Adapter) Synthetic index "tool_calls" Supported via bridge Hoisted / Merged Mapped from Anthropic errors
Perplexity Not supported on all models "stop" Ignored Converted to system Standard { "error": {...} }
OpenRouter Passes upstream payload Dependent on model Dependent on upstream Normalized Normalized envelope
Cohere Explicit integer "tool_calls" Supported Converted to system Custom error codes
xAI Explicit integer "tool_calls" Supported Supported Standard { "error": {...} }
Nebius Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
Lepton AI Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
Cloudflare AI Gateway Passes upstream payload Dependent on upstream Dependent on upstream Passes upstream Upstream or custom 5xx
SambaNova Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
FriendliAI Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
Modal (vLLM) Matches vLLM version Matches vLLM version Matches vLLM version Matches vLLM version FastAPI 422 on bad types
Lambda Labs Explicit integer "tool_calls" Varies by template Converted to system Standard { "error": {...} }
Hyperbolic Explicit integer "tool_calls" Varies Converted to system Standard { "error": {...} }
Novita AI Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }
Predibase Explicit integer "tool_calls" Supported Converted to system Standard { "error": {...} }

The data confirms that while top-line text streaming works almost universally, structural consistency fractures as soon as an application exercises complex orchestration patterns.

Streaming Tool Calls and Missing Indices

Streaming tool execution is the most fragile interaction pattern across compatible backends. According to the official OpenAI Chat Completions reference, an assistant calling a function emits an initial chunk containing the tool index, the tool call identifier (id), the type declaration ("type": "function"), and the start of the function name. Subsequent chunks stream string fragments of the JSON arguments object under choices[0].delta.tool_calls[i].function.arguments.

/* Canonical OpenAI first tool chunk */
{
  "id": "chatcmpl-A1B2",
  "object": "chat.completion.chunk",
  "choices": [
    {
      "index": 0,
      "delta": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "index": 0,
            "id": "call_abc123",
            "type": "function",
            "function": {
              "name": "lookup_record",
              "arguments": ""
            }
          }
        ]
      },
      "finish_reason": null
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Several independent implementations deviate from this sequence. In earlier releases of local engines and third-party gateways, the index property inside the tool_calls array was omitted after the first chunk. Without an explicit array index, the official OpenAI Python SDK cannot safely associate incoming argument fragments with parallel tool invocations.

Furthermore, implementations such as Gemini's compatibility endpoint and community wrappers have intermittently omitted the id field on subsequent chunks or skipped "type": "function". If an agent orchestrator relies on strict Pydantic deserialization, missing attributes trigger an immediate schema validation exception mid-stream.

Platforms deploying Bifrost prevent these errors because its Go-based pipeline normalizes tool deltas in flight. Bifrost tracks active tool call slots per request and injects required structural fields before re-emitting the Server-Sent Events (SSE) to the downstream client.

An intricate industrial sorting mechanism with several mechanical track switches separating data packets into misaligned

Finish Reasons: When "tool_calls" Becomes "stop"

In the OpenAI protocol, finish_reason signals the operational outcome of a generation step. If the model determines that an external function must be executed, the final message choice MUST set finish_reason: "tool_calls". If the generation ends naturally, it emits finish_reason: "stop".

OpenAI Canonical Finish Sequence:
Stream Chunks ──► Argument Deltas ──► Last Chunk: finish_reason = "tool_calls"
                                            │
                                            ▼
                               [Client Executes Tool]

Broken Non-Canonical Sequence:
Stream Chunks ──► Argument Deltas ──► Last Chunk: finish_reason = "stop"
                                            │
                                            ▼
                     [Client Assumes Conversation Complete -> Hangs]
Enter fullscreen mode Exit fullscreen mode

Multiple providers fail to preserve this distinction:

  • Certain local servers (reported across various Open WebUI and Ollama issue threads) return finish_reason: "stop" even when tool_calls payloads are present in the message.
  • Several open-weight model templates emit text content in the same turn as tool calls. In contrast, OpenAI's canonical API historically leaves content null when tool_calls are populated. Models like GLM or customized Qwen builds output conversational commentary directly alongside function calls.
  • Some endpoints report finish_reason: "length" prematurely when reasoning models consume token capacity in unallocated internal buffers.

When an agent framework (such as LangChain, AutoGen, or an enterprise agent loop) evaluates finish_reason: "stop", it assumes the dialogue turn is complete. It fails to dispatch the requested tools, leaving the conversation stalled.

Token Usage in Streaming and the stream_options Disconnect

Before OpenAI introduced the stream_options parameter, streaming requests provided zero token accounting data in the stream itself. To inspect consumption, developers were forced to make a secondary non-streaming call or calculate tokens locally using tokenizers like tiktoken.

OpenAI standardized this by accepting:

{
  "model": "gpt-4o",
  "messages": [{"role": "user", "content": "Ping"}],
  "stream": true,
  "stream_options": {
    "include_usage": true
  }
}
Enter fullscreen mode Exit fullscreen mode

When enabled, the server emits a final dedicated chunk right before the data: [DONE] signal. In this chunk, choices is an empty array ([]), and a top-level usage object details prompt_tokens, completion_tokens, and total_tokens.

/* Canonical OpenAI trailing usage chunk */
{
  "id": "chatcmpl-XYZ789",
  "object": "chat.completion.chunk",
  "created": 1715000000,
  "model": "gpt-4o",
  "choices": [],
  "usage": {
    "prompt_tokens": 12,
    "completion_tokens": 8,
    "total_tokens": 20
  }
}
Enter fullscreen mode Exit fullscreen mode

Among the 25 surveyed endpoints, handling of this property falls into three divergent camps:

  1. Standard Trailing Chunk: Providers like Groq, Together AI, and current vLLM implementations mirror OpenAI, emitting the empty choices array with the usage block.
  2. Unconditional Injection: Some inference platforms include usage on the final generation chunk alongside finish_reason: "stop", ignoring whether stream_options was explicitly set. While informative, this violates the expectation of older client parsers that assume chunk structures remain static.
  3. Silent Drop: Smaller hosting providers and older inference servers ignore stream_options entirely. The stream terminates at data: [DONE] without token counts. For organizations tracking billing or enforcing budgets through gateway virtual keys, this creates blind spots unless the proxy calculates usage independently.

To enforce consistent usage metrics across infrastructure, platform teams use Bifrost's governance to track token consumption directly at the proxy layer, reconciling divergent provider streams into uniform metrics.

System Prompts, Developer Roles, and Message Hoisting

With the release of reasoning-focused architectures (such as the o-series), OpenAI introduced the developer role, intended to replace system prompts in specific model families to prioritize instructions against adversarial jailbreaks.

When sending messages containing role: "developer" to standard OpenAI-compatible endpoints:

  • Native OpenAI endpoints accept both system and developer roles.
  • Cloud providers such as Mistral AI strictly enforce allowed role enumerations (system, user, assistant, tool). Passing role: "developer" yields an immediate HTTP 400 invalid_request_error.
  • Anthropic does not support raw system messages interspersed throughout a conversation. In the official Anthropic OpenAI SDK compatibility documentation, the provider explains that it hoists and concatenates multiple system or developer messages into a single top-level system parameter separated by newlines.
  • Inference runtimes running chat templates from Hugging Face may strip unknown roles or throw a template rendering error if the underlying Jinja template does not define an explicit handler for the role.

Applications seeking cross-provider portability must implement role mapping layers that convert developer to system when routing to open-weight models or alternative frontier providers.

Reasoning Tokens: Hidden Chains vs Exposed reasoning_content

The emergence of reasoning models has exposed another architectural rift across compatible APIs. OpenAI's reasoning models do not expose raw Chain-of-Thought (CoT) tokens directly in the message payload of the Chat Completions API. Instead, reasoning tokens are hidden server-side and summarized numerically inside usage.completion_tokens_details.reasoning_tokens.

OpenAI Reasoning Model Payload:
choices[0].message.content = "Final synthesized answer"
usage.completion_tokens_details.reasoning_tokens = 450

Open-Source Reasoning Model Payload (DeepSeek R1 / Qwen):
choices[0].message.content = "<think>\nStep-by-step trace...\n</think>\nFinal answer"
-- OR --
choices[0].message.reasoning_content = "Step-by-step trace..."
choices[0].message.content = "Final answer"
Enter fullscreen mode Exit fullscreen mode

In contrast, open-weight reasoning models (such as DeepSeek R1) output thoughts explicitly. Because there was no existing OpenAI field for this, providers improvised divergent schemas:

  • Embedded Tags: Serving frameworks like Ollama often stream thoughts directly within content, enclosed in <think>...</think> XML blocks. Client applications displaying content verbatim end up showing internal reasoning traces to end users.
  • Custom Schema Fields: Runtimes such as vLLM, Fireworks AI, and DeepSeek's native API introduced an auxiliary property: choices[0].delta.reasoning_content. The standard OpenAI SDK client discards or fails to serialize this field unless configured with loose parsing or custom parameter maps.
  • Multi-Turn Breakage: When an application takes an assistant response containing reasoning_content and passes it back into messages on the next conversation turn, standard OpenAI endpoints reject the payload because reasoning_content is not an accepted input property.

A glowing crystal node acting as an architectural central hub, seamlessly gathering fractured streams of colored light f

Error Schemas: FastAPI Details vs OpenAI Error Envelopes

Error handling is critical for automated recovery, circuit breaking, and user feedback. OpenAI defines an explicit, nested error JSON object accompanied by appropriate HTTP status codes:

/* Canonical OpenAI Error Response (HTTP 400 / 429 / 500) */
{
  "error": {
    "message": "The model `gpt-fake` does not exist",
    "type": "invalid_request_error",
    "param": "model",
    "code": "model_not_found"
  }
}
Enter fullscreen mode Exit fullscreen mode

When calling alternative inference endpoints, client applications frequently encounter non-standard error structures that break SDK error-handling logic:

  • FastAPI Validation Errors: Runtimes built on Python web frameworks often return standard Pydantic validation errors under { "detail": [ { "loc": [...], "msg": "...", "type": "..." } ] } with an HTTP 422 Unprocessable Entity status code. When the official OpenAI client parses this response, it cannot find the root .error.message attribute, masking the real failure behind an uninformative parsing exception.
  • Raw String Errors: Certain local engines emit plain text strings (e.g., 500 Internal Server Error: model unloaded) without a JSON envelope.
  • HTTP Code Collapsing: Multiple compatible endpoints map all backend exceptions (including provider rate limits, context overflows, and upstream timeouts) to a generic HTTP 500 status code rather than using HTTP 429 for rate limits or HTTP 400 for invalid arguments. This prevents client libraries from executing targeted exponential backoff.

A production-grade infrastructure layer must catch, parse, and normalize these irregular responses into standard OpenAI error envelopes before they reach client applications.

Normalizing Discrepancies at the AI Gateway Layer

Managing cross-provider incompatibilities within application code leads to brittle conditional branches across codebases. A cleaner architectural pattern is to centralize protocol normalization at the gateway layer.

Bifrost serves as a drop-in routing gateway that addresses these wire-level discrepancies. Because Bifrost implements a high-performance proxy written in Go, it inspects and normalizes requests and responses with negligible overhead. In sustained performance tests, Bifrost adds only 11 microseconds of overhead per request at 5,000 requests per second.

Incoming Request (OpenAI SDK)
            │
            ▼
┌────────────────────────────────────────────────────────┐
│                   Bifrost Gateway                      │
│  ├─ Normalizes parameters (e.g., developer -> system)   │
│  ├─ Dispatches to upstream provider (20+ providers)    │
│  ├─ Validates SSE stream structure & tool call indices │
│  ├─ Synthesizes missing usage or error envelopes       │
│  └─ Applies caching, failover, and virtual key budgets │
└────────────────────────────────────────────────────────┘
            │
            ▼
Outgoing Response (Standardized OpenAI Protocol)
Enter fullscreen mode Exit fullscreen mode

Through its drop-in replacement capabilities, Bifrost enables engineering teams to switch between supported providers without writing custom adapters for tool streaming, token accounting, or error handling. When an upstream provider returns transient errors or rate limits, Bifrost executes automatic fallbacks to backup models or secondary keys seamlessly.

Discrepancy Category Raw Upstream Behavior Normalized Gateway Behavior
Streaming Tool Indices Omitted on incremental deltas by some endpoints Gateway tracks tool calls and injects explicit sequential indices
Streaming Token Usage Ignored by legacy endpoints; missing usage blocks Gateway calculates usage or captures provider metadata via observability metrics
Reasoning Traces Exposed as <think> tags or custom fields Gateway can strip, isolate, or route reasoning content transparently
Developer Roles Rejected by strict providers with HTTP 400 Gateway automatically maps developer messages to system prompts
FastAPI / HTTP 422 Errors Returns unparseable { "detail": [...] } Gateway converts validation errors to { "error": { "message": ... } }

Beyond routing and protocol normalization, Bifrost applies governance and security controls (virtual keys, budgets, guardrails, audit logs) centrally, and Bifrost Edge extends that same governance and security to AI traffic on employee machines, with endpoint enforcement on each device. This ensures that whether traffic originates from a production backend or a developer's local coding assistant, interactions conform to enterprise compliance and operational standards.

Frequently Asked Questions

What does "OpenAI-compatible" actually mean in practice?

An OpenAI-compatible API is an HTTP service that implements endpoints mirroring OpenAI's specification, primarily /v1/chat/completions. In practice, compatibility exists on a spectrum. While most providers successfully handle basic non-streaming prompt-response calls, compatibility often breaks when applications use advanced features like streaming tool calls, structured outputs, specific role types, or streaming usage options.

Why do my streaming tool calls break when switching from OpenAI to an open-source model?

Streaming tool calls often break because open-source serving runtimes may omit the required index attribute inside the tool_calls delta array on subsequent chunks. Additionally, some runtimes fail to emit "type": "function" on every chunk or return finish_reason: "stop" instead of "tool_calls". This causes strict SDKs like the official OpenAI Python library to fail schema validation.

How does Bifrost handle differences between OpenAI-compatible providers?

Bifrost acts as an intermediary normalization layer. It intercepts divergent upstream responses, reconstructs missing stream attributes (such as tool indices and standard finish reasons), maps unsupported message roles, and converts non-standard error payloads into compliant OpenAI schemas before returning them to the client application.

Why do some providers return HTTP 422 errors instead of standard OpenAI error envelopes?

Many open-source model serving engines are built using Python web frameworks like FastAPI and Pydantic. When an invalid parameter or malformed request is received, the underlying framework rejects the request before it reaches the inference code, returning a default { "detail": [...] } schema with HTTP status 422 instead of OpenAI's nested { "error": { "message": "...", "type": "..." } } schema.

Can I use the OpenAI Python SDK directly with alternative providers?

Yes, you can initialize the OpenAI client with an alternative base_url and api_key. However, if the target provider deviates from OpenAI's wire protocol during streaming, tool calls, or error states, the client may raise unexpected exceptions. Routing traffic through an AI gateway like Bifrost ensures full wire-level compatibility across providers.

How do reasoning models like DeepSeek R1 break standard OpenAI clients?

OpenAI's reasoning models do not output raw thought chains in the Chat Completions API, reporting reasoning only as a token count in usage. In contrast, open models like DeepSeek R1 return thought processes as raw <think> text or under a non-standard reasoning_content field. If client code sends reasoning_content back to a strict provider on the next turn, the request is rejected.

Building Resilient Infrastructure for Multi-Provider AI

Relying on the assumption that every OpenAI-compatible endpoint behaves identically creates fragility in production AI systems. Wire-level differences in streaming tool deltas, finish reasons, token tracking, and error envelopes mean that swapping model providers involves more than updating an environment variable.

Teams evaluating multi-provider architectures should audit upstream behavioral contracts and isolate their core applications from protocol drift. By deploying a dedicated, high-performance gateway like Bifrost, organizations normalize these differences, enforce unified governance, and achieve genuine model portability. Teams evaluating AI gateways can request a Bifrost demo or review the open-source repository.

Sources

Top comments (0)