OpenAI Responses API: A Practical Guide

TL;DR

• The OpenAI Responses API replaces Chat Completions for new projects, providing native multi-turn state handling and an autonomous tool execution loop within a single API request.
• Developers can maintain conversation state through server-side storage using response identifiers, or operate statelessly using encrypted reasoning tokens for strict data retention compliance.
• Built-in tools allow models to run web searches, execute file retrievals, and invoke remote Model Context Protocol (MCP) servers without requiring client-side orchestration loops.
• Routing traffic through an open-source gateway like Bifrost enables cross-provider translation, automatic failover, semantic caching, and unified policy enforcement across production workloads.

Production AI applications running across multiple LLM endpoints frequently suffer from client-side orchestration bloat, rate-limit bottlenecks, and brittle tool execution loops. The OpenAI Responses API is OpenAI's unified interface designed to consolidate model inference, persistent multi-turn state, and autonomous tool calling into a single endpoint primitive. Engineering teams adopting this primitive often deploy Bifrost, an open-source AI gateway written in Go by Maxim AI, to handle provider routing, fallback failover, and request governance across their production infrastructure. This guide covers how the Responses API works, how its architecture differs from earlier endpoints, and how to operate it reliably at scale.

What is the OpenAI Responses API?

The OpenAI Responses API is a unified programmatic interface hosted at /v1/responses that combines conversational model inference, server-managed interaction state, and autonomous tool execution into a single request lifecycle. It serves as the architectural successor to /v1/chat/completions and the beta Assistants API, streamlining how reasoning models interact with tools and maintain conversational context.

Earlier iterations of OpenAI APIs required developers to choose between two divergent paradigms:

1. Chat Completions (/v1/chat/completions): A stateless primitive where developers had to send the entire conversation history with every request, manually parse tool calls, run the local tools, and send back tool outputs in iterative round-trip network hops.
2. Assistants API (/v1/assistants): A heavily abstracted, stateful ecosystem involving persistent assistants, threads, runs, and polling steps that introduced latency overhead and complex error recovery.

The Responses API resolves this divide by providing a unified request model. Developers can initiate complex reasoning tasks, grant the model access to hosted or remote tools, and let the model execute multiple sequential steps within a single server-side execution loop. Responses can run either statelessly or statefully, retaining reasoning tokens across turns while significantly reducing client-side boilerplate code.

Chat Completions vs. Responses API: Architectural Differences

The architectural difference between the Chat Completions API and the Responses API centers on execution ownership and context propagation. In Chat Completions, the client owns the workflow loop; in the Responses API, the model runtime coordinates tool execution and chain-of-thought progression natively.

When building agentic workflows with Chat Completions, every tool call forces a round-trip network boundary: the model emits a tool call, the client executes the tool, formats the payload, and initiates a subsequent POST request containing the cumulative history. In contrast, the Responses API allows the model to execute built-in tools (such as web search or file search) autonomously in the background before emitting the final answer.

The following table summarizes the core differences between the two architectures:

Capability	Chat Completions (/v1/chat/completions)	Responses API (/v1/responses)
Primary Paradigm	Stateless turn-by-turn chat	Unified reasoning and tool execution loop
State Management	Manual (client resends entire chat history)	Hybrid (server-stored via IDs or client-managed)
Tool Execution	Client-orchestrated round trips	Autonomous server loop for hosted tools and MCP
Reasoning Tokens	Ephemeral or omitted between turns	Preserved across turns via state or encrypted tokens
MCP Integration	Manual client-side plumbing	Direct support for remote MCP endpoints
Input Interface	messages array requiring role definitions	Flexible input string or content blocks, plus instructions
Output Structure	choices[0].message.content	Structured output array containing text, reasoning, and tool calls

This consolidation lowers integration complexity while unlocking higher prompt cache utilization, as static prefixes and server-managed turn states avoid redundant payload re-uploads.

Core Parameters and Request Structure in the Responses API

The request schema for /v1/responses simplifies prompt delivery while introducing parameters tailored for agentic execution. Instead of assembling nested system, developer, and user message dictionaries, applications supply high-level instructions alongside dynamic input items.

The primary parameters accepted by /v1/responses include:

• model: The model identifier (such as gpt-4o, gpt-4.1-mini, or reasoning models like o3).
• input: The prompt string or an array of rich content items (text, images, or previous response references).
• instructions: System-level operational rules that guide model behavior across the lifecycle of the task.
• tools: An array defining built-in tools (e.g., web_search, file_search, code_interpreter) or custom client functions.
• store: A boolean flag determining whether OpenAI retains the response object server-side for subsequent conversational turns.
• previous_response_id: The identifier of an earlier response used to continue a multi-turn thread without resending historical context.
• max_output_tokens: An integer specifying the maximum allowable generation limit.

The following Python example illustrates a basic call using the official OpenAI SDK:

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))

response = client.responses.create(
    model="gpt-4o-mini",
    instructions="You are an enterprise system engineer. Respond with concise technical steps.",
    input="How do I verify TCP socket reuse in Linux?",
    store=True
)

# Output items represent discrete components: reasoning, text, or tool calls
print(response.output_text)
print(f"Response ID for chaining: {response.id}")

The response object returns an output array containing the generated content blocks. For quick retrieval of plain text, the SDK exposes the output_text convenience property, which concatenates text content blocks automatically.

Managing State: Stateful vs. Stateless Workflows

Managing conversation history has historically created difficult trade-offs between network bandwidth, token costs, and data privacy. The Responses API addresses this by supporting two distinct operating modes: server-stored state and client-managed stateless turns.

Server-Stored State (store: True)

When store=True is passed, the platform retains the interaction context, including tool executions and internal chain-of-thought tokens, on OpenAI infrastructure. Subsequent turns simply reference the prior generation using previous_response_id:

# Turn 2: Referencing the previous turn without re-transmitting history
follow_up = client.responses.create(
    model="gpt-4o-mini",
    previous_response_id=response.id,
    input="Can you show the ss command flags for that verification?",
    store=True
)

print(follow_up.output_text)

By passing only the previous_response_id, network payload sizes remain minimal even when conversing over dozens of turns. This approach also maximizes server-side prompt cache hit rates, as the upstream provider retains the exact key-value cache tokens from the previous turn.

Stateless Operation and Encrypted Reasoning (store: False)

Enterprises bound by zero-data-retention (ZDR) mandates or strict compliance frameworks cannot store conversational data on external infrastructure. The Responses API supports these requirements by allowing store=False.

When reasoning models run statelessly, discarding intermediate tokens harms subsequent reasoning steps. To resolve this, the Responses API returns an encrypted_content string inside reasoning output blocks. The client stores this opaque blob locally and passes it back into the input array on the next turn. The upstream model decrypts its own internal chain of thought without exposing intermediate reasoning traces or retaining data on OpenAI disks.

Autonomous Tool Execution and the Built-in Agent Loop

A core feature of the Responses API is its built-in agentic execution loop. In previous APIs, calling a search index or python interpreter required the client application to parse the JSON request, run the execution locally, and post the output back to the model. The Responses API executes hosted tools directly inside OpenAI's execution boundary.

Supported tool types in the Responses API include:

1. web_search: Enables real-time web retrieval during generation to answer queries regarding recent data or documentation.
2. file_search: Connects to vector stores to run semantic retrieval against uploaded organizational documents.
3. code_interpreter: Runs generated Python code inside an isolated sandbox to perform statistical calculations, data analysis, or chart creation.
4. Remote MCP Servers: Connects natively to remote Model Context Protocol endpoints to discover and invoke tools published by internal corporate systems.
5. Custom Functions (function): Standard client-side function calling, where the model halts generation (finish_reason: "tool_calls") and delegates execution to the application.

The following example configures a response request that combines hosted web search with a client-defined function:

response = client.responses.create(
    model="gpt-4o",
    instructions="Analyze current infrastructure advisories and return status reports.",
    input="Check if there are any active network incidents reported for US-East regions.",
    tools=[
        {"type": "web_search"},
        {
            "type": "function",
            "name": "log_incident_ticket",
            "description": "Log an internal tracking ticket for an infrastructure incident",
            "parameters": {
                "type": "object",
                "properties": {
                    "severity": {"type": "string", "enum": ["low", "medium", "critical"]},
                    "summary": {"type": "string"}
                },
                "required": ["severity", "summary"]
            }
        }
    ]
)

During this call, the model can query the web multiple times autonomously. If it determines that an internal ticket must be filed, it halts and returns the function arguments for log_incident_ticket to the application.

Routing and Governing Responses API Traffic with Bifrost

Deploying the Responses API directly against vendor endpoints introduces operational vulnerabilities in enterprise production. Provider-level rate limits, unexpected API outages, lack of egress budget enforcement, and vendor lock-in create production risks.

Bifrost, an open-source AI gateway, solves these operational challenges by acting as a high-performance proxy between client applications and AI providers. Bifrost adds only 11 microseconds of overhead per request at 5,000 requests per second in sustained benchmarks, delivering governance without measurable latency penalties.

Drop-In Configuration

Bifrost functions as a drop-in proxy for OpenAI SDK clients. By updating the base_url parameter, all calls to client.responses.create route directly through Bifrost's routing pipeline:

from openai import OpenAI

# Route Responses API requests through Bifrost with virtual key governance
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="bifrost-access-token",
    default_headers={"x-bf-vk": "vk_engineering_agent_prod"}
)

response = client.responses.create(
    model="gpt-4o-mini",
    input="Generate a status check script for Kubernetes nodes.",
    instructions="Return bash code only."
)

The gateway handles the connection seamlessly, mapping authentication, injecting telemetry tags, and logging token metrics to centralized stores.

Cross-Provider Translation

One of Bifrost's primary architectural capabilities is cross-provider schema translation. While developer code speaks the OpenAI Responses API schema, Bifrost can translate those requests into native upstream formats for providers such as Anthropic Claude, AWS Bedrock, Google Vertex AI, and Cerebras.

If an upstream OpenAI outage occurs, Bifrost's automatic fallbacks route the payload to Anthropic Claude or AWS Bedrock without requiring code modifications or application redeployments. The gateway maps the instructions parameter to Claude's system prompt, translates content blocks, and normalizes the completion back into the Responses API output format.

Beyond server-side routing, Bifrost applies governance and security controls (virtual keys, budgets, guardrails, audit logs) centrally, and Bifrost Edge extends that same governance and security to AI traffic on employee machines, with endpoint enforcement on each device. This dual-layer architecture prevents shadow AI usage while keeping both internal agent scripts and developer CLI tools compliant.

Cost Optimization: Semantic Caching and Prefix Reuse

High-frequency reasoning calls can quickly inflate inference budgets. The Responses API optimizes token efficiency internally through provider prompt caching, but infrastructure-level caching delivers broader savings across an entire organization.

Bifrost implements a multi-tiered caching pipeline for Responses API payloads:

1. Direct Hash Matching: Bifrost checks an exact SHA-256 hash of incoming request structures. If an identical prompt matches a valid cached entry, Bifrost returns the stored response immediately without executing an embedding calculation or forwarding traffic upstream.
2. Semantic Caching: On an exact-match miss, Bifrost evaluates the prompt vector against configured similarity thresholds using semantic caching. Rephrased queries that match historical intent within a threshold (typically 0.8) are served from cache, reducing downstream token costs to zero and cutting round-trip latency.

The table below outlines how caching strategies interact across provider-native and gateway layers:

Optimization Layer	Mechanism	Scope	Primary Benefit
OpenAI Prompt Cache	Automatic 1024+ token prefix matching	Single provider (OpenAI)	Up to 50% discount on cached input tokens
Bifrost Direct Cache	Exact cryptographic payload hash	Gateway layer across all teams	0ms model latency; 100% token cost reduction
Bifrost Semantic Cache	Vector similarity distance match	Gateway layer across all teams	Absorbs rephrased queries; reduces model load
Responses store: True	Server-side conversational session state	Per-thread conversational turns	Eliminates payload re-transmission across turns

Combining gateway semantic caching with the Responses API turn structure provides defense against redundant model invocations in high-throughput customer support or automated triage agents.

Enterprise Governance and MCP Integration

As organizations connect the Responses API to corporate databases and internal tools, controlling access rights becomes critical. The Responses API supports remote Model Context Protocol (MCP) integrations, allowing models to query remote services. However, granting models unconstrained access to remote tools creates data leakage risks.

Bifrost functions as an enterprise-grade MCP gateway that sits between the Responses API and organizational tool servers. Through virtual keys, platform teams enforce granular policies:

• Per-Key Tool Filtering: Restrict which MCP tools an application or model instance can see and call. An analytics agent can be limited to read-only database tools, preventing unauthorized write operations.
• Hierarchical Spend Budgets: Assign rigid monthly or daily financial limits to individual virtual keys, preventing runaway loops from exhausting company API credits.
• Content Guardrails: Inspect inputs and outputs for PII, API tokens, or proprietary source code using enterprise guardrails before data enters or exits vendor networks.
• Compliance Audit Logging: Capture full trace payloads, tool invocation records, and latency metrics to immutable audit logs for SOC 2, HIPAA, and ISO 27001 regulatory validation.

For distributed development teams, Bifrost Edge ensures that terminal tools like Cursor, Claude Code, and Codex CLI follow these same enterprise access parameters. Edge provides native app governance and MCP governance, silently rolling out across fleets via MDM deployment through Jamf or Microsoft Intune.

Migration Guide: Moving from Chat Completions to Responses

Migrating an existing codebase from /v1/chat/completions to /v1/responses requires minimal restructuring while offering immediate architectural cleanup. The following step-by-step procedure guides the transition:

Step 1: Update SDK Method Calls

Replace calls to client.chat.completions.create with client.responses.create. Update parameter mappings according to the revised schema:

• Rename messages containing system prompts to the top-level instructions string.
• Pass user prompts directly as a string to input, or supply a simplified content list.
• Convert max_tokens or max_completion_tokens to max_output_tokens.

Step 2: Modernize Context Management

Evaluate whether client-side conversation arrays can be eliminated. If data retention policies permit, set store=True and pass previous_response_id=response.id on subsequent requests instead of appending messages to a local array. For stateless compliance, extract and return encrypted_content blocks.

Step 3: Shift to Hosted and Remote Tools

Audit existing function-calling implementations. If custom code was used solely to search documents or perform web retrieval, replace those functions with built-in file_search or web_search tools. If connecting to external corporate backends, expose them via standard MCP endpoints rather than maintaining custom API wrappers.

Step 4: Route Through Bifrost for Observability and Fallbacks

Configure your application base URL to point at Bifrost. This allows you to monitor token usage across both legacy Chat Completions endpoints and new Responses API routes through OpenTelemetry tracing and Prometheus metrics, ensuring an uninterrupted transition with fallback redundancy.

Frequently Asked Questions

Is the Chat Completions API deprecated?

No, OpenAI continues to support /v1/chat/completions for existing applications, but all new agentic features, enhanced reasoning models, and built-in tool execution primitives are prioritized on the Responses API. OpenAI recommends the Responses API for all new production projects.

How does the Responses API handle streaming?

The Responses API supports Server-Sent Events (SSE) for streaming responses in real time. When stream=True is passed, the endpoint emits discrete chunk events containing delta text, reasoning tokens, and tool call progress indicators, concluding with final token usage summaries.

Can I use non-OpenAI models with the Responses API?

Natively, the Responses API is an OpenAI proprietary endpoint. However, routing through an AI gateway like Bifrost allows applications to submit Responses API payloads that are translated automatically into upstream formats for Anthropic Claude, AWS Bedrock, Google Vertex AI, and local inference engines.

What is the difference between instructions and input?

The instructions parameter defines global, system-level behavior and developer directives that govern how the model acts across turns. The input parameter represents the immediate turn payload, including user text, uploaded files, images, or conversation items.

How does previous_response_id reduce token costs?

Passing previous_response_id enables the model to reference prior conversation turns stored on server infrastructure rather than requiring the client to resend historical message arrays. This reduces network payload sizes and increases prompt cache hit rates, lowering input token costs.

Does the Responses API work with zero data retention policies?

Yes, setting store=False disables server-side conversation storage. For multi-turn reasoning conversations under zero data retention, the API emits encrypted reasoning tokens that the client preserves and returns on subsequent turns without exposing or persisting data.

Can Bifrost cache Responses API requests?

Yes, Bifrost provides native support for caching /v1/responses traffic. It executes an exact cryptographic hash check first, followed by semantic similarity vector checks, serving repeated or semantically equivalent requests directly from memory or vector stores.

Next Steps and Implementation Strategy

The OpenAI Responses API represents a significant evolution in how engineering teams interact with frontier language models, shifting burden away from client orchestration loops into cohesive model-level execution. Adopting its unified request interface reduces application complexity, optimizes token usage, and unlocks powerful multi-tool agentic capabilities.

To ensure resilience and cost control in production, teams should decouple their client code from vendor-specific endpoints. Routing Responses API traffic through an open-source gateway provides the cross-provider fallbacks, virtual key rate limits, and observability necessary for enterprise scale. Engineering teams evaluating AI gateway architecture can explore the Bifrost open-source repository or request a Bifrost demo to inspect production routing and governance configurations.

Sources

• OpenAI Developers: Migrate to the Responses API
• OpenAI API Reference: Responses Overview
• OpenAI Engineering: Why We Built the Responses API
• Microsoft Learn: Use the Azure OpenAI Responses API