Stop Rebuilding Context on Every Agent Turn

Every senior developer I know has hit the same wall: you build a promising agentic workflow, it works beautifully on a five-step demo, and then it falls apart in production when the tool call count climbs past fifteen or twenty. The latency isn't just annoying. It's architectural.

The reason is almost always the same thing: stateless HTTP round-trips compounding in a loop that was never designed for them.

The Hidden Cost of "Just Send Another Request"

A typical agentic loop looks deceptively simple on paper. Determine the next action, call a tool, receive the output, feed it back, repeat. What makes this painful at scale is what each iteration actually does over HTTP.

Every new turn fires a fresh request. That request carries the full context: system prompt, all prior messages, every previous tool call and its result. By turn twelve, you're transmitting kilobytes of conversation history that the model already processed three turns ago. By turn twenty, that overhead has compounded into something that visibly degrades user experience and burns tokens you didn't need to spend.

This isn't a bug in your implementation. It's the natural consequence of treating a stateful conversation as a series of stateless transactions.

What Persistent Connections Actually Change

WebSocket mode in OpenAI's Responses API takes a different approach entirely. Instead of tearing down and rebuilding the connection on every turn, it keeps a single connection alive for the duration of the session. Each subsequent turn passes only the new input items plus a previous_response_id reference. The model reconstructs context from what it already holds, not from what you re-transmit.

The practical result is significant. OpenAI reports roughly 40% faster end-to-end execution for agentic rollouts with twenty or more tool calls, and that number makes sense when you map out what's actually being eliminated. You're removing repeated serialization of large payloads, repeated TCP handshakes, repeated token processing of already-seen context. Each turn gets lighter instead of heavier.

A minimal implementation looks something like this:

import asyncio
import websockets
import json

async def run_agent_loop(initial_input):
    uri = "wss://api.openai.com/v1/realtime"
    headers = {"Authorization": f"Bearer {OPENAI_API_KEY}"}

    async with websockets.connect(uri, extra_headers=headers) as ws:
        # First turn: send full context
        await ws.send(json.dumps({
            "type": "conversation.item.create",
            "item": {"role": "user", "content": initial_input}
        }))
        await ws.send(json.dumps({"type": "response.create"}))

        previous_response_id = None

        async for message in ws:
            event = json.loads(message)

            if event["type"] == "response.done":
                previous_response_id = event["response"]["id"]

                # Subsequent turns: reference previous response only
                tool_result = run_tool(event)
                await ws.send(json.dumps({
                    "type": "conversation.item.create",
                    "previous_response_id": previous_response_id,
                    "item": {"role": "tool", "content": tool_result}
                }))
                await ws.send(json.dumps({"type": "response.create"}))

The key shift is that previous_response_id reference. You're no longer the source of truth for conversation history on every turn. The session state lives in the connection, not in the payload.

Why This Pattern Generalizes Beyond OpenAI

The deeper insight here isn't specific to any particular API. It's about what happens when you stop treating continuous processes as discrete transactions.

The same pattern appears in database connection pooling, streaming data pipelines, and real-time event systems. Repeatedly establishing and tearing down connections to communicate incremental state is expensive regardless of the protocol. The overhead is proportional to how much context you're re-transmitting, and in long-running agent sessions, that context grows every turn.

Turboline's Turbostream handles continuous event streams using exactly this architecture: persistent, stateful connections that pass only new data rather than re-broadcasting full state on each event. The efficiency gains come from the same principle. Reducing redundant transmission per interaction compounds positively across sessions rather than compounding negatively.

For agentic systems specifically, this matters more as AI workflows get more complex. A research agent running thirty tool calls isn't an edge case anymore. It's becoming a normal workload. Infrastructure designed around stateless HTTP treats every one of those thirty turns as equally expensive. Infrastructure designed around persistent connections makes turn thirty nearly as cheap as turn two.

The Takeaway

If you're building agentic workflows and you're still defaulting to HTTP for every turn because that's how your existing API integrations are structured, it's worth treating that as a performance debt rather than an acceptable baseline. The 40% improvement number will vary by workload, but the direction is consistent: longer sessions with more tool calls benefit more from persistent connections, not less. The architecture that serves a five-step demo and the architecture that serves a fifty-step production agent are not the same thing, and now there's native tooling, including support through Vercel's AI Gateway as of July 2026, to bridge that gap without building it yourself.