An agent demo usually has one endpoint: send a message, wait, return an answer. Production breaks that shape fast. A request may stream for a minute, consult a tool, wait for approval, enqueue work, and make a change that must never happen twice. If all of that lives behind one POST /chat route, retries and reconnects become accidental business logic.
The agent API gateway pattern fixes this by giving traffic three explicit lanes: streaming for progress, durable state for resumable work, and side effects for controlled writes. The payoff is concrete: a browser can reconnect without restarting a run, a worker can resume after a deploy, and a retry cannot quietly send a duplicate email or charge a customer twice.
This is not a product category or a proxy configuration. It is an API boundary you can implement with an ordinary gateway, application service, queue, and database.
The mismatch between chat endpoints and agent work
Classic request/response APIs assume a bounded operation: validate input, do work, return a result. Agent work violates several assumptions at once:
| API assumption | What agent traffic does instead | Design response |
|---|---|---|
| The client stays connected | Mobile networks and browser tabs disconnect | Stream replayable events |
| A request maps to one action | A run calls several tools and may pause | Persist a run and step state |
| Retrying is harmless | A retry can repeat a write | Require an operation key |
| The response is the result | The useful result may arrive later | Return a stable run URL and status |
| Rate limits measure load | Each request has different cost and risk | Limit by tenant, operation, and budget |
Recent gateway documentation reflects this shift: Microsoft documents policies for model, MCP, and A2A traffic, while Kong documents protocol-aware A2A routing and tracing. AWS’s agent exposure guidance also calls out large contexts, streaming, and per-tenant budgets. The implementation details differ, but the common lesson is useful: agent-shaped traffic needs a deliberate exposure layer, not just an LLM endpoint with a longer timeout.
The three lanes
Think of the gateway as the place where an inbound request becomes a durable, observable run. It owns public API concerns: authentication, tenant resolution, input limits, request IDs, coarse rate limits, and protocol translation. It does not need to become the agent framework.
1. Streaming lane: progress is an event log
Use Server-Sent Events (SSE) or WebSocket delivery for tokens and progress, but do not treat a live connection as the source of truth. Persist small, ordered events first, then fan them out.
Useful event types include:
run.created
assistant.delta
tool.requested
tool.started
tool.completed
approval.requested
run.completed
run.failed
Each event should have a monotonically increasing sequence number. When the browser reconnects, it sends the last sequence it saw; the gateway replays later events and resumes the live stream. The client does not need to guess whether “generating…” means a new run or an old one.
Do not persist every token forever by default. Keep the event log long enough for reconnects and support diagnosis, then apply a retention policy. Store the final artifact separately when it matters.
2. State lane: the run is durable, not the HTTP request
Create a run before doing expensive work. The synchronous request returns quickly with a run ID, an event URL, and a status URL. A worker advances durable steps; the client observes them.
type RunStatus = "queued" | "running" | "waiting_for_approval" |
"completed" | "failed" | "cancelled";
type AgentRun = {
id: string;
tenantId: string;
actorId: string;
status: RunStatus;
inputRef: string; // encrypted or redacted reference, not arbitrary logs
releaseId: string;
budgetCents: number;
createdAt: string;
version: number;
};
The releaseId matters. A replay should know which prompt, tool schema, policy, and routing rules produced a decision. The version supports compare-and-set updates, so two workers cannot both claim the same step.
For short, read-only requests, a synchronous “fast path” is fine. Set a firm deadline. If the work crosses it, convert to a run instead of holding a reverse proxy connection open until something times out.
3. Side-effect lane: writes are operations with receipts
Model output is advisory. A write is an operation. Put a narrow operation service between the agent and a system that can change money, records, messages, access, or infrastructure.
Every requested write needs an operation key that is stable across retries. The key should represent user intent, not a random attempt:
const operationKey = sha256([
tenantId,
actorId,
"create_invoice",
customerId,
invoiceDraftHash,
].join(":"));
Persist the key with pending, approved, executing, succeeded, or failed status. Enforce a unique constraint on (tenant_id, operation_key). If a worker crashes after the downstream system receives the request, reconciliation queries the downstream system using the same idempotency key before trying again.
For high-impact operations, the gateway should return 202 Accepted and a receipt URL—not pretend the agent completed the action. A human approval can move pending to approved; a policy engine can deny it with a reason code. Either path becomes an event in the run timeline.
A small API surface that survives reconnects
Here is a useful starting contract:
POST /v1/runs
Idempotency-Key: client-generated-key
201 Created
{
"run_id": "run_123",
"status": "queued",
"events_url": "/v1/runs/run_123/events",
"status_url": "/v1/runs/run_123"
}
GET /v1/runs/run_123/events
Last-Event-ID: 42
POST /v1/runs/run_123/approvals/approve
POST /v1/runs/run_123/cancel
GET /v1/operations/op_456
Keep request idempotency separate from business-operation idempotency. The first protects POST /runs when a client retries. The second protects a downstream write when a worker retries. Reusing one key for both makes it harder to reason about duplicate creation versus duplicate execution.
Put policies where the facts are available
A gateway can reject an oversized request, authenticate a caller, and apply a tenant quota. It cannot infer whether an invoice total is reasonable without business context. Split policies accordingly.
- At ingress: identity validation, tenant lookup, body size, allowed content type, basic per-tenant concurrency, and request redaction.
- At the run service: maximum steps, cost budget, allowed models, cancellation, and release identity.
- At the tool/operation service: schema validation, resource ownership, scope checks, approval requirement, idempotency, and audit receipt.
- At the backend: final authorization. Gateway authentication is not a substitute for a backend checking that the represented user can access a record.
Pass a signed, minimal actor context downstream: subject, tenant, scopes, request ID, and expiry. Never let the agent invent a tenant ID or pass raw provider credentials in tool arguments.
Streaming safely without hiding failures
Streaming makes a system feel fast, but it can conceal a stalled tool call. Emit explicit state transitions instead of only text deltas. A UI can show “waiting for approval” or “retrieving records” without exposing private tool arguments.
Use heartbeats to keep intermediaries from silently closing a quiet SSE connection. Put a time limit on each tool step, and distinguish three outcomes: retryable failure, terminal failure, and unknown completion. “Unknown” deserves special care: check the operation receipt before retrying a write.
Backpressure is equally important. A client that reads slowly should not cause unbounded buffering. Cap the live buffer, persist a replay point, and disconnect slow consumers with a documented reconnect path.
The observability record to keep
Trace a run across the gateway, orchestrator, tools, and downstream operations with a shared trace ID. The most useful fields are boring and specific:
{
"run_id": "run_123",
"step": 4,
"event": "tool.completed",
"tool": "crm.create_note",
"outcome": "succeeded",
"operation_id": "op_456",
"latency_ms": 811,
"cost_bucket": "within_budget",
"policy_reason": "write_allowed_after_approval"
}
Avoid logging raw prompts, secrets, full customer records, or hidden reasoning. Log references, hashes, outcome codes, and redacted summaries. That gives operators enough evidence to debug a run without turning observability into a second data leak.
A practical rollout plan
Start with one agent workflow that has a visible failure mode, such as a support assistant that drafts a reply and creates a ticket note. Keep the model call read-only first. Then add a single write tool through the operation service.
- Create a durable run table and return a run ID before work begins.
- Add event replay with sequence IDs; test a browser reconnect during a tool call.
- Route one write through an idempotent operation table and unique constraint.
- Add approval for that write, with a clear approve/reject receipt.
- Test worker crash points: before dispatch, after dispatch, and before the receipt is stored.
- Add per-tenant concurrency and budget limits after you can see actual run behavior.
The key test is simple: start a run, disconnect the client, restart a worker, retry the request, and verify that the final timeline is understandable and the write happened zero or one times—never two.
When you do not need this pattern
Do not build a durable gateway layer for a single, read-only completion that can finish in a few seconds. A straightforward authenticated endpoint is easier to operate. Introduce the pattern when work becomes long-running, multi-step, stateful, reconnectable, or capable of causing side effects. Those are the points where a chat endpoint becomes a liability.
Common implementation mistakes
The most expensive mistake is using a queue as the only record of a run. Queues are excellent delivery mechanisms, but messages may be retried, expire, or be consumed before an operator looks. Keep a queryable run record and let queues carry commands to advance it. Another mistake is retrying every exception automatically. A timeout after a read is usually safe to retry; a timeout after a write is an unknown outcome until the operation receipt is reconciled.
Finally, do not make the event stream an audit log by accident. Client-facing events are optimized for useful progress. Operational records are optimized for durable evidence and access control. They may share IDs, but they need different retention, redaction, and access rules.
FAQ
Is an agent API gateway the same as an LLM gateway?
No. An LLM gateway usually focuses on model routing, keys, quotas, and provider observability. An agent API gateway also owns durable runs, streaming replay, tool-operation boundaries, approval states, and client-facing lifecycle APIs. A system can use both layers.
Should I use SSE or WebSockets for agent streaming?
Use SSE when the main requirement is server-to-client progress and easy reconnect with event IDs. Use WebSockets when the client needs frequent bidirectional messages during a run. In either case, persist events so reconnecting does not restart work.
How do I prevent an agent from repeating a write after a retry?
Give the business operation a stable idempotency key, store an operation receipt behind a unique constraint, and reconcile unknown outcomes with the downstream system before retrying. HTTP request retries alone do not solve duplicate side effects.
Where should approvals happen in an agent workflow?
Place approval after the agent has produced a structured, validated operation proposal and before the downstream write. Show the reviewer the action, target, fields, and policy reason—not raw chain-of-thought or secret-bearing context.
Can a gateway authorize access to customer data by itself?
It can authenticate the caller and enforce coarse policy, but the system that owns the data should still enforce resource-level authorization. Carry a minimal signed actor context rather than trusting identity information generated by the agent.
What should happen if a client disconnects during a run?
The run continues according to its policy. Persist progress events and expose a status endpoint; when the client reconnects, replay events after its last sequence number. Do not bind work completion to the lifetime of one HTTP connection.
Closing checklist
Before exposing an agent to real users, make sure you can answer: What is the stable run ID? How does a client reconnect? Which operation receipt proves a write occurred? Which policy allowed it? Which release made the decision? If those answers are durable and observable, the system is ready for the kinds of failures that demos do not show.
Top comments (0)