I Thought an LLM Gateway Was Just a Proxy. Then I Built One.

Three boundaries an inference request crosses before it reaches a GPU.

When I started building Infera(https://inferai.co.in), my mental model of an inference gateway was simple:

Client → Gateway → Model Server

The gateway would accept an OpenAI-compatible request, choose a model server, forward the request, and stream the result back. In other words, it would be a smarter reverse proxy.

That model did not survive contact with the implementation.

The gateway was not simply moving bytes between two HTTP connections. It was translating between different contracts, making routing decisions from information that was already going stale, and reconciling what the platform believed with what a worker could actually do.

The more useful mental model became:

The request crosses three boundaries: public intent becomes an internal contract, a routing decision meets local worker reality, and an internal stream becomes a public API response.

A single inference request crosses several boundaries, and each boundary has a different version of the truth.

Three of those boundaries changed how I think about building an inference platform.

Boundary 1: The client's request is not the platform's request

A client sends something familiar:

{
  "model": "your-model",
  "messages": [
    { "role": "user", "content": "Explain continuous batching." }
  ],
  "temperature": 0.7,
  "stream": true
}

This describes what the caller wants: a model, a conversation, generation settings, and a streaming preference. That is enough for a public API. It is not enough for the platform operating behind it.

Before Infera can route the request, the gateway has to establish context the client was never asked to provide: who is making the request, which workspace it belongs to, what request ID should follow it through the system, what deadline applies, which generation defaults to fill in, and whether the request has passed admission checks.

So the public OpenAI-compatible request enters a Go gateway, which validates it and converts it into an internal inference request — one carrying platform-generated identity, normalized parameters, workspace context, timestamps, and routing metadata. None of that should ever be the client's responsibility.

Small as it sounds, this creates an architectural rule that pays off everywhere downstream:

The public API should describe what the caller wants. The internal contract should describe what the platform needs to execute it.

Collapse that separation and internal details leak outward. Clients start depending on routing fields, worker identities, or runtime-specific options — and once they do, changing the platform means breaking the product, because the internals have quietly become part of the contract.

The gateway's first job, then, is not forwarding. It is interpretation.

Boundary 2: The router's decision is not the worker's truth

The first time this boundary bit me, it looked like a bug in the router.

A request came in for a model only two workers had loaded. The router picked Worker A, which had reported healthy and idle a second earlier. By the time the request arrived, Worker A had accepted three other requests for that same model, filled its slots, and returned a rejection. The client saw a 503. The logs said the router had sent work to a worker that couldn't take it.

My first instinct was that the router had chosen wrong. It hadn't. It had chosen correctly, from information that was already stale by the time the packet landed. Nothing was broken. I was holding the wrong mental model.

Workers register with the platform and continuously report on themselves — whether they're healthy, which models they've loaded, how much work they're handling, whether they have capacity, which engine and provider they're running. The router evaluates its latest view of the fleet and selects a destination. At first I treated that selection as the answer:

The router selected Worker A.
Therefore, Worker A can run the request.

But the router decides from a snapshot, and the snapshot is always behind. This is the ordinary condition of any distributed control plane, not a defect I introduced: the thing making the decision and the thing doing the work observe the system at different instants. Between selection and execution, the worker may have accepted other requests, started draining, lost the requested model, or begun shutting down.

So the router's decision is not proof the worker is ready. It is the best decision available from the evidence at hand — and the worker still has to verify its own reality.

In Infera, the Python worker checks that its engine is initialized, its state permits new work, and the requested model is still loaded, and it must acquire a local slot before generation begins. The request is effectively admitted twice.

The router asks: based on the latest fleet information, which worker appears capable of handling this? The worker asks: based on my state right now, can I actually handle it? Those checks are not redundant — they defend different boundaries. The gateway holds a distributed, approximate view; the worker holds authoritative knowledge of its own process.

The second check is also cheap insurance against an expensive mistake. Admitting a request a worker can't serve means it occupies — or briefly holds and then releases — a GPU slot a serviceable request could have used. On a fleet you're running to be cost-efficient, a rejection at the door is far cheaper than a request that reaches the engine and dies there.

Which reframes the lesson:

Routing is a decision made from evidence, not a guarantee that the destination is still ready.

Once I accepted that, the interesting question stopped being how do I stop workers from rejecting requests and became what does the platform do when they do. A rejection isn't a failure of the router. It's the system telling the truth about a race the router couldn't have won. Three design questions actually matter:

How is the failure classified? A worker rejecting because it's draining is a different event from one rejecting because the model genuinely failed to load. The first is safe to retry elsewhere; the second may repeat everywhere. The gateway needs these as distinct signals, not one opaque error.

How fast does the registry learn? If Worker A just rejected for lack of capacity, the router's view of Worker A is now known-stale. The sooner that rejection updates the fleet snapshot, the less likely the next request repeats the mistake. This is also why naive "always pick the least-loaded worker" routing backfires — every gateway instance sees the same stale idle worker and stampedes it. Approaches like power-of-two-choices exist precisely to blunt decisions made on aging load data.

Is retrying safe? For a buffered request, retrying elsewhere is usually fine — nothing has reached the client yet. And this is exactly where Boundary 2 collides with Boundary 3.

The moment a request is streaming, retry-safety changes completely. Before the first event goes out, a worker rejection is an ordinary HTTP error: the gateway catches it, picks another worker, and the client never knows. But once the first data: chunk is on the wire, the response status is committed. There is no second worker to fail over to, because the client is already reading a response. A worker that dies mid-stream cannot be transparently retried — the tokens it emitted are gone, and the ones it hadn't reached can't be reissued from a fresh worker without the client seeing a seam.

So the "admitted twice" property and the "committed early" property of streaming are the same constraint viewed from two ends. Double admission is what lets the gateway fail over safely — but only inside the window before streaming begins. That window is the entire retry budget for a streamed request. Once it closes, the worker's local reality is the only reality left, and the platform's job shifts from routing around failure to reporting it honestly. That reporting problem lives at the next boundary.

Worker A was a reasonable choice when the router inspected it. Its rejection does not make the original decision irrational—the system changed between observation and execution.

Boundary 3: The worker's stream is not the client's stream

Once a worker accepts a request, it hands it to an inference engine. Infera's workers are written in Python to stay close to the model-serving ecosystem; the gateway is written in Go and owns the public API, routing, and control-plane behavior. That creates a language and process boundary:

Go gateway → Python worker → Inference engine

For streaming requests, it also creates a protocol boundary.

The client expects an OpenAI-compatible Server-Sent Events response:

data: {"choices":[{"delta":{"content":"Continuous"}}]}

data: {"choices":[{"delta":{"content":" batching"}}]}

data: [DONE]

The Python worker doesn't need to produce that exact public format. It emits a simpler internal stream of newline-delimited chunks describing generated text, tool-call updates, usage, and finish information in the form Infera expects internally. The Go gateway reads that stream and converts each internal chunk into the public OpenAI-compatible response:

Internal worker chunks
        ↓
Gateway translation
        ↓
OpenAI-compatible SSE

The gateway is translating, not blindly forwarding. That separation buys both sides freedom: the worker protocol can evolve around the needs of Infera's engines, and the public API can stay stable for clients. Neither side has to pretend its contract is appropriate everywhere.

It also forces the question Boundary 2 handed off: once the stream is committed and a worker fails, how do you report it?

An ordinary HTTP error is no longer available — the 200 OK and the SSE content type went out with the first chunk. The gateway has two honest options, and Infera uses both depending on where the failure surfaces:

• Terminate the stream with an error finish. The gateway emits a final chunk carrying an error finish reason before closing, so a client parsing the stream can distinguish a real end from an abrupt one. This is the preferred path when the worker fails cleanly enough to signal it.
• Close the connection without [DONE]. When the worker dies hard, there may be no clean chunk to emit. The absence of the terminal [DONE] sentinel is itself the signal: a well-behaved client treats a stream that ends without it as truncated, not complete.

Before streaming begins, the gateway can still change the HTTP response or reroute. After committing the stream, failure must be represented inside the protocol—or by an incomplete stream.

Both paths share one rule — the client must never be able to mistake a partial response for a finished one. A stream that stops early has to be distinguishable from a stream that finished, even though both stop sending bytes.

That is why streaming is not a normal JSON response delivered slowly. It is a separate execution path with its own error and cancellation semantics, and the terminal sentinel is the only thing standing between "the model finished" and "the worker vanished."

Compatibility at the edge does not require every internal component to speak the public protocol — as long as the translation boundary, including how it fails, is explicit.

The gateway was never just a proxy

The original diagram wasn't wrong:

Client → Gateway → Model Server

It was hiding the difficult parts inside the arrows. The gateway has to translate client intent into platform context. The router has to choose a destination from information that may already be aging. The worker has to defend its local state. The response then crosses the same boundaries in reverse — and if it fails on the way back, it has to fail in a way the client can actually interpret.

The result is better described as a chain of contracts than a chain of network hops. Building Infera changed how I think about those contracts:

1. Public and internal requests serve different audiences.
2. A routing decision is evidence-based, not authoritative.
3. Workers must remain the authority on their own local state.
4. Retry-safety has a deadline: the moment the stream commits.
5. A stable external API can sit in front of internals that speak different languages — provided the translation boundary is explicit, failures included.

None of these ideas is specific to one inference runtime. They hold whether the worker eventually delegates to vLLM, SGLang, TensorRT-LLM, or something else. The GPU is important, but it is almost the last participant in the request. Most of the platform's decisions happen before generation ever begins.

In the next article, I'll go deeper into the second boundary: how workers register, how their state reaches the router, and how an inference platform makes decisions when its view of the fleet can never be perfectly current.

---

About Infera

Infera is an inference platform I'm building to provision GPU infrastructure, run model workers, and expose models through an OpenAI-compatible API. (Please do check it out at: https://inferai.co.in)

This series documents the architecture, mistakes, and engineering lessons behind its internals.