Why Is Serving a Model Harder Than Training One?
Everyone building AI thinks the hard part is the model. It's not. It's the plumbing.
Training a model is a batch job. You get a big pile of GPUs, you run for a while, you either get a checkpoint or you get a bill. It's expensive and it's annoying but it is fundamentally a scheduled, finite, tolerably-failing operation. If it dies at hour 40 you restart it and you're annoyed.
Serving a model is a live system with a latency budget, a cost ceiling, and users who notice. It's just a different class of problem, and almost every team that ships an AI feature discovers this the hard way about six weeks after launch.
Let me break down why, because "it's just an API call" has cost people more money than any outage I've seen.
It's just a queue with a weird failure mode
Strip away the branding. An inference endpoint is a queue in front of a GPU pool. Requests come in, get batched, get computed, get returned. That's it. You've built this. It's a web service with an expensive backend.
The weird part is the variance.
A normal API has roughly predictable per-request cost. Hash a string, hit a cache, run a query. Inference does not work like that. The cost of a request depends on how many input tokens, how many output tokens, and output is generated token by token, so a request that looks small can run for thirty seconds if the model decides to be verbose.
That means your capacity planning is wrong by default. You provisioned for requests per second. What you actually have is a system limited by token throughput, where a single bad prompt can eat the capacity of fifty good ones.
The fix is boring and nobody wants to hear it: you have to cap output. max_tokens is not an optional nicety, it's a load-bearing wall. And you have to stream, because a 20-second non-streaming response will time out at the load balancer and now you're debugging a timeout that was actually a slow generation.
Batching is where the money is
Here's the number that matters. GPU inference is monstrously inefficient at batch size 1. The hardware is designed to move matrices. One request at a time leaves most of the silicon idle.
Continuous batching — where the server dynamically packs requests into the running batch as slots free up — is the single biggest lever on cost per token. I've seen 5-10x throughput improvements moving off naive serving. Not from a better model. From a better queue.
Everyone reaches for a bigger model when their bill goes up. Almost nobody checks whether they're serving at batch size 4.
Agents are where this gets genuinely hard
Now multiply all of that by an agent loop.
An agent is not one inference call. It's a loop: call model, get tool call, execute tool, feed result, repeat. That means your latency isn't one generation, it's N generations plus tool time, and the whole thing has to fit inside a user's patience.
Three things go wrong every time.
Retries cascade. Tool call fails, model retries, fails differently, retries again, and now a single user action has burned twelve inference calls and a hundred thousand tokens. Without a hard step budget and a dedup check on tool calls you will discover this from your invoice.
Context grows quadratically in cost. Each turn re-sends the whole history. Turn 20 costs 20x turn 1 in input tokens. You have to be aggressive about truncation, summarization, and caching. Prefix caching where available is not an optimization, it's table stakes.
State lives in the wrong place. If the loop runs in your request handler, a deploy kills every running agent mid-task. Agents need durable state, ideally a workflow engine, so the loop survives the process.
The observability gap nobody budgets for
Traditional APM will tell you the endpoint is 200 OK in 240ms. It will not tell you that the agent looped nine times and one of the tool calls was made twice.
You need four things logged per inference call: token counts in and out, model, latency to first token, total latency. Then per agent run: step count, tool calls with args hash, and terminal state. Without these you are guessing at both cost and quality, and guessing at cost with a variable-cost backend is how a product goes from profitable to not.
So what do you actually build
Cache everything you can. Cap output. Stream. Batch your server. Put a step budget on agents. Log tokens per call from day one. Use a durable workflow for anything that loops.
None of that is novel. It's just ordinary backend discipline applied to a backend that behaves worse than usual. The teams that struggle aren't short on ML talent, they're short on someone who's run a queue in production before.
Everything is just layers.
Top comments (1)
The 'provisioned for requests per second, limited by token throughput' framing is the one teams learn too late — a single verbose generation eating fifty good requests' capacity is exactly why max_tokens is a load-bearing wall, not a nicety. I'd push the agent section one step further: a step budget alone still lets a loop burn tokens on tool-call retries, so the dedup check on tool args matters as much as the cap. Do you log per-call token counts from day one, or did the observability gap only become obvious after the first surprise invoice?