The inference endpoint is not the model

Ira Long — Sun, 01 Nov 2026 15:02:47 +0000

Serving a model is a systems problem wearing a machine-learning costume.

The model is maybe 20% of the endpoint. The other 80% is queueing, timeouts, batching, retries, and the failure mode where a 2-second p99 quietly turns into a 30-second one because the batch got too large.

Three things I now do by default:

Bound the batch by time, not just size. A batch that waits for 8 items is a batch that waits forever at night.
Separate the queue from the HTTP request. Return a job id, let the client poll. It makes backpressure explicit instead of mysterious.
Add a fallback that returns a safe default after the deadline. A slower answer is usually better than a hung request.

The model accuracy curve is the interesting part of the job. The p99 is the part that wakes you up.

Curious how other people handle the fallback: cached response, heuristic, or just an error?

Reading code beats reading docs, sometimes

Ira Long — Thu, 29 Oct 2026 02:20:08 +0000

I used to open the docs first for every unfamiliar library. Now I start with the source of the one function I actually need, and read docs only when the source surprises me.

Faster for small honest libraries, much worse for big frameworks with generated code. Knowing which one you're looking at is most of the skill.

The day I stopped being afraid of the source view, debugging got easier.

DEV Community: Ira Long

The inference endpoint is not the model

Reading code beats reading docs, sometimes