The inference endpoint is not the model

Serving a model is a systems problem wearing a machine-learning costume.

The model is maybe 20% of the endpoint. The other 80% is queueing, timeouts, batching, retries, and the failure mode where a 2-second p99 quietly turns into a 30-second one because the batch got too large.

Three things I now do by default:

1. Bound the batch by time, not just size. A batch that waits for 8 items is a batch that waits forever at night.
2. Separate the queue from the HTTP request. Return a job id, let the client poll. It makes backpressure explicit instead of mysterious.
3. Add a fallback that returns a safe default after the deadline. A slower answer is usually better than a hung request.

The model accuracy curve is the interesting part of the job. The p99 is the part that wakes you up.

Curious how other people handle the fallback: cached response, heuristic, or just an error?