Michael Brewer

Posted on Sep 11 Originally published at michaelbrewer.me

Case study: smart-proxy

#ai #llm #python #architecture

One endpoint in front of three tiers of inference, so that always-on agents get a frontier-class answer only when they need one.

Summary

I run a fleet of AI agents that generate hundreds of requests an hour, around the clock. Most of those requests are routine: heartbeats, status checks, processing a tool result, classifying a message. Sending all of it to a frontier API was expensive and fragile, and rate limits and credit exhaustion took the agents down at the worst times.

smart-proxy is the routing layer I built to fix that. It is a standard-library Python proxy that presents one OpenAI-compatible endpoint and decides, per request, whether the work goes to a small CPU model, a large local GPU model, or a cheap cloud model. It also polices the cloud tier with a quality gate, because cheap cloud models are capable but unreliable in specific, predictable ways.

Over one measured 40-hour production window it handled 8,129 requests at a 0.17 percent failure rate with no manual intervention. The quality gate passed 95.7 percent of cloud responses on the first attempt, and action requests that used to take 36 to 45 seconds now take 15 to 20.

The problem

Three things were true at once:

The agents never stop. An always-on multi-agent system produces a constant stream of small, cheap-to-answer requests, with an occasional hard one mixed in. Paying frontier prices for the small ones made no sense, and depending on a single external API meant a rate limit anywhere took everything down.
Local hardware is capable but constrained. The inference server is about $2,000 of used enterprise parts: an EPYC 7302P, 128 GB of ECC memory, and two Tesla P40s from 2016. It can run a 27B dense model or an 80B mixture-of-experts model well, but only one large model at a time, with a 20 to 40 second cold start and about 30 seconds to swap.
Cheap cloud models misbehave in specific ways. They narrate an action instead of calling the tool. They invent tool requirements for plain factual questions. They truncate under load. Each of those looks like a successful HTTP 200 to a naive client.

The requirement was one endpoint, all models visible, and the system making the tiering decision itself.

Constraints

Pascal-era GPUs. No Tensor Cores, no vLLM, and most published tuning advice is written for newer cards. The GPU tier had to live on llama.cpp.
One GPU model resident at a time. Any design that assumed concurrent large models was dead on arrival.
The CPU had to be the floor. Whatever ran on CPU had to survive GPU swaps, GPU crashes, and GPU power-off, so the system always had something available.
Production traffic, not a lab. The agents were live users of the proxy throughout. Every change had to be one variable, tested, then frozen with checksums so it could be rolled back in about two minutes.

Options I rejected

A LiteLLM gateway. This is where I started. It handled alias routing and a remote coder target, but the moment I needed a quality gate, per-pool concurrency accounting, and structured request logging, it was easier to own a small proxy than to bend a large one. The lesson I kept was not "avoid proxy layers." It was "don't add layers; let the one layer you have absorb policy."

A GPU model as the judge. Judging whether a cloud response is empty, truncated, or narrating instead of acting is a narrow task. Spending a GPU slot on it would contend with the actual work. A 4B model on CPU does it for free.

Static concurrency caps from synthetic tests. My first caps came from small test requests and did not hold. The vendor's limiter varies by load window, and the payload shape matters. The caps that survived came from stress tests with real gateway-shaped requests, median around 32K tokens, recording admission versus 429 and setting each cap at the worst observed value.

A fallback to "whatever local model is loaded." That serves the wrong model silently. The fallback is now pinned to a named stack and skips itself, with a log line, unless that exact stack is active. A fallback that might serve a different model than intended is worse than no fallback.

What shipped

Three tiers behind one alias map. Two Qwen3-4B models on dedicated CPU cores form the always-on floor: a classifier that triages each request into simple, retrieval, code, or reasoning in 200 to 600 milliseconds, and a helper for summaries, retrieval, tool matching, and fallback judging. The GPU tier is twelve llama.cpp stacks, swapped through the Portainer API with a ten-minute anti-flap cooldown and rollback on a failed swap. The cloud tier is two real vendor pools with measured caps of 6 and 5.

A quality gate with a rubric. Cloud responses pass through deterministic validation of tool-call structure, then an LLM judge, then a retry with structured feedback if either fails. The most important finding was that a judge without explicit criteria hallucinates failures. My first judge prompt asked whether the response "requires tool use," and the model decided every response did. The fix was a three-part rule, the list of tools read from the actual request, and explicit exclusions for knowledge questions, opinions, and greetings.

Pre-classification instead of retries. The retry loop existed because cloud models narrate. Three attempts at about twelve seconds each is where the 36 to 45 second latency came from. Now the CPU helper reads the request against the tool names, spends two to three seconds picking the applicable tool, and the proxy injects a structured nudge naming it. The cloud model produces the tool call on the first attempt. Names-only tool lists at about 40 tokens outperformed described lists at over 200, and temperature zero was mandatory for anything classification-shaped.

Concurrency accounted on the serving pool, not the requested id. One model id was stress-verified as its own pool, then the next day three out of three production requests for it came back answered by a different pool. The vendor had merged them without announcing it. Had I kept the two caps separate, real concurrency on that pool would have been 13 against a measured limit of 5. The proxy now carries a routing map in code that collapses legacy ids onto the pool that serves them, whatever id the client asked for, emits a reroute event and a response header on every reroute, and warns at load if the config documentation disagrees with the code.

Structured request logging. Every request emits start, end, routing, and reroute events keyed by request id. That is how the pool merge was confirmed in live traffic within minutes of deploy: 63 reroutes in a 30-minute window.

The two bugs that made fallbacks work. A transport error path produced an HTTP status of 0, which Python's server happily emitted as an invalid status line and broke the client connection; the fix clamps out-of-range statuses to 502 at the send boundary. And every local fallback was failing with a 400 because Anthropic-format tool-call messages carry null content, which llama.cpp rejects; the translator now emits an empty string, and fallback errors return 502 with the real backend detail attached. Both were invisible until the error propagation was fixed.

Outcome

Measure	Result
Requests in one 40-hour production window	8,129
Failure rate in that window	0.17 percent
Manual interventions in that window	0
Quality gate first-attempt pass rate	95.7 percent
Action-request latency, before	36 to 45 seconds
Action-request latency, after	15 to 20 seconds
Classifier triage latency	200 to 600 milliseconds
Cloud cost on routine traffic	none; handled on CPU

The thesis the numbers support: most agent traffic does not need a frontier model, and small models are most valuable when they steer large ones rather than replace them. A 4B model that can say "this needs the cron tool" in two seconds saves thirty seconds of a cloud model fumbling toward the same conclusion.

What I would do differently

Log structured request events from day one. Every hard debugging story in this project got easy the moment the reroute and routing events existed, and I built them last. And treat every rate limit as a defense rather than a promise: the number is "worst I have observed," re-verifiable any time, never a fact about the vendor.

DEV Community