I built a gateway that puts four Chinese model providers behind one OpenAI-compatible endpoint. Not because it is clever. Because wiring up four vendors by hand got old fast, and every vendor describes the same model differently.
Here is what actually broke.
Timeouts are not one thing. Our first version used one 90-second timeout for the whole request. That is wrong the moment you have a fallback. If provider A hangs for 80 seconds and then fails, your fallback gets 10 seconds and dies too. Now each attempt gets its own deadline, and the caller gets a clear 504 instead of a mystery 500.
Empty 200s are worse than errors. A provider can return HTTP 200 with an empty body. Our health check counted that as success for a week. Not sure how nobody noticed, tbh. Now we parse the body, and an empty completion counts as a failure.
Model lists drift between deploys. A model silently disappeared from one vendor's models endpoint. Our check passed because it only verified auth. It never verified the model still existed. Now we run one real generation per provider per hour and alert on 4xx, not on a clean status.
Retries make cost attribution lie. When a request fails over, two providers did work but only one gets billed for the response. If you reconcile on response count you will be wrong. We log the attempt, then the cost.
Was any of this hard to design? No. It was hard to notice. Does your fallback path even test the same thing as your primary?
If you want to see the model list and the routing rules this turned into, it's here.
Top comments (0)