Our status page was green for weeks. Generations were failing the whole time.
That one still bugs me. Not because the check was broken in an obvious way. Because it was answering a question nobody needed answered.
What the check actually did
It sent a request with our own API key and looked at the status code. If we got a success back, we painted the dot green.
The thing is, that request never touched a model. It hit an endpoint that reads a list from our database and returns it. Auth works, database is up, dot goes green.
So we were testing one thing and reporting on another. The dot meant "our keys still work". We were reading it as "the product works".
How we noticed
A customer emailed. Not angry, just confused. They said generation had been failing since the morning, and asked whether we knew.
We did not know. We looked at the dashboard. Green. We looked at the graphs. Flat, quiet, boring.
Then we ran an actual generation by hand.
It failed. Fast. An upstream call came back with HTTP 402 and our code turned that into a generic error. Nothing about credit, nothing about which provider. Just a failure.
402 means payment required. In our case the internal account behind one provider had run dry. The check never got near that code path, so it never saw it.
Why "just check the status code" fails
I think this is the most common shape of a bad health check, and it hides well.
A real generation touches a lot of things. Auth. Routing. The upstream provider. Their billing. Their model actually existing at the name we asked for. Token accounting on the way back.
A shallow check touches auth and stops. So the check has a tiny failure surface and the product has a huge one. Any problem in the huge part is invisible, and the dashboard stays calm while the support inbox does not.
Worse, the calm dashboard makes you doubt the customer. You think: our monitoring is fine, maybe they are holding it wrong. That is a bad instinct and it costs you a day.
What we changed
Two things, and neither is clever.
We do a real generation at a fixed interval. Once an hour, one small prompt, one known model, and we assert that text came back. Not that the request succeeded. That there is content in the response.
And we stopped treating the status code as the verdict. We alert on the classes that mean "this will not fix itself": payment required, auth rejected, model not found, rate limited hard. Those are the ones where a human has to do something.
A slow provider is not the same as a dead one, and a dead one is not the same as a misconfigured one. They need different people to look at them.
We also log the upstream name on failure now. That one line would have saved us the whole morning. When something fails, you want to know whose fault it probably is before you open a terminal.
The part that is still uncomfortable
An hourly real generation is a tiny cost, and it is honest. But it is still synthetic traffic. It does not prove a customer's specific model works, with their specific prompt, at their specific size.
So the check tells us the plumbing is alive. It does not tell us the product is good.
I have made peace with that. There is no check that proves the product is good. But there is a big gap between "our keys still work" and "a generation returns text", and we were standing on the wrong side of it.
If your monitor is green right now, when did you last make it fail on purpose?
The check we ended up with, and the model list it runs against, are here.
Top comments (0)