Your AI Returns a 200 OK. That Doesn't Mean It's Right.
A problem I kept hitting while building developer tools, and what I learned trying to solve it.
A few months ago I started noticing something strange in a feature I'd built. The flow was simple: send some text to an LLM, ask for a structured JSON response, use that response in the app. It had been working fine for weeks. Then, with no code changes on my end, a field that used to always be a number started showing up as a string. Nothing crashed. No error in the logs. The API call returned 200 OK every single time. The response just quietly stopped looking the way my code expected it to.
It took me longer than I'd like to admit to figure out what was happening, mostly because I was looking in the wrong place. I kept checking my own code, assuming I'd introduced a bug. I hadn't. The model's output had simply drifted.
Why this is different from a normal API breaking
If you've worked with REST APIs for any length of time, you have a mental model for how things break. A provider deprecates a field, they put it in a changelog, you get a few months' notice, you migrate. Annoying, but predictable. There's a contract, and when the contract changes, there's usually a paper trail.
LLMs don't work like that, and I don't think enough people building with them have fully internalized why.
When you call an AI API and ask for JSON, you're not getting a structured response in the way a traditional API returns one. You're getting text that the model generated, which happens to look like JSON because you asked it to. There's no schema enforcement on the provider's end unless you've gone out of your way to add it yourself. The model is doing its best to satisfy your prompt, and "its best" can shift for reasons you have zero visibility into: a model version update, a subtle change in how it interprets ambiguous input, even just natural variance in generation.
I'd ask the same prompt with the same model and get a slightly different shape back depending on the day. A field would get renamed. An array would occasionally come back as a single object when there was only one item, instead of a list with one item. A model would decide to wrap its JSON in a markdown code fence one time and not the next. None of this is the model being "wrong," exactly. It's just not playing the same enforcement role as a versioned API contract does.
The part that actually worried me
Here's the thing that changed how seriously I took this. The scariest version of this problem isn't a prompt change you made yourself, because at least you'd remember making it and could investigate. The scariest version is when nothing on your end changed at all, and the output still drifted, because the model provider updated something behind the scenes.
You ship a feature. It works. Weeks go by. You're not actively looking at it because it's "done." And then one day a field your downstream code depends on just isn't there in the format you expect, and you find out from a user, or from an error several layers removed from the actual cause, by which point you're debugging backwards trying to figure out when this started and why.
For a feature you control end to end, that's a bad afternoon. For something with real consequences attached, the cost of that gap between "the output changed" and "I noticed the output changed" is the whole problem.
What I ended up building
The fix, once I framed it correctly, was a familiar pattern borrowed from a different problem. If you've ever set up monitoring for a third-party API you depend on, you already know the shape of this solution: define what a healthy response looks like, check periodically that reality still matches the definition, get told immediately when it doesn't.
So that's what I built. You define a JSON Schema describing the shape your AI response should always have, you give it the actual prompt and provider you use in production, and it runs that prompt on a schedule, checking the real output against your schema. The moment something drifts, you get an alert, not a support ticket from a confused user three weeks later.
I also added a version of this that runs synchronously inside a CI/CD pipeline, because there's a second failure mode worth catching separately: a developer (often me) tweaking a prompt to "sound more natural" and accidentally breaking the structure the application depends on. Scheduled monitoring catches drift you didn't cause. A CI check catches drift you're about to cause, before it ships.
What I'd tell someone building with LLM APIs today
If you're returning structured output from an AI model and feeding it directly into application logic, the single most useful thing you can do is stop assuming a 200 response means a correct response. Those are two different claims, and the gap between them is exactly where this kind of bug lives.
You don't need anything elaborate to start. Even just logging the shape of what comes back and eyeballing it occasionally is better than nothing. But if the feature matters enough that a silent structural change would actually hurt — a customer-facing flow, anything billing-adjacent, anything where wrong data propagates somewhere consequential — it's worth treating the AI's output with the same seriousness you'd treat any other external dependency you don't fully control.
I ended up building this into Fixzi, the developer tools product I work on, because it was a problem I genuinely needed solved and couldn't find a simple version of anywhere else. If you're hitting the same thing, I'd be curious to hear how you're currently catching it — or if you're not catching it at all yet, which I suspect is more common than people want to admit.
Top comments (0)