A customer asked us a normal question. They got a reply that looked like a form letter.
It was polite. It was fast. It was completely useless. And nobody got an alert, because nothing failed in a way that looked like failing.
That is the kind of bug I want to talk about.
What we thought was happening
We have an internal helper that answers support mail. The idea is simple. Take the incoming question, add our own notes about the product, ask a model, send the answer.
The notes are not huge. About a page of text. We appended them to every prompt because it was easy, and because the model seemed happier with context.
For a while it worked. Then replies started degrading in a way nobody could name. Less specific. More filler. Shorter.
We blamed the model at first. Then we blamed the prompt. Both were wrong.
What was actually happening
Our consulting endpoint validates the incoming message length. There is a cap: 3000 characters. Anything longer is rejected before it reaches a model.
Our helper never saw the rejection as a rejection. The call came back as HTTP 400, the code treated anything that was not a success as "provider unavailable", and the fallback fired. The fallback is a friendly template. So the customer got a template, and our logs got one quiet line.
The annoying part is that HTTP 400 is not an outage. It is the server telling you the request was wrong. We read it as bad luck and tried again.
We found it by replaying the customer's own message through the helper. That took about a minute. What took days was not believing the evidence, because the logs said the reply had been sent, and it had. It was just the wrong reply.
If you want the shortest version of this story: our monitoring watched the wrong layer. The prompt assembly was broken, the transport was fine, and we were staring at the transport.
Nobody reads the quiet line.
Why this one took so long to find
Three things lined up badly.
The failure was silent by design. We had built a fallback so that users never see a raw error. That is usually a good instinct. Here it meant the symptom and the cause lived in different places.
The error was classified wrong. A length check returning a client error is not the same as a provider being down. Ours went into the same bucket, so the retry logic kept retrying something that could never succeed.
And the input was only sometimes too long. Short questions passed. Long ones with pasted logs did not. So it looked random, and random makes people guess.
What we changed
The validation error now has its own name in our code, and it does not retry. Retrying a too-long message is just spending money to fail again.
We also log the actual length of the message, next to the cap. That one number would have ended the investigation on day one.
And the fallback stopped being silent. When the fallback fires, it says so. A template that pretends to be an answer is worse than an error, tbh.
The part I am still thinking about
We removed the whole-document paste, which fixed the immediate problem. But the underlying shape is still there. Any time we add context to a prompt, we are spending a budget we do not measure.
So now we trim the notes per question, and we check the assembled size before sending. It is a few lines of code. It should have been there from the start.
If you run a helper like this, I would check one thing today. When your fallback fires, does anything actually tell you?
I think most of us have a template quietly answering for us right now, and we just do not know it yet.
If you want the token budget and the trimming rules we settled on, they're here.
Top comments (0)