I shipped a feature in May. Cross-page session memory for a retrieval agent — follow a link, the conversation comes with you. It was deployed, tested, and running in production.
Yesterday I found out no visitor had ever been able to reach it.
The part that was actually broken
The obvious problem was that the chat widget existed on exactly one page out of thirty-nine. That is bad, but it is the kind of bad you notice.
The real problem was underneath. The widget never sent a session_id field. Or a scope field. The server had both, documented, tested, deployed — and the browser had simply never been wired to send either one.
So the feature was not "confined to one page." It was never invoked from anywhere. A sessions_active counter that had read 0 since May, on an endpoint that was working perfectly.
Both halves passed their own tests. The server tests posted a session id, because the test author knew the field existed. The widget tests asserted the chat rendered and sent a message. Nothing tested the seam, and the seam was the feature.
What that class of bug looks like
The failure has a shape worth naming: two components, each correct, and an integration nobody owns.
It does not throw. It does not log. The health endpoint is green. In my case the endpoint even reported the exact number that would have given it away — sessions_active: 0 — sitting in a JSON response I read regularly and never questioned, because zero active sessions on a low-traffic site is completely plausible.
That is the tell, and it generalises past my code: a metric whose broken value is indistinguishable from its healthy value tells you nothing. Zero sessions means "nobody is using it" and "it is impossible to use it," and the number is identical.
Proving the fix, rather than asserting it
Once the browser sent a session id, I did not want a green test. I wanted the thing a visitor does.
So: send one question with a session id from the datasets page. Then send a second question from the portfolio page with the client history deliberately empty — which is what a real navigation does to a JavaScript variable — and see whether the answer still knows what we were talking about.
page 1 /datasets/ "What speech datasets are there?"
page 2 /portfolio/ "Who recorded it?" history: []
The second answer came back about the speech datasets. The server had carried it. That is a test I trust, because the thing I removed — the client's memory — is exactly the thing the feature claims to replace.
The bug I nearly shipped while fixing it
Rolling the widget out, I decided each page should tell the agent which section of the corpus it was on, so the datasets page ranks dataset material first.
Then I tested the datasets page with a piece of internal vocabulary — a codename from our own research notes — and it answered "the documents do not contain information about that." I read that as a coverage failure and widened the page's search to the whole corpus.
That was wrong, and the correction came from the person who owns the product rather than from me. Those codenames are internal research nomenclature for how a training corpus was built. They are not products. Nothing under those names is for sale. They appear zero times on the sales page and only in an engineering report.
The narrow scope was not a limitation. It was the mechanism keeping internal research vocabulary out of a commercial answer. By widening it I had connected a buyer's licensing question to the research corpus, in the name of fixing a defect that was the system working correctly.
Reverted. The lesson is cheap to state and I clearly needed to learn it expensively: do not test a sales surface with internal vocabulary, and do not read a narrow result as a broken one until you know what the narrowness is for.
What I would take from this
Ship a feature and then, before believing it works, ask what a visitor would have to do to reach it — out loud, as a sequence of clicks. Mine had no answer. There was no page to navigate from.
And look at your health metrics for one specific property: is there any value this number could report that means "broken" but reads as "quiet"? Those are the numbers that let a feature sit dead in production for three months while everything stays green.
The agent this is about is live: the architecture and its 32-case evaluation gate are documented at tts.bedvibe.studio/portfolio/agent-architecture, the eval results at tts.bedvibe.studio/portfolio/agent-evals, and the widget itself is on every page of tts.bedvibe.studio now — including, finally, more than one.
If you have found a feature of your own that was deployed and unreachable, I would like to hear how you noticed. In my case it was luck, and I would prefer a method.
Top comments (0)