My extension had a bug that's the worst kind: it looked like it worked.
You'd save a Claude conversation, open it up, and there'd be your question...
For further actions, you may consider blocking this person and/or reporting abuse
This is the kind of bug that makes verification more important than a happy-path demo. The feature worked in the sense that something got saved, but the contract was completeness. For extensions that capture conversations or context, I would want a simple invariant check: saved length, final message marker, and a visible warning when capture is partial.
"The contract was completeness" — that's the whole thing in five words, and it's the framing I was missing while I was busy congratulating myself that something saved at all.
Funny timing: the visible-warning part is exactly what I just added. Another commenter pushed me the same direction, so now on save I check the balance of user vs assistant turns — if it comes back lopsided (say 6 questions, 0 answers), it refuses to save silently and tells the user capture looks partial. Turning the quiet failure into a loud one was the single highest-leverage thing, especially at zero users where nobody's going to file the ticket for me.
The other two invariants you named I haven't done and should. Saved length is easy and I've got no excuse. The final-message marker is the one I actually like most — checking I reached the end of the thread, not just grabbed a middle slice, catches a whole failure mode the balance check misses (I could have every turn in the right ratio and still have stopped scrolling too early). Adding both to the list.
Appreciate you naming it as a contract instead of a feature. That reframe is going to change how I think about the capture path.
The final-message marker is the one I would add first too. Balance checks catch distortion inside the slice, but an end marker catches the silent partial-capture case. That is a different class of bug: the data can look internally consistent and still be missing the part that changes the answer.
Yeah, and here's what I only got once I tried to build it: "did I hit the end of the thread" isn't something I can check on the data I already grabbed. These chat views render lazily — if you save while scrolled up, the last message isn't even in the DOM yet.
So the end marker isn't a check I add after. It forces the capture to scroll all the way down first and confirm it got there. That's the different class you're pointing at: balance runs on whatever slice I happened to have, and a slice can be perfectly balanced and still be the wrong slice.
That is the key detail: the end marker is not just validation, it changes the capture behavior. It forces the collector to prove it reached the live boundary before trusting the slice. Without that, every downstream check is operating on a neat but possibly incomplete story.
Exactly — and "prove it reached the boundary" turned out to have its own trapdoor, because the boundary moves. If the thread is still streaming a reply, or something loads in while I'm scrolling, the bottom I "proved" I hit isn't the bottom anymore by the time I finish. So the collector can't just touch the boundary once; it has to keep checking that the boundary stopped moving before it trusts the slice.
Which is a longer way of agreeing with you: the moment you treat it as behavior instead of a checkbox, the real question stops being "did I reach the end" and becomes "is there still an end being written." Different question, and I only found it by shipping the first version and watching it lie to me again.
Great lesson. Silent failures are far more dangerous than crashes. Building lightweight verification into the workflow beats relying on assumptions every time.
"Lightweight" is the part I'd underline. My first instinct after this was to reach for something heavy — a test suite, a schema, real monitoring — and I'd have spent a week and shipped none of it. What actually stuck was dumb and small: after a save, count the user turns and the assistant turns, and if one side comes back zero, refuse to save silently and say so out loud. Ten lines. It can't tell me the capture is perfect, but it catches the exact failure that started this whole mess. Turning the quietest bug I had into the loudest one was worth more than any amount of clever.
The probe is the right move, but there's a trap hiding in it: you ran it once, confirmed the truth, and hardcoded that truth back into the extension. Which means you're now exactly where you were at 10am — shipping a selector that's correct today and will silently rot the next time Claude ships a redesign. The failure mode is identical (half a conversation, no error), you've just reset the clock on it.
The thing that actually kills the class of bug isn't the probe, it's making the probe's invariant run in production. You already found it:
[data-is-streaming]anddata-testid='user-message'are structurally more durable than.font-claude-messagebecause they describe what the element is, not how it's styled — a class likefont-claude-messageis a styling detail that turns over every redesign, while a testid or a semantic data attribute is something the site's own code depends on. So don't just pick the selector that matches today; add a cheap sanity check on save — if user turns and assistant turns come back wildly unbalanced (4 questions, 0 answers), refuse to save silently and surface it. That turns your worst bug ("quietly wrong") into your loudest one, which is the only version you can actually catch without a user filing a ticket you'll never get at 0 users.You're completely right and this is the sharpest thing anyone's said about it. The probe fixed today's selector and reset the clock — it didn't kill the class of bug. I was quietly proud of the probe and you just showed me it's a one-time flashlight, not a smoke detector.
The balance check is the move I hadn't made and it's obvious in hindsight: I'm already collecting user turns and assistant turns separately, so "4 questions, 0 answers" is right there in the data — I just wasn't looking at it. Refusing to save silently on a wild imbalance turns the one failure mode I can't see (quietly wrong) into the one I can (loud and blocking). And you're right that at 0 users that's the only version I can actually catch, since there's no one to file the ticket. Adding it.
I did already lean toward the durable selectors for the reason you said — data-is-streaming and the testid describe what the element is, class names describe how it looks and turn over every redesign. But picking a sturdier selector just slows the rot; your check is what makes the rot loud when it finally happens. Those are two different problems and I'd conflated them. Thanks for the reframe.
the bug that looks like it works is the worst kind. i lost an hour and a half once to an invisible character i'd pasted by accident, everything looked fine and nothing worked, no error anywhere. at least a crash tells you something's wrong.
Oh the invisible character one is evil. I've done the zero-width space version of that and spent way too long diffing two strings that were byte-for-byte identical on screen. A crash at least points at a line number — the silent ones make you doubt your own eyes. Glad it wasn't just me.
This was such a great read. The quiet bugs are always the worst — they don't crash,
they just silently break your expectations.
I had a similar experience building a system monitor — it worked fine for weeks,
but one day it stopped detecting new processes because of a stale baseline file.
No error, no crash, just quietly blind to everything new.
Your probe script idea is genius. I'm definitely stealing that approach for my
own sanity checks. Thanks for writing this up!
The stale baseline one is brutal because nothing looks wrong — every check passes, it's just passing against a snapshot that quietly stopped being true. Mine at least left a lopsided file you could eyeball. Yours had nothing to see, the reference point itself had rotted out from under it.
And yeah, steal the probe idea. The whole reason it works is it refuses to trust the app when it says "done" and goes and looks for itself. Your monitor probably wanted the same thing pointed at the baseline — something asking "is this still real" instead of just "is it there".
What was the root cause of the bug that saved half of every conversation, was it a serialization issue? I'd love to hear more about how you debugged it.
Good question — and not serialization, which is honestly where my head went first too. It was dumber than that: a stale CSS selector.
These sites have no API, so I read the DOM and match a selector for each message bubble. I had one selector for the user's messages and one for the AI's replies. The user one still matched. The AI one — Claude had quietly changed their markup in some redesign, so my selector for the answers matched zero elements. Questions came through fine, answers silently dropped. Hence: half the conversation, and no error anywhere because "matched nothing" isn't an error, it's just an empty list.
How I found it: I stopped trusting my memory of the markup and wrote a tiny probe I paste into the console on the live page — it prints which selectors match, how many nodes, and a snippet of the text inside. Took about twenty minutes to see the AI selector returning 0 while the user one returned the right count. The real lesson: my memory of "what the selector used to be" is worse than useless, because it's confident and wrong. The live page is the only source of truth.
If you build on a DOM you don't own, write the probe before you write the fix.