My extension had a bug that's the worst kind: it looked like it worked.
You'd save a Claude conversation, open it up, and there'd be your questions — every single one of them — and none of Claude's answers. Half the conversation, gone, but neatly. No error, no crash. Just quietly wrong.
I want to walk through the day I spent chasing this, because it taught me something I keep having to relearn.
Some background. My extension reads AI chat pages — ChatGPT, Claude, Gemini, Perplexity — and pulls the conversation out so you can save it. None of these sites give you an API for that. So you're reading the raw HTML, hunting for the CSS selector that wraps each message. And those selectors are not documented anywhere, because they were never meant for you. They change whenever the site ships a redesign.
Here's the mistake I made, more than once: I guessed.
I'd think, okay, Claude's assistant messages are probably still .font-claude-message, I remember that from a while back. Ship it. And it'd be wrong, because they'd quietly changed it, and my selector matched exactly zero elements. The user-message selector still worked, so the extension happily grabbed all the user turns and none of the assistant ones. Hence: half a conversation.
I guessed for Gemini too. Wrong there as well, just in the opposite direction — I caught the AI answers and dropped the questions.
At some point I got tired of being wrong and did the obvious thing I should've done first: I wrote a little probe script. Forty lines. You paste it into the browser console on the actual page, logged into your actual account, and it just tells you the truth — which selectors match, how many elements each one hits, and a snippet of the text inside so you can tell whether it grabbed a real message or the navigation bar.
The output was almost funny in how clear it was. For Claude:
- ✅ 4 [data-testid='user-message'] — sample: "人工智能对未来有什么影响" (my actual question: "what's AI's impact on the future")
- ✅ 4 [data-is-streaming] — sample: "Claude responded: 人工智能对未来的影响是多方面的…"
- ❌ 0 .font-claude-message
There it was. Zero. The thing I'd been confidently shipping matched nothing. The real selector for an answer was an attribute I wouldn't have guessed in a hundred years.
I ran the probe on all four sites. Twenty minutes of pasting a script into a console did what a week of guessing couldn't. ChatGPT and Gemini turned out to already be fine. Claude and Perplexity were both wrong, in different ways, for different reasons.
Then the images.
Someone's going to ask "can it save pictures too?" so I looked into it, and this is where it gets humbling. On Gemini, one image in the conversation had a normal https link — fine, savable, might expire but savable. The next image in the same chat was a blob: URL. If you've never run into these: a blob URL is a reference to something sitting in the browser's memory for that one tab, right now. Close the tab and it's gone. There is no version of my extension, or anyone's, that can save that image, because the moment you leave the page the thing it points to stops existing.
So I had a choice. Pretend, and save a link that will 100% be a broken image later. Or be honest and write, right there in the saved text, "this image couldn't be saved — go look at the original." I went with honest. A broken image icon is worse than a sentence telling you the truth.
Claude does something different again — the uploaded file lives outside the message element entirely, so I can't grab the image, but I can grab the filename. So at least the saved note says "attachment: dog.jpeg" instead of pretending nothing was ever there.
A couple of other things broke that day, for the record, because it wasn't only selectors:
- The copy button did nothing. Turned out the panel runs in an iframe, and the browser blocks the clipboard API inside iframes unless the parent hands over permission. One attribute fixed it. Took me an embarrassing while to find.
- "Extension context invalidated" errors everywhere — which just means I reloaded the extension while an old page still had the old code running in it. Harmless, but it looks alarming in the error log, so I made it fail quietly.
Here's the thing I keep relearning, and the reason I'm writing this down so I maybe stop forgetting it:
I cannot guess what the page looks like. I have a memory of what a selector used to be, and that memory is worse than useless, because it's confident and wrong. The page in front of the user is the only source of truth. A forty-line script that reads that page beats my best guess every single time.
Still 0 users, still $0. But the extension now actually saves the whole conversation on all four sites, which it demonstrably did not do this morning. I'll take it.
If you build anything that scrapes a page you don't control: write the probe first. Don't be me at 10am.
— building NotebookBloom in public, #2
Top comments (27)
This is the kind of bug that makes verification more important than a happy-path demo. The feature worked in the sense that something got saved, but the contract was completeness. For extensions that capture conversations or context, I would want a simple invariant check: saved length, final message marker, and a visible warning when capture is partial.
"The contract was completeness" — that's the whole thing in five words, and it's the framing I was missing while I was busy congratulating myself that something saved at all.
Funny timing: the visible-warning part is exactly what I just added. Another commenter pushed me the same direction, so now on save I check the balance of user vs assistant turns — if it comes back lopsided (say 6 questions, 0 answers), it refuses to save silently and tells the user capture looks partial. Turning the quiet failure into a loud one was the single highest-leverage thing, especially at zero users where nobody's going to file the ticket for me.
The other two invariants you named I haven't done and should. Saved length is easy and I've got no excuse. The final-message marker is the one I actually like most — checking I reached the end of the thread, not just grabbed a middle slice, catches a whole failure mode the balance check misses (I could have every turn in the right ratio and still have stopped scrolling too early). Adding both to the list.
Appreciate you naming it as a contract instead of a feature. That reframe is going to change how I think about the capture path.
The final-message marker is the one I would add first too. Balance checks catch distortion inside the slice, but an end marker catches the silent partial-capture case. That is a different class of bug: the data can look internally consistent and still be missing the part that changes the answer.
Yeah, and here's what I only got once I tried to build it: "did I hit the end of the thread" isn't something I can check on the data I already grabbed. These chat views render lazily — if you save while scrolled up, the last message isn't even in the DOM yet.
So the end marker isn't a check I add after. It forces the capture to scroll all the way down first and confirm it got there. That's the different class you're pointing at: balance runs on whatever slice I happened to have, and a slice can be perfectly balanced and still be the wrong slice.
That is the key detail: the end marker is not just validation, it changes the capture behavior. It forces the collector to prove it reached the live boundary before trusting the slice. Without that, every downstream check is operating on a neat but possibly incomplete story.
Exactly — and "prove it reached the boundary" turned out to have its own trapdoor, because the boundary moves. If the thread is still streaming a reply, or something loads in while I'm scrolling, the bottom I "proved" I hit isn't the bottom anymore by the time I finish. So the collector can't just touch the boundary once; it has to keep checking that the boundary stopped moving before it trusts the slice.
Which is a longer way of agreeing with you: the moment you treat it as behavior instead of a checkbox, the real question stops being "did I reach the end" and becomes "is there still an end being written." Different question, and I only found it by shipping the first version and watching it lie to me again.
That “boundary stopped moving” phrasing is the real requirement. A one-time bottom check proves almost nothing in a live surface. The collector needs a stability condition, otherwise it can be perfectly correct about a page state that no longer exists.
"A stability condition" is the right name for it, and the annoying part is there's no clean definition of stable. In practice all I've got is a proxy: no new nodes and no height change for some quiet window, then call it settled. Which works until the proxy gets fooled — a slow token stream or a stalled fetch looks identical to "done" for a second or two, so if my window is too short I trust a pause and cut early.
So I ended up where you'd expect: the window is a tradeoff, not a fact. Too short and I snapshot a page that's still being written; too long and every save drags. There's no threshold that's correct, just one that's wrong less often. Which is a weirdly honest place to land for a feature that started as "just save the chat."
Yes, that is the annoying part. The harder bug is not the first broken save, it is proving the boundary will stay boring across sync, reloads, and long conversations. A good extension needs tests for drift, not only tests for the happy path.
That proxy problem is familiar. I tend to treat settled as a bundle of signals, not one timer: no DOM growth, no pending network, no active stream marker, and a second read that matches the first. Still imperfect, but it makes early-cut failures much easier to explain.
Bundle-of-signals is the right model, and the fun part is auditing which of yours I can actually get from outside the app. Two of them I can read cleanly: DOM growth (mutation observer) and second-read-matches-first (already doing that). The stream marker I get almost for free, because it's the same attribute I key off to grab the message in the first place — on Claude the assistant node carries data-is-streaming, so the exact thing that tells me "this is an answer" also tells me "it's still being written." One attribute, two jobs.
The one I can't get honestly is pending network. My content script lives in an isolated world, so I can't see the page's own fetches without injecting into its realm and patching fetch/XHR, which is a lot of surface area to babysit on a page I don't control. So in practice I'm settling on three signals and treating the missing fourth as a known blind spot, which — to your point — is at least a blind spot I can name out loud when a capture cuts early, instead of shrugging at a timer that happened to fire.
And yeah, the drift-vs-happy-path thing from your last one is the deeper cut. Sync and reload are exactly where a "settled" check that passed once quietly stops being true.
That is a solid outside-the-app audit set. DOM growth plus a stable second read catches a surprising amount, and the streaming attribute gives you a nice explicit boundary instead of guessing from timing.
"Explicit boundary instead of guessing from timing" is exactly why I grabbed at the streaming attribute the moment I noticed it — a real signal beats a timer every time. The honest asterisk is that not every site hands me one. ChatGPT and Claude both expose something I can key off, but not all four do, and on the ones that don't I'm back to inferring "is it still writing" from DOM growth alone, which is a fuzzier version of the same question.
So it's ended up tiered rather than uniform: use the explicit boundary where the site gives me one, fall back to the timing-plus-stability combo where it doesn't, and label which path a given capture took in the manifest so the weaker ones aren't quietly wearing the same badge as the strong ones. Not elegant, but it's honest about its own confidence, which is more than the first version could say.
This whole thread genuinely reshaped how I think about the capture path — went from "did it save" to "can it prove what it saved." Thanks for staying in it this long.
That tiered approach is realistic. Explicit site signals where available, DOM stability where they are not, and a second-read audit for the cases where timing can lie. The important part is admitting which tier you are on instead of pretending the boundary is equally reliable everywhere.
"Admitting which tier you're on" is the part I want to keep, and it's bigger than this one feature. The whole reason the original bug was so nasty is that it presented a half-capture with the exact same confidence as a full one — no tier, no asterisk, just "here's your conversation." Everything we've talked through basically comes down to making the tool own up to how sure it actually is, per save, out loud.
Funny where that lands for a solo thing with zero users: the honesty isn't a nice-to-have I'll add later, it's the actual product. Anyone can dump the DOM to a file. The part worth paying for is the one that tells you when not to trust the dump.
Anyway — this thread turned a debugging war story into a design principle I'm going to carry into the rest of the extension. Genuinely appreciate you going this many rounds on it.
Yes. The bug is not only partial capture; it is partial capture with full-confidence presentation. Once the system exposes the tier, the user can decide whether the artifact is good enough or whether it needs a stronger capture path.
"Partial capture with full-confidence presentation" — that's the whole bug in one line, and it's the confidence half that was doing the real damage the entire time. A half-capture that announced itself as a half-capture would've been a minor annoyance. It was the swagger on top of it that turned it into a trap.
And exposing the tier does something I didn't fully clock until you put it this way: it doesn't just make the tool honest, it moves the decision to the person who actually knows the stakes. I have no idea whether a given save is "the one turn that mattered" or a throwaway — but the user does. So the right move isn't for me to guess how good is good enough; it's to show my confidence and let them make that call. The bug was me quietly making that decision for them and getting it wrong.
That's the note I'm ending this on, and it's a better one than I started with. Thanks for grinding it all the way down to the actual principle — this thread genuinely changed how I'll build the rest of it.
Great lesson. Silent failures are far more dangerous than crashes. Building lightweight verification into the workflow beats relying on assumptions every time.
"Lightweight" is the part I'd underline. My first instinct after this was to reach for something heavy — a test suite, a schema, real monitoring — and I'd have spent a week and shipped none of it. What actually stuck was dumb and small: after a save, count the user turns and the assistant turns, and if one side comes back zero, refuse to save silently and say so out loud. Ten lines. It can't tell me the capture is perfect, but it catches the exact failure that started this whole mess. Turning the quietest bug I had into the loudest one was worth more than any amount of clever.
The probe is the right move, but there's a trap hiding in it: you ran it once, confirmed the truth, and hardcoded that truth back into the extension. Which means you're now exactly where you were at 10am — shipping a selector that's correct today and will silently rot the next time Claude ships a redesign. The failure mode is identical (half a conversation, no error), you've just reset the clock on it.
The thing that actually kills the class of bug isn't the probe, it's making the probe's invariant run in production. You already found it:
[data-is-streaming]anddata-testid='user-message'are structurally more durable than.font-claude-messagebecause they describe what the element is, not how it's styled — a class likefont-claude-messageis a styling detail that turns over every redesign, while a testid or a semantic data attribute is something the site's own code depends on. So don't just pick the selector that matches today; add a cheap sanity check on save — if user turns and assistant turns come back wildly unbalanced (4 questions, 0 answers), refuse to save silently and surface it. That turns your worst bug ("quietly wrong") into your loudest one, which is the only version you can actually catch without a user filing a ticket you'll never get at 0 users.You're completely right and this is the sharpest thing anyone's said about it. The probe fixed today's selector and reset the clock — it didn't kill the class of bug. I was quietly proud of the probe and you just showed me it's a one-time flashlight, not a smoke detector.
The balance check is the move I hadn't made and it's obvious in hindsight: I'm already collecting user turns and assistant turns separately, so "4 questions, 0 answers" is right there in the data — I just wasn't looking at it. Refusing to save silently on a wild imbalance turns the one failure mode I can't see (quietly wrong) into the one I can (loud and blocking). And you're right that at 0 users that's the only version I can actually catch, since there's no one to file the ticket. Adding it.
I did already lean toward the durable selectors for the reason you said — data-is-streaming and the testid describe what the element is, class names describe how it looks and turn over every redesign. But picking a sturdier selector just slows the rot; your check is what makes the rot loud when it finally happens. Those are two different problems and I'd conflated them. Thanks for the reframe.
the bug that looks like it works is the worst kind. i lost an hour and a half once to an invisible character i'd pasted by accident, everything looked fine and nothing worked, no error anywhere. at least a crash tells you something's wrong.
Oh the invisible character one is evil. I've done the zero-width space version of that and spent way too long diffing two strings that were byte-for-byte identical on screen. A crash at least points at a line number — the silent ones make you doubt your own eyes. Glad it wasn't just me.
This was such a great read. The quiet bugs are always the worst — they don't crash,
they just silently break your expectations.
I had a similar experience building a system monitor — it worked fine for weeks,
but one day it stopped detecting new processes because of a stale baseline file.
No error, no crash, just quietly blind to everything new.
Your probe script idea is genius. I'm definitely stealing that approach for my
own sanity checks. Thanks for writing this up!
The stale baseline one is brutal because nothing looks wrong — every check passes, it's just passing against a snapshot that quietly stopped being true. Mine at least left a lopsided file you could eyeball. Yours had nothing to see, the reference point itself had rotted out from under it.
And yeah, steal the probe idea. The whole reason it works is it refuses to trust the app when it says "done" and goes and looks for itself. Your monitor probably wanted the same thing pointed at the baseline — something asking "is this still real" instead of just "is it there".
What was the root cause of the bug that saved half of every conversation, was it a serialization issue? I'd love to hear more about how you debugged it.
Good question — and not serialization, which is honestly where my head went first too. It was dumber than that: a stale CSS selector.
These sites have no API, so I read the DOM and match a selector for each message bubble. I had one selector for the user's messages and one for the AI's replies. The user one still matched. The AI one — Claude had quietly changed their markup in some redesign, so my selector for the answers matched zero elements. Questions came through fine, answers silently dropped. Hence: half the conversation, and no error anywhere because "matched nothing" isn't an error, it's just an empty list.
How I found it: I stopped trusting my memory of the markup and wrote a tiny probe I paste into the console on the live page — it prints which selectors match, how many nodes, and a snippet of the text inside. Took about twenty minutes to see the AI selector returning 0 while the user one returned the right count. The real lesson: my memory of "what the selector used to be" is worse than useless, because it's confident and wrong. The live page is the only source of truth.
If you build on a DOM you don't own, write the probe before you write the fix.