AI Application Engineer · Node.js & PHP · LLM tool calling, MCP, SSE & payments. I build WooCommerce speed fixes, Stripe billing and AI features at jeemmo.com
"Every caller was told the write succeeded" is the scariest kind of bug. I had a quieter version: a sitemap index with stale lastmod dates meant search engines skipped about 102k URLs for weeks, while every job and health check reported green. Since then I check output counts against expected counts (rows in, rows out, URLs indexed) rather than trusting success flags. Counting the rows, like you did, is what catches it.
Yeah, quite a realistic problem for when I used to deal with search index generation. Some pages we expected to change when they didn’t change, we may know the index generation or refresh is broken somewhere.
AI Application Engineer · Node.js & PHP · LLM tool calling, MCP, SSE & payments. I build WooCommerce speed fixes, Stripe billing and AI features at jeemmo.com
Exactly. "It should have changed and didn't" is the one test that catches it. Since then I check the count of pages in the sitemap, not just whether the job finished.
“Success” needs to be defined at the delivery boundary, not at the point a daemon accepts work. I would separate accepted, persisted, delivered, and reconciled counters, then alert on the gaps between them. That turns silent loss into an observable invariant rather than an anecdote after the fact.
Thank you, accepted, persisted, delivered, reconciled is a cleaner vocabulary than the one I was using, so I am stealing it. Mapping it to this story: accepted is the hook handing the message off, persisted is the append landing on disk, delivered is what the store says it holds, reconciled is the O(n) sweep re-verifying the chain. The bug lived entirely in the accepted-to-persisted gap, and the success flag only ever measured accepted. The cruel part is that the sweep was the reconciled counter, but it re-read the whole file while holding the lock the hot path needed, so the invariant detector became the thing creating the gap. Alerting on the deltas between the four is the version that catches it in minutes.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
Rows in vs rows out saved me too. The next check I added was a floor on the output: if a run that normally gives hundreds comes back with a handful, treat it as failed even when every call returned fine. The success flag never saw the problem but the count did
Thank you, the floor check is the complement to counting rows. In my case every caller got a success back and the daemon logged nothing, so the only signal was 9,016 against 10,000. A floor like "a run that normally gives hundreds came back with a handful, treat as failed" would have caught it a whole soak earlier than the row count did. The nasty detail was that the success flag was technically accurate and still meaningless, since send() returned False and nobody read it. Did you set the floor at a fixed number, or as a percentage of the trailing runs?
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. kevin@outricher.com
Expected vs actual counts is the check that works. Same story on lead lists: for an ordinary sales ICP, more than half of the matches could not answer, they had left the role or had no mailbox, and nothing reported a problem. I counted that share before writing a word of copy (my post): dev.to/kevinhall/243-of-417-leads-...
Product leader and systems analyst in fintech and edtech. Founder of AnalystCraft. Writing about requirements, acceptance criteria, agent evaluation, and Test-Driven Product Development.
Could you separate blocked appends, timed-out send attempts and records missing at the end in the soak report? The 20 × 48 estimate counts work arriving during lock holds, but those appends need not all be lost: some can drain after the lock releases. Also, the post says both that callers were told success and that send() returned False and the hook ignored it. Showing the result at each boundary would make the loss mechanism much easier to reproduce than inferring drops from lock duration alone.
Yes — and you've caught something I glossed. The "20 × 48" is an estimate of work arriving during lock holds, and not all of it is necessarily lost: some of those appends drain after the lock releases, so it's an upper bound on loss, not a count of it. I shouldn't have stated the loss from lock duration alone. The report needs three separate outcomes at the boundaries you name — append blocked, send timed out, present at end — so the drop is measured, not inferred. And you're right that the post is loose on the other point: "callers were told success" and "send() returned False and the hook ignored it" describe two different call paths, and collapsing them hides which one actually ate the 984. I'll add the per-boundary counters and disambiguate the two paths.
We're a place where coders share, stay up-to-date and grow their careers.
"Every caller was told the write succeeded" is the scariest kind of bug. I had a quieter version: a sitemap index with stale lastmod dates meant search engines skipped about 102k URLs for weeks, while every job and health check reported green. Since then I check output counts against expected counts (rows in, rows out, URLs indexed) rather than trusting success flags. Counting the rows, like you did, is what catches it.
Yeah, quite a realistic problem for when I used to deal with search index generation. Some pages we expected to change when they didn’t change, we may know the index generation or refresh is broken somewhere.
Exactly. "It should have changed and didn't" is the one test that catches it. Since then I check the count of pages in the sitemap, not just whether the job finished.
“Success” needs to be defined at the delivery boundary, not at the point a daemon accepts work. I would separate accepted, persisted, delivered, and reconciled counters, then alert on the gaps between them. That turns silent loss into an observable invariant rather than an anecdote after the fact.
Rows in vs rows out saved me too. The next check I added was a floor on the output: if a run that normally gives hundreds comes back with a handful, treat it as failed even when every call returned fine. The success flag never saw the problem but the count did
Expected vs actual counts is the check that works. Same story on lead lists: for an ordinary sales ICP, more than half of the matches could not answer, they had left the role or had no mailbox, and nothing reported a problem. I counted that share before writing a word of copy (my post): dev.to/kevinhall/243-of-417-leads-...
Exactly, I read your article on ICP. Very interesting idea.
Could you separate blocked appends, timed-out send attempts and records missing at the end in the soak report? The 20 × 48 estimate counts work arriving during lock holds, but those appends need not all be lost: some can drain after the lock releases. Also, the post says both that callers were told success and that send() returned False and the hook ignored it. Showing the result at each boundary would make the loss mechanism much easier to reproduce than inferring drops from lock duration alone.
Yes — and you've caught something I glossed. The "20 × 48" is an estimate of work arriving during lock holds, and not all of it is necessarily lost: some of those appends drain after the lock releases, so it's an upper bound on loss, not a count of it. I shouldn't have stated the loss from lock duration alone. The report needs three separate outcomes at the boundaries you name — append blocked, send timed out, present at end — so the drop is measured, not inferred. And you're right that the post is loose on the other point: "callers were told success" and "send() returned False and the hook ignored it" describe two different call paths, and collapsing them hides which one actually ate the 984. I'll add the per-boundary counters and disambiguate the two paths.