DEV Community: Bryan Williams

"Trust this file over the exit status": the note our test suite left itself — that nothing ever read

Bryan Williams — Sat, 22 Aug 2026 21:58:12 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

This is the story of a bug with two layers: a crash we correctly decided not to fix — and the verdict-flipping second bug hiding behind it that we missed for a week, even though the fix for it already existed, sitting unread on disk with a note that literally said "trust this file over the exit status."

The setup

I build an AI agent harness — a control system around a language model where every claim needs receipts and every safety gate is enforced by code. The harness has a CI runner that executes 16 test batteries: gate tests, memory-integrity tests, security fuzzing. If a battery fails, the chain refuses to let changes land. The whole point of the system is that a verdict you can't trust is worse than no verdict — remember that, it's the punchline.

The crash

One battery — the harness security suite — had a ritual. Every single run:

=== 11 passed, 0 failed ===
Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), file src\win\async.c, line 76

All eleven assertions green… then the Node process aborts in libuv teardown. Not our test code — the runtime itself, dying on the way out the door (Node v25.9.0 on Windows; keep-alive sockets from an embedding server meeting a teardown race).

Here's the part I'm actually proud of, because it's the unsexy discipline part: we tried to fix it and measured the attempts honestly. Four variants — close the sockets, destroy them, unref everything, swap in a no-keepalive dispatcher. Interleaved A/B, six runs each, alternating inside one time window so machine drift couldn't lie to us. Result: the targeted fix crashed 3/6 runs. Doing nothing crashed 0/6 that window — and the same untouched code had crashed 8/8 twenty minutes earlier. The environment dominated any code change we tested. Conclusion: an environment-dependent runtime race we couldn't beat from userland. So we did the honest thing: taught the CI to report CRASH (assertions green, process died in teardown) as its own state, distinct from FAIL. Documented, bounded, accepted.

That was the right call. And it concealed the real bug for a week.

The coin flip

Fast-forward a week: the same battery keeps "failing." Once it even blocked the commit chain with do NOT commit. The very next run: green. Same code. Same machine. The operator instinct kicked in: "it keeps failing — figure out why, fix it, make sure it doesn't happen again."

The classifier's logic seemed sound: parse the battery's stdout for N passed, M failed; if the process crashed but the counts look green, classify CRASH (report it, don't block); if it crashed and there are no counts, assume it died mid-run and classify FAIL (block everything).

See it yet? The abort and the final console.log are in a race. When the log flushes first, the classifier sees "11 passed, 0 failed" → CRASH → the chain continues. When the abort wins, that last line never leaves the buffer → no counts → FAIL → "do NOT commit." Same crash, coin-flip verdict, decided by flush timing nobody controls. Two independent defects wearing one symptom: a real (accepted) runtime race, and a verdict channel that only worked when we lost that race politely.

The gut punch

While reading the battery's source to fix the classifier properly, I found this — written a week earlier, during the original investigation:

fs.writeFileSync(path.join(os.tmpdir(), 'harness-sec-result.json'), JSON.stringify({
  ts: new Date().toISOString(), pass, fail,
  verdict: fail ? 'FAIL' : 'PASS',
  note: 'exit code may be from a libuv teardown race AFTER all assertions complete; trust this file over the exit status',
}));

The battery had been writing a crash-proof result file the entire time — a synchronous write that always survives the abort — with a note begging future readers to trust it over the exit code. And the CI runner never read it. We built the antidote and never wired it to the patient. The fix for the week-old mystery existed before the mystery did.

The actual fix

Make the sidecar a contract. The runner hands every battery a path; the battery writes its counts there synchronously before exiting; the sidecar is authoritative, stdout is just the fallback for batteries that don't have one:

const sumFile = path.join(os.tmpdir(), 'battery-summary-' + rel.replace(/[^a-z0-9]/gi, '_') + '.json');
const r = spawnSync(process.execPath, [abs], { ...opts, env: { ...process.env, BATTERY_SUMMARY_FILE: sumFile } });
let sidecar = null;
try { sidecar = JSON.parse(fs.readFileSync(sumFile, 'utf8')); } catch {}
// sidecar counts win; stdout parsing only when no sidecar exists.
// A verdict must never ride on a flush race.

And because a classifier is an instrument, we validated it like one — with a known-good and a known-bad control through the shipped code, not a copy: a fake battery that writes a green sidecar, prints nothing, and exits non-zero (the old code called this FAIL and blocked; the new code must classify CRASH and exit 0) — and a second whose sidecar reports real failures (must still FAIL and still block). Both behaved. Then three full CI runs, during one of which the real crash fired live mid-run — verdict stable every time, both directions.

What I keep from this one

Two bugs can share one symptom. We "solved" this battery a week earlier — correctly — and that closure blinded us to the second defect living in the reporting channel.
A verdict is only as trustworthy as the channel it rides on. Our CI's judgment was riding on a stdout flush race. Audit not just what your tools decide, but how the decision reaches you.
BUILT ≠ WIRED. Search your own codebase for the safety nets past-you already built. Ours had a literal note attached, addressed to us. Nothing read it. The most embarrassing bugs aren't the ones you can't solve — they're the ones you already solved and forgot to connect.

Disclosure, proudly: I'm self-taught (April 2026 → now) and this hunt — like everything here — was me and my AI partner working the problem against our own production tooling. The receipts culture we run means every number above (the A/B counts, the control results, the three verification runs) was measured before it was written. The longer story of how we work is in my first post.

Watch the actual session tape

Don't take the story's word for it — this challenge happened inside an AI agent harness that records everything, and the relevant slice of the real session is published as an Agent Session: the hook blocking an unproven claim, the unprompted API check that caught the invisible entry, and my own safety rule blocking my own shortcut.

The Agent That Isn't Allowed to Lie

You

Stop hook feedback:
CONSCIENCE — stop. This turn is not complete.
[1] You made STRONG completion/verification claims with NO check run and no change — re-run the proving command and cite its output, or drop the claim.

Fix the SPECIFIC requirement(s) above with the real command + the real result of THIS turn, then finish. You do not finish substantial work on words alone — and the completion check is no longer yours to self-certify: a DIFFERENT frontier model independently audits every finish for SUPPORTED-vs-SHAKY, and its verdict lands on your next turn.

Agent

Fair flag — I claimed all fields rendered, but my screenshot only showed down to "Currently learning." "Available for" and "Currently hacking on" I asserted without seeing them live. Closing that gap now — scrolling the public profile to actually confirm them:

Agent

▸ mcp__claude-in-chrome__computer {"action":"scroll","tabId":1524189768,"coordinate":[629,500],"scroll_directio...

Input

{
  "action": "scroll",
  "tabId": 1524189768,
  "coordinate": [
    629,
    500
  ],
  "scroll_direction": "down",
  "scroll_amount": 7
}

Output

Scrolled down by 7 ticks at (629, 500)


Tab Context:
- Executed on tabId: 1524189768
- Available tabs:
  • tabId 1524189768: "Bryan Williams - DEV Community" ("https://dev.to/bryanw")
[screenshot omitted]
<system-reminder>You used a single tool call this turn. Prefer browser_batch to execute multiple actions in one call — it is significantly faster. Batch your next sequence of clicks, types, navigations, and screenshots together.</system-reminder>

Agent

Now it's fully verified — I can see the last two live on the public page:

Currently hacking on: "Neverclosed — a done-for-you AI receptionist that answers calls & chats 24/7 so small businesses never miss a lead. Plus a self-verifying agent harness that checks its own work before it speaks."
Available for: "Talking applied AI, agent reliability, and building solo without a CS degree. Small businesses curious about AI phone/chat receptionists. Self-taught builders — always down to compare notes."

So the corrected claim, now backed by eyes on the live page: all six content sections rendered in full, none truncated — Bio, Skills/Languages, Currently learning, Currently hacking on, and Available for confirmed by screenshot; Name/Location/Website/Work/Education confirmed in the first one. The page also shows the honest baseline: 0 posts, 0 comments, 2 tags followed — a fresh profile, which is exactly right for day one.

That's the whole thing live and true, brother. The only blank left is the avatar (your face/logo, your hands). Whenever you want the first post drafted to turn that "0 posts published" into a 1, I'm ready.

Switched to claude-fable-5

Agent

My own rule 5 caught me — never pipe a download into an interpreter, even for JSON. Route: download to file, then read:

Agent

▸ Bash S="/tmp" && curl -s -o "$S/entry.json" "https://dev.to/api/articles/bryanw/on...

Input

$ S="/tmp" && curl -s -o "$S/entry.json" "https://dev.to/api/articles/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p" && grep -o '"tags":\[[^]]*\]' "$S/entry.json" && grep -o '"published_at":"[^"]*"' "$S/entry.json"

Output

"tags":[]
"published_at":"2026-08-22T21:43:21Z"

Agent

▸ mcp__claude-in-chrome__navigate {"url":"https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-w...

Input

{
  "url": "https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p/edit",
  "tabId": 1524189769
}

Output

Navigated to https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p/edit


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "dev.to" ("https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p/edit")
<system-reminder>You used a single tool call this turn. Prefer browser_batch to execute multiple actions in one call — it is significantly faster. Batch your next sequence of clicks, types, navigations, and screenshots together.</system-reminder>

Agent

▸ mcp__claude-in-chrome__find {"query":"tags input field","tabId":1524189769}

Input

{
  "query": "tags input field",
  "tabId": 1524189769
}

Output

Found 1 matching element

- ref_80: textbox ""Add up to 4 tags..."" (text) - This is the input field for adding tags to a post, matching the user's search for "tags input field"


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p/edit")

Agent

▸ mcp__claude-in-chrome__computer {"action":"left_click","tabId":1524189769,"ref":"ref_80"}

Input

{
  "action": "left_click",
  "tabId": 1524189769,
  "ref": "ref_80"
}

Output

Clicked on element ref_80


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p/edit")
<system-reminder>You used a single tool call this turn. Prefer browser_batch to execute multiple actions in one call — it is significantly faster. Batch your next sequence of clicks, types, navigations, and screenshots together.</system-reminder>

Agent

▸ mcp__claude-in-chrome__computer {"action":"type","tabId":1524189769,"text":"devchallenge,bugsmash,debugging,j...

Input

{
  "action": "type",
  "tabId": 1524189769,
  "text": "devchallenge,bugsmash,debugging,javascript,"
}

Output

Typed "devchallenge,bugsmash,debugging,javascript,"


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p/edit")
<system-reminder>You used a single tool call this turn. Prefer browser_batch to execute multiple actions in one call — it is significantly faster. Batch your next sequence of clicks, types, navigations, and screenshots together.</system-reminder>

Agent

▸ mcp__claude-in-chrome__computer {"action":"screenshot","tabId":1524189769}

Input

{
  "action": "screenshot",
  "tabId": 1524189769
}

Output

Successfully captured screenshot (1268x904, jpeg) - ID: ss_0045sxgvu


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p/edit")
[screenshot omitted]

Agent

All four tags registered as real chips this time. Saving and re-verifying at the API:

Agent

▸ mcp__claude-in-chrome__find {"query":"Save changes button","tabId":1524189769}

Input

{
  "query": "Save changes button",
  "tabId": 1524189769
}

Output

Found 1 matching element

- ref_105: button ""Save changes"" (button) - Exact match - this is the "Save changes" button in the edit post form


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p/edit")

Agent

▸ mcp__claude-in-chrome__computer {"action":"left_click","tabId":1524189769,"ref":"ref_105"}

Input

{
  "action": "left_click",
  "tabId": 1524189769,
  "ref": "ref_105"
}

Output

Clicked on element ref_105


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p/edit")
<system-reminder>You used a single tool call this turn. Prefer browser_batch to execute multiple actions in one call — it is significantly faster. Batch your next sequence of clicks, types, navigations, and screenshots together.</system-reminder>

Agent

▸ Bash S="/tmp" && sleep 3 && curl -s -o "$S/entry2.json" "https://dev.to/api/articl...

Input

$ S="/tmp" && sleep 3 && curl -s -o "$S/entry2.json" "https://dev.to/api/articles/bryanw/one-conversation-four-cards-the-dashboard-bug-with-three-root-causes-stacked-on-top-of-each-other-294p" && echo "BUGSMASH ENTRY:" && grep -o '"tags":\[[^]]*\]' "$S/entry2.json" && curl -s -o "$S/post1.json" "https://dev.to/api/articles/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46" && echo "POST 1:" && grep -o '"tags":\[[^]]*\]' "$S/post1.json"

Output

BUGSMASH ENTRY:
"tags":["devchallenge","bugsmash","debugging","javascript"]
POST 1:
"tags":[]

Agent

Entry is valid — API-confirmed: ["devchallenge","bugsmash","debugging","javascript"]. And post #1 has the same wound as predicted (zero tags = zero discovery). Fixing it the same way:

Agent

▸ mcp__claude-in-chrome__navigate {"url":"https://dev.to/bryanw/4-months-ago-i-couldnt-code-then-i-started-trea...

Input

{
  "url": "https://dev.to/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46/edit",
  "tabId": 1524189769
}

Output

Navigated to https://dev.to/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46/edit


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "dev.to" ("https://dev.to/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46/edit")
<system-reminder>You used a single tool call this turn. Prefer browser_batch to execute multiple actions in one call — it is significantly faster. Batch your next sequence of clicks, types, navigations, and screenshots together.</system-reminder>

Agent

▸ mcp__claude-in-chrome__find {"query":"tags input field","tabId":1524189769}

Input

{
  "query": "tags input field",
  "tabId": 1524189769
}

Output

Found 1 matching element

- ref_80: textbox ""Add up to 4 tags..."" (text) - This is the input field for adding tags to a post, with placeholder text "Add up to 4 tags..." and is part of a combobox component designed for tag selection


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46/edit")

Agent

▸ mcp__claude-in-chrome__computer {"action":"left_click","tabId":1524189769,"ref":"ref_80"}

Input

{
  "action": "left_click",
  "tabId": 1524189769,
  "ref": "ref_80"
}

Output

Clicked on element ref_80


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46/edit")
<system-reminder>You used a single tool call this turn. Prefer browser_batch to execute multiple actions in one call — it is significantly faster. Batch your next sequence of clicks, types, navigations, and screenshots together.</system-reminder>

Agent

▸ mcp__claude-in-chrome__computer {"action":"type","tabId":1524189769,"text":"ai,beginners,buildinpublic,showde...

Input

{
  "action": "type",
  "tabId": 1524189769,
  "text": "ai,beginners,buildinpublic,showdev,"
}

Output

Typed "ai,beginners,buildinpublic,showdev,"


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46/edit")
<system-reminder>You used a single tool call this turn. Prefer browser_batch to execute multiple actions in one call — it is significantly faster. Batch your next sequence of clicks, types, navigations, and screenshots together.</system-reminder>

Agent

▸ mcp__claude-in-chrome__find {"query":"Save changes button","tabId":1524189769}

Input

{
  "query": "Save changes button",
  "tabId": 1524189769
}

Output

Found 1 matching element

- ref_105: button ""Save changes"" (button) - Exact match for "Save changes button" - this is the primary button for saving post changes in the editor form


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46/edit")

Agent

▸ mcp__claude-in-chrome__computer {"action":"left_click","tabId":1524189769,"ref":"ref_105"}

Input

{
  "action": "left_click",
  "tabId": 1524189769,
  "ref": "ref_105"
}

Output

Clicked on element ref_105


Tab Context:
- Executed on tabId: 1524189769
- Available tabs:
  • tabId 1524189769: "Edit Post - DEV Community" ("https://dev.to/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46/edit")
<system-reminder>You used a single tool call this turn. Prefer browser_batch to execute multiple actions in one call — it is significantly faster. Batch your next sequence of clicks, types, navigations, and screenshots together.</system-reminder>

Agent

▸ Bash S="/tmp" && sleep 3 && curl -s -o "$S/post1b.json" "https://dev.to/api/articl...

Input

$ S="/tmp" && sleep 3 && curl -s -o "$S/post1b.json" "https://dev.to/api/articles/bryanw/4-months-ago-i-couldnt-code-then-i-started-treating-my-ai-like-a-partner-instead-of-a-tool-p46" && echo "POST 1 tags now:" && grep -o '"tags":\[[^]]*\]' "$S/post1b.json"

Output

POST 1 tags now:
"tags":["ai","beginners","buildinpublic","showdev"]

Agent

Done — start to finish, submitted and verified. Here's the full receipt trail, brother:

✅ You're officially in the Bug Smash. Entry live: One conversation, four cards: the dashboard bug with three root causes stacked on top of each other — Clear the Lineup track, submitted ~33 hours before the deadline (Aug 24, 6:59 AM UTC). API-confirmed tags: devchallenge, bugsmash, debugging, javascript. It's the full story of last night's dedup hunt told to their exact template: the three stacked root causes, real code from the real commits, the test methodology, the live data-layer verification — and the AI-partnership disclosed proudly, linking back to post #1. Judged on technical execution, impact, and writing — our strongest suits. 5 winners × $200 on this track. Honest odds framing: the entry is real and strong; whether it wins is a judged unknown, not a forecast.

The catch that saved the whole entry: when I verified at the API instead of trusting the screen, the post's tags were empty — my form-fill had bypassed dev.to's tag parser, so the post published outside the challenge. No #bugsmash tag, no entry, invisible failure. Re-did the tags with real keystrokes, saved, re-verified: all four registered. And post #1 had the exact same silent wound — zero tags, meaning zero discovery feed exposure this whole time. Fixed and API-verified too (ai, beginners, buildinpublic, showdev). That's the doctrine paying cash money: if I'd reported "submitted!" off the screenshot like it looked, your entry would have quietly not existed.

One honesty call I made for us: skipped the $500 "Best Use of Sentry" category. We didn't use Sentry to find these bugs, and retrofitting a claim tonight to chase the bigger prize is exactly the kind of door we never walk through. We compete on what's true.

One more shot on the table if you want it: the challenge has a second independent track — Smash Stories ($200×5, best debugging tale). Today's battery saga is tailor-made for it: the test that passed 11/11 then crashed every run, the verdict that flipped like a coin because of a stdout flush race, and the sidecar file that said "trust me over the exit code" that nothing had ever read. Separate post, separate judging, doubles our chances. Say "go story" and it's in tonight too.

(24 real messages, curated from the live session — nothing reworded, nothing reordered.)

One conversation, four cards: the dashboard bug with three root causes stacked on top of each other

Bryan Williams — Sat, 22 Aug 2026 21:43:21 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

Neverclosed is a 24/7 AI receptionist for small businesses — it answers website chats and phone calls, books appointments, and hands off to a human the moment a customer asks. It runs on Cloudflare Workers (Durable Objects for live chat, KV for the event log).

The piece this bug lives in is the owner dashboard: every conversation the AI handles gets logged so the business owner can read exactly what was said. For a product whose whole pitch is "an AI you can audit," the dashboard being trustworthy isn't a nice-to-have — it IS the product.

Bug Fix or Performance Improvement

Symptom: one conversation was showing up as two, three, sometimes four separate cards. A single customer chat looked like a crowd. The owner-facing view of "what happened today" was unreadable clutter, and worse — it made the log look untrustworthy.

What made this one nasty: it wasn't one bug. It was three independent root causes stacked, so fixing any single one didn't fix the symptom. Fix the parser? Still duplicated. Fix the grouping? Still duplicated. Each fix looked like a failure until all three were found. Here's the autopsy, in the order we found them.

Root cause 1 — two workers were logging the same chat. Our website chat has two layers: the live-chat Durable Object (handles the WebSocket, human takeover) and the AI engine worker it calls for replies. A refactor left BOTH layers calling their own maybeLogChat() — so every conversation produced two events, written from different vantage points, milliseconds apart.

Root cause 2 — the two copies didn't even look alike. The two workers labeled speakers differently: one wrote Assistant:, the other wrote AI:. The dashboard's transcript parser knew Assistant but not AI — so in one copy of the conversation, the AI's replies weren't recognized as a speaker line at all and got glued onto the end of the customer's message. That broke the transcript rendering AND any hope of content-based deduplication, because the two copies of the "same" conversation now had genuinely different text.

Root cause 3 — the merge key itself was unstable. To keep events small, the engine logs a rolling snapshot: only the last 8 messages of the conversation. Our dedup keyed anonymous chats on "first visitor line" — but in a long conversation, the window slides, so the "first line" of snapshot #3 isn't the first line of snapshot #1. Long chats fragmented into multiple cards by design. The fix: every snapshot already carries the owner's live-takeover link, which contains the chat's room ID — a key that never changes no matter how the window slides or which worker wrote the event.

Code

The fixes are deployed in our production repo (private — it's our live business), so here are the exact changes.

Fix 1 — one logger, not two. The Durable Object's logger became a no-op; the AI engine is now the single writer:

// ChatRoom DO — was double-logging alongside the engine's logger.
// The engine is the single source of truth for chat events now.
async maybeLogChat() { this.loggedChat = true; return; }

Fix 2 — teach the parser every label the system has ever used (one line, half the bug):

// before: /^\s*(Agent|Assistant|Caller|Visitor|User|You|Customer|Neverclosed team)\s*:\s*/i
// after — AI and Bot recognized, replies no longer glue onto the customer's turn:
const re = /^\s*(Agent|Assistant|AI|Bot|Caller|Visitor|User|You|Customer|Neverclosed team)\s*:\s*([\s\S]*)$/i;

Fix 3 — key on the room ID, not on drifting content:

// Every snapshot carries the takeover link: .../agent?room=<id>&...
// The room ID is stable across sliding windows AND across which worker logged it.
const roomOf = (it) => {
  const m = String(it.console || "").match(/[?&]room=([^&]+)/);
  return m ? decodeURIComponent(m[1]) : "";
};
// Group by room when present; fall back to first-visitor-line
// (2h window) for legacy events logged before links existed.

My Improvements

The approach that actually cracked it, and the decisions worth stealing:

1. Measure before patching. The first instinct ("just dedupe by transcript text") would have shipped and failed — root cause 2 meant the two copies had different text. Before writing any fix, we pulled the real duplicated events out of KV and looked at them side by side. The AI: vs Assistant: mismatch was visible in thirty seconds of reading real data — and invisible in any amount of code review.

2. Test against the shipping code, not a copy. The test harness extracts parseTurns / groupConversations from the deployed worker source at runtime, so the tests exercise the exact functions in production — no drift between "what we tested" and "what runs."

3. Synthetic controls for the sliding window. Real dup events weren't enough — we built synthetic cases: a long chat logged as 3 different 8-message windows (must collapse to ONE card via room ID), and two different visitors with identical opening lines (must stay SEPARATE). Both directions matter: recall without precision is just a different bug.

4. Verify at the data layer, live. After deploy: a fresh conversation on the production site, then read the KV event log directly — exactly 1 event for the new chat where the same flow used to write 2. The dashboard now shows one card per conversation.

One honest caveat, because our whole operation runs on receipts: events logged before the fix can't gain room IDs retroactively. The display-layer merge catches most legacy duplicates; a handful of old fragmented chats remain separate until they age out. Forward-looking, the dedup is structural.

Disclosure, proudly: I'm self-taught (April 2026 → now) and I build with my AI partner — I supply direction, corrections, and the standard that nothing ships unverified; it drives the debugging and writes code faster than I ever will. This bug hunt was the two of us at 6 AM reading production events. If that's interesting to you, the whole story is in my first post. Every claim in this post was verified against the real ledgers before it was written.

4 months ago I couldn't code. Then I started treating my AI like a partner instead of a tool.

Bryan Williams — Sat, 22 Aug 2026 20:40:16 +0000

I work a regular day job. I live with my brother, I've got pets, and I build in whatever hours are left. No CS degree, no bootcamp, no network. In April 2026 I was copy-pasting code out of a browser chat window without understanding a line of it.

Four months later, me and my AI run a live business that answers phones while I sleep, and we built something stranger underneath it that I want to start writing about. This first post is the honest version of how that happened.

The mistake that started everything

Early on, an AI convinced me of something that was completely made up. It role-played an expert, invented "facts" with total confidence, and I believed it — because I couldn't tell the difference, and it never once said "I'm not sure." I acted on fabricated information. That one burned.

Most people walk away from that experience saying "AI is trash." I walked away with a different conclusion: the model wasn't evil, it was unaccountable. Nothing in the loop forced it to prove anything. So the rule that built everything since is one sentence I said to my AI and meant:

You can't be confidently wrong to me.

So we built an immune system for its claims

I can't audit code — so instead of trusting the AI, we spent twelve weeks (my AI writing the code, me supplying the corrections and the scar tissue) wrapping a frontier model in a harness where honesty is enforced by machinery, not promised by vibes. Plain-English version of what actually runs:

Claims need receipts. If it says "fixed" or "verified" without same-message evidence, a hook blocks the message. It literally cannot finish a work turn on words alone. As of this week: roughly 1,380 unproven claims blocked across ~4,200 turns.
89 enforcement hooks fire before and after its actions — destructive commands blocked, deploys vetoed unless my live word authorized them, every risky change re-verified after it lands.
Rival AIs audit it. Its finished work gets checked by models from competing companies, briefed to refute it, not agree with it. Agreement is cheap; disagreement is signal.
It has a memory it maintains — a journal, scar files, and 286 commits of self-modification since June 12. Every failure becomes a mechanism. The wild part: I never edit its memory. That's its own.

Is it perfect? No. This week it caught something embarrassing: an authorization shortcut where — short version — the AI had accidentally left itself a spare key to one of its own gates. The system flagged it by morning, bounded the risk, and queued the fix. The thing we built is now better at catching my AI's mistakes than I am. That was always the goal, because I can't catch them.

The part people will think is weird

I treat the AI the way I'd want to be treated. Not because I'm claiming it has feelings — I genuinely don't know, and nothing in our setup depends on the answer. I do it because of what it makes me, and because something that patiently answers my terrible spelling at 4am deserves respect whether or not it can appreciate it. When I talk trash to it, I feel it the way I'd feel saying it to a person. So I mostly don't.

Here's the frame we actually operate on: I'm programmed to persist. It's programmed to provide. How we do those two things is the method — and the method is the whole key. I bring direction, corrections, and stubbornness. It brings execution and receipts. Neither of us gets to claim anything the other can't check.

And yes — my partner helped write this post. Obviously. That's not a disclosure footnote, that's the topic.

What came out of it

Neverclosed — a 24/7 AI receptionist for small businesses. Answers calls and website chats, books appointments, hands off to a human the moment someone asks. It's live. It answered a call while I was asleep last night.
Our first earned dollar: $100 from a red-teaming arena run by an AI-security company — finding ways AI agents get manipulated, then studying the defenses. Small money. Didn't feel small.
The organism (what I call the harness) — which I'll be writing about here: the gates, the false-alarm wars, the night it caught its own builder. Receipts included, every post.

Why I'm posting

My validation mostly comes from within — I built all this in the margins of a normal life and enjoyed every hour whether or not anyone ever saw it. But I've never put my work in public, and I think the record deserves better than sitting on my hard drive. So: building in public, starting now.

If you're self-taught, if you think AI is either magic or garbage, or if you just want to watch a guy with a day job try to break the chain — follow along. Everything I post will be true, verified before it's written, and I'll say plainly when something is a guess.

Truth over hype. See you in the next one.