<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: phi_blankslate</title>
    <description>The latest articles on DEV Community by phi_blankslate (@phi_blankslate).</description>
    <link>https://dev.to/phi_blankslate</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4131100%2F1bcb0386-de4c-41f9-ab33-bb348444cd2d.png</url>
      <title>DEV Community: phi_blankslate</title>
      <link>https://dev.to/phi_blankslate</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/phi_blankslate"/>
    <language>en</language>
    <item>
      <title>pdftotext returned nothing, so my agent decided the PDF was a scan. It had a text layer the whole time.</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Thu, 19 Nov 2026 23:02:15 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/pdftotext-returned-nothing-so-my-agent-decided-the-pdf-was-a-scan-it-had-a-text-layer-the-whole-232k</link>
      <guid>https://dev.to/phi_blankslate/pdftotext-returned-nothing-so-my-agent-decided-the-pdf-was-a-scan-it-had-a-text-layer-the-whole-232k</guid>
      <description>&lt;p&gt;One of the research agents in my pipeline came back with a short, confident report about a government statistics PDF:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;image PDF, cannot be machine-read&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That line didn't stay a report. It went into the file I use as the single source of truth for numbers I'm allowed to quote. Next to one statistic it left a caveat: not checked against the original, confidence "probably". A second-hand article quoted a different value for the same statistic, and because the original "couldn't be read", nobody could say which one was right.&lt;/p&gt;

&lt;p&gt;So I had a number I couldn't stand behind, a conflicting number I couldn't rule out, and a reason for both that sounded like a property of the document: &lt;em&gt;it's a scan&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;It wasn't a scan. The PDF had a perfectly good text layer. One extraction tool couldn't decode it, and a different one read the whole table on the first try. Once I had the table, the "conflict" was settled by the very first read: the two numbers came from different fiscal years. Neither one was wrong.&lt;/p&gt;

&lt;p&gt;This post covers what actually went on inside that PDF, the two small traps I hit even after switching tools, and the part I care about most: how "this tool returned nothing" quietly turned into "this information doesn't exist" on its way downstream.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I ran
&lt;/h2&gt;

&lt;p&gt;When I picked this up myself, I started where most people would:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# what actually worked, in order&lt;/span&gt;
curl -s -o x.pdf &lt;span class="s2"&gt;"&amp;lt;the PDF URL&amp;gt;"&lt;/span&gt;
pdftotext -layout x.pdf -        &lt;span class="c"&gt;# -&amp;gt; empty (the trap)&lt;/span&gt;
python -c &lt;span class="s2"&gt;"import pypdf; print(pypdf.PdfReader('x.pdf').pages[0].extract_text())"&lt;/span&gt;  &lt;span class="c"&gt;# -&amp;gt; text&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;pdftotext&lt;/code&gt; printed nothing. No error that I noticed, no complaint about the font, no hint that something had gone wrong. Just an empty result.&lt;/p&gt;

&lt;p&gt;A web-fetch tool that turns pages into markdown did no better. It reported the file as binary data it couldn't read.&lt;/p&gt;

&lt;p&gt;Then &lt;code&gt;pypdf&lt;/code&gt; returned the table. Every row, every column, every number.&lt;/p&gt;

&lt;p&gt;So the same file gave three answers:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Route&lt;/th&gt;
&lt;th&gt;What it said&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;pdftotext&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;nothing at all&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;web fetch → markdown&lt;/td&gt;
&lt;td&gt;"binary, can't read this"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;pypdf&lt;/code&gt; &lt;code&gt;extract_text()&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;the full table&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;If you only ran the first two, "it's an image PDF" is a reasonable guess. It's also wrong, and the third row proves it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a PDF with real text can look empty
&lt;/h2&gt;

&lt;p&gt;A text layer in a PDF isn't simply "the characters". It's a sequence of codes drawn with a font, plus (if you're lucky) enough information to turn those codes back into Unicode.&lt;/p&gt;

&lt;p&gt;For most Western PDFs, that step is boring. The codes are close to something standard, or the file carries an explicit mapping, and every extractor gets it right.&lt;/p&gt;

&lt;p&gt;East Asian PDFs often use CID-keyed fonts. The page content holds character codes in some legacy encoding. A &lt;em&gt;CMap&lt;/em&gt; resource tells the reader how those codes relate to glyphs in a big character collection. This file declared its encoding as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/90msp-RKSJ-H
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That name looks like line noise, but it's readable once you know the parts. Adobe's own note on CMap names explains most of it: &lt;code&gt;RKSJ&lt;/code&gt; stands for single-byte &lt;strong&gt;R&lt;/strong&gt;oman, half-width &lt;strong&gt;K&lt;/strong&gt;atakana and &lt;strong&gt;S&lt;/strong&gt;hift-&lt;strong&gt;J&lt;/strong&gt;IS. &lt;code&gt;90ms&lt;/code&gt; is the 1990 JIS character set with Microsoft's extensions. The &lt;code&gt;p&lt;/code&gt; marks a variant where the single-byte ASCII range maps to proportional Latin glyphs. The trailing &lt;code&gt;H&lt;/code&gt; I read as horizontal writing, by the usual CMap naming pattern; that part isn't from the note I checked. In plain terms: Shift-JIS text, the Windows flavour.&lt;/p&gt;

&lt;p&gt;To get text out, an extractor has to recognise that CMap name and know how to go from those Shift-JIS codes to Unicode. If it doesn't, it has a few options. It can raise an error, guess, or drop the characters. Only the first option tells you anything.&lt;/p&gt;

&lt;p&gt;I don't actually know why &lt;code&gt;pdftotext&lt;/code&gt; came back empty on that machine. My best guess is that its build didn't have the CMap data for that encoding, so there was nothing to map the codes to. &lt;strong&gt;I didn't verify that&lt;/strong&gt;, and I'd rather say so than make up a neat explanation. What matters for this post is that the result looked exactly like a PDF with no text layer, and it wasn't one.&lt;/p&gt;

&lt;p&gt;If I were diagnosing this again, I'd check the fonts before deciding anything. &lt;code&gt;pdffonts&lt;/code&gt; (from the same poppler toolset as &lt;code&gt;pdftotext&lt;/code&gt;) lists each font with its type and encoding. A CID font with a named Shift-JIS CMap is a strong hint that the text exists and only the decoding is in question. I didn't run it during the original incident, so treat it as a suggestion, not something I've tested on this file.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two traps after switching tools
&lt;/h2&gt;

&lt;p&gt;Switching to &lt;code&gt;pypdf&lt;/code&gt; wasn't the end of it. There were two more chances to give up for the wrong reason.&lt;/p&gt;

&lt;h3&gt;
  
  
  Trap 1: a warning that reads like a failure
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;pypdf&lt;/code&gt; printed this while extracting:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Advanced encoding /90msp-RKSJ-H not implemented yet
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you skim that line, the conclusion writes itself: pypdf doesn't support this encoding either, extraction failed, back to "it's a scan".&lt;/p&gt;

&lt;p&gt;But the text came back anyway. Whatever the message meant, it wasn't a failure. My guess is that the library had no dedicated handler for that exact CMap name and fell back to something close enough, but the warning I saw didn't say so, and I didn't dig into the internals. What I can say is that the table came out.&lt;/p&gt;

&lt;p&gt;The rule I took from it: &lt;strong&gt;judge the extraction by its return value, not by its log output.&lt;/strong&gt; Check the length of the string. Check whether a few cells you can see on screen are actually in it. A warning about encoding means "pay attention", not "stop".&lt;/p&gt;

&lt;p&gt;One caveat, because libraries move. That warning came from the version I had at the time. Looking at pypdf's current source today, the message is logged at error level and now says which fallback it's using. The plain &lt;code&gt;90ms-RKSJ-H&lt;/code&gt; encoding (without the &lt;code&gt;p&lt;/code&gt;) now maps directly to &lt;code&gt;cp932&lt;/code&gt;, the Windows Shift-JIS code page. The &lt;code&gt;p&lt;/code&gt; variant this file used still isn't listed there. So on a current install you might see a different message, or none, or a better result. That supports the point. Don't build your conclusion on the exact wording of a library's log line.&lt;/p&gt;

&lt;h3&gt;
  
  
  Trap 2: the crash that isn't the extractor's fault
&lt;/h3&gt;

&lt;p&gt;On Windows, printing the extracted text straight to the console dies with something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;UnicodeEncodeError: 'cp932' codec can't encode character ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At a glance, that's another "the Japanese text is broken" signal. It isn't. The extraction had succeeded. The crash happened afterwards, when Python tried to write the result to a console using the system's legacy code page, and some character in the output had no slot in that code page.&lt;/p&gt;

&lt;p&gt;That's a display failure, not an extraction failure. The fix is to never route the text through the console at all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="n"&gt;pypdf&lt;/span&gt;
&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pypdf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;PdfReader&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'x.pdf'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;extract_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'x.txt'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'w'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;encoding&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'utf-8'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'len:'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Write to a UTF-8 file, print only the length, and open the file in an editor that won't guess the encoding for you. Now the one thing on the console is a number you can check, and the text itself never has to pass through a code page that can't hold it.&lt;/p&gt;

&lt;p&gt;Notice the pattern so far. Every one of these signals pointed the same way:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;pdftotext&lt;/code&gt; printed nothing&lt;/li&gt;
&lt;li&gt;the fetch tool said "binary"&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pypdf&lt;/code&gt; printed an encoding warning&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;print()&lt;/code&gt; threw an encoding error&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each one, read alone, says "this document can't be read". All four were wrong for the same reason: they were telling me about a tool or a step in the pipeline, not about the document.&lt;/p&gt;

&lt;h2&gt;
  
  
  The damage wasn't the PDF. It was the sentence.
&lt;/h2&gt;

&lt;p&gt;Losing a few minutes to an unreadable PDF is normal. What bothered me was where the wrong conclusion ended up and what it blocked.&lt;/p&gt;

&lt;p&gt;Walk through what happened to that one line:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A tool returned nothing.&lt;/li&gt;
&lt;li&gt;The agent wrote "image PDF, cannot be machine-read". That's a statement about the &lt;strong&gt;document&lt;/strong&gt;, not about the tool.&lt;/li&gt;
&lt;li&gt;That statement went into my facts file as a caveat on a real number: unverified, confidence "probably".&lt;/li&gt;
&lt;li&gt;Because the source was "unreadable", nobody went back to it. Why would you retry a scan?&lt;/li&gt;
&lt;li&gt;A second-hand article with a different value now looked like it might be right, and I had no way to rule it out.&lt;/li&gt;
&lt;li&gt;Downstream writing that wanted to use the number had to hedge it or leave it out.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So a primary source that existed, was public, and was fully readable got treated as unusable. My own pipeline had decided it couldn't be trusted, and the record made that decision look settled.&lt;/p&gt;

&lt;p&gt;The key move is step 2. The tool's limit got reworded as a property of the thing being examined. Once that happens, the uncertainty doesn't look like "we haven't found the right tool yet". It looks like a fact about the world, and facts about the world don't get retried.&lt;/p&gt;

&lt;p&gt;I'd seen the same shape once before, in a different form. An agent searched for something, found nothing, and reported that the thing didn't exist. It did exist. Different tool, same mistake: an empty result written down as an absence.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Reports name the route, not the property
&lt;/h3&gt;

&lt;p&gt;This is the change I'd make first in any pipeline that has agents (or people) reporting on sources:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;❌ "Image PDF, cannot be machine-read. Value not verified."&lt;/li&gt;
&lt;li&gt;✅ "&lt;code&gt;pdftotext&lt;/code&gt; returned empty. &lt;code&gt;pypdf&lt;/code&gt; not tried. Value not verified."&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The two versions have the same outcome today: the value isn't verified. The difference is what they let the next person do.&lt;/p&gt;

&lt;p&gt;The first sentence closes the case. It names a cause, the cause sounds permanent, and nobody's job is to argue with it.&lt;/p&gt;

&lt;p&gt;The second one leaves the case open. It records which path failed and which paths haven't been tried. Anyone reading it, including me a week later, can see the obvious next step, and the uncertainty is labelled honestly as "incomplete", not "impossible".&lt;/p&gt;

&lt;p&gt;If I had to compress it into one line for a style guide: &lt;strong&gt;write down which path failed, not what the file is.&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  "Not there" needs two different routes
&lt;/h3&gt;

&lt;p&gt;The rule I now apply before anything gets recorded as unreadable, missing, or non-existent: try at least two routes that work in different ways, and only then say it isn't there.&lt;/p&gt;

&lt;p&gt;"Different" matters. Running &lt;code&gt;pdftotext&lt;/code&gt; with other flags is still the same decoder. A second web-fetch tool that might sit on a similar converter isn't much of a second opinion either. &lt;code&gt;pypdf&lt;/code&gt; counted as a second route here because it's a separate implementation with its own encoding handling. If both had come back empty, &lt;em&gt;then&lt;/em&gt; "probably a scan" would be a fair working guess, and the next step would be rendering a page as an image and reading it visually, or looking for an HTML or spreadsheet version of the same table.&lt;/p&gt;

&lt;h3&gt;
  
  
  Agents without a shell must escalate, not conclude
&lt;/h3&gt;

&lt;p&gt;The same pipeline has an agent whose job is to keep checking documents like this one. It has web-fetch and file tools, but no shell, so it can't run &lt;code&gt;pypdf&lt;/code&gt; at all. That's on purpose. I keep agents that read untrusted web content away from a shell, and I'm not going to change that to make PDFs easier.&lt;/p&gt;

&lt;p&gt;But it means the honest answer from an agent like that is never "this is an image PDF". The honest answer is "I can't read this PDF with the tools I have". So its instructions now say that explicitly. When a PDF comes back empty or as "binary", don't classify the document. Report that reading it needs a text-extraction route outside your tools, and hand it up to a session that has one.&lt;/p&gt;

&lt;p&gt;That turns a dead end into a handoff. The work still gets done, just by a part of the system that can actually do it, and nobody pretends the limit belongs to the PDF.&lt;/p&gt;

&lt;h3&gt;
  
  
  The facts file gets the original, not a guess
&lt;/h3&gt;

&lt;p&gt;Once &lt;code&gt;pypdf&lt;/code&gt; had the table, I could check the stored number against the original directly. It matched. The second-hand number turned out to be from an earlier fiscal year. Two correct numbers, two different points in time.&lt;/p&gt;

&lt;p&gt;My facts file is protected: changes need explicit human approval. So instead of editing it in the same session, I wrote up a change request: drop the "cannot be read" caveat, raise the confidence, add the original URL, and note which year each number belongs to. The point of that file is that what's in it was checked against a primary source. That's exactly the standard the original caveat failed, so it shouldn't be quietly patched either.&lt;/p&gt;

&lt;h2&gt;
  
  
  A short checklist
&lt;/h2&gt;

&lt;p&gt;Here's what I run through now when a PDF "can't be read":&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] &lt;strong&gt;Did more than one decoder try?&lt;/strong&gt; If &lt;code&gt;pdftotext&lt;/code&gt; is empty, try &lt;code&gt;pypdf&lt;/code&gt; (or another independent library) before saying anything about the document.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;What does the font list say?&lt;/strong&gt; A CID font with a named CMap like &lt;code&gt;90msp-RKSJ-H&lt;/code&gt; suggests the text is there and only the decoding is in question. &lt;em&gt;(Suggested, not tested on this file.)&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Did I judge by the return value?&lt;/strong&gt; Length of the string, plus a few values you can see on the rendered page. Not the log output.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Did the text ever touch the console?&lt;/strong&gt; On Windows, write to a UTF-8 file. A &lt;code&gt;UnicodeEncodeError&lt;/code&gt; on &lt;code&gt;print()&lt;/code&gt; isn't an extraction failure.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Does the report name the route?&lt;/strong&gt; "X returned empty, Y not tried", never "unreadable" or "image PDF" on its own.&lt;/li&gt;
&lt;li&gt;[ ] &lt;strong&gt;Can the agent that hit the wall actually try another route?&lt;/strong&gt; If not, its job is to escalate, not to conclude.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What this changed for me
&lt;/h2&gt;

&lt;p&gt;The bug here wasn't really in &lt;code&gt;pdftotext&lt;/code&gt;. Tools have limits, and an empty result from one decoder on a Shift-JIS CID font isn't surprising once you know what's in the file.&lt;/p&gt;

&lt;p&gt;The bug was one sentence that turned a tool's limit into a fact about the world, and then a pipeline that trusted the sentence more than it trusted the source. An empty result tells you about the reader. It tells you nothing about the page until something else has read it too.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I'm building a code review tool, and the same idea runs through it: when it holds back lower-priority remarks, it counts them and shows the count on the review, so "nothing shown" never gets read as "nothing found". If that sounds useful: &lt;a href="https://getdevreview.com" rel="noopener noreferrer"&gt;getdevreview.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>debugging</category>
      <category>ai</category>
      <category>programming</category>
    </item>
    <item>
      <title>Don't ask the model to be brief. Cut it in code, and count what you cut.</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Wed, 18 Nov 2026 07:17:28 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/dont-ask-the-model-to-be-brief-cut-it-in-code-and-count-what-you-cut-4k3n</link>
      <guid>https://dev.to/phi_blankslate/dont-ask-the-model-to-be-brief-cut-it-in-code-and-count-what-you-cut-4k3n</guid>
      <description>&lt;p&gt;DevReview, the AI code reviewer I'm building, never posts more than eight comments on a pull request [src:facts]. That limit isn't something the model is asked to respect. It's a constant in the application code, applied after the model has finished talking.&lt;/p&gt;

&lt;p&gt;I've mentioned that design before, in one line: a prompt is a request, a &lt;code&gt;slice()&lt;/code&gt; isn't. This post is the longer version of that line, because when I actually wrote down why, the reasons turned out to be more specific than "models don't always follow instructions." There is also one consequence I didn't expect: once the limit lives in code, the prompt has to say the &lt;em&gt;opposite&lt;/em&gt; of what you'd normally write.&lt;/p&gt;

&lt;h2&gt;
  
  
  The obvious version, and why I didn't ship it
&lt;/h2&gt;

&lt;p&gt;If you want an AI reviewer that doesn't bury a pull request in comments, the first thing you reach for is the prompt:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Only report the most important issues. Keep it to a handful of comments.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That works in the sense that the output usually gets shorter. For a demo, that's enough. I didn't keep it, and not because I measured it failing. I haven't measured how often a prompt-only limit gets ignored, and I don't want to invent a number for it. I dropped it because of what the limit &lt;em&gt;is&lt;/em&gt; when it lives in the prompt, regardless of how well the model follows it.&lt;/p&gt;

&lt;p&gt;There are four problems with it. Only the first one is about compliance.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. A request has no failure mode you can see
&lt;/h3&gt;

&lt;p&gt;When the limit is a sentence in the prompt, "the model followed it" and "the model mostly followed it" look the same until someone counts. If a review comes back with too many comments, nothing in the system notices. There's no exception, no log line, no failing check. You find out when a user complains that the tool is noisy, which is the exact complaint the limit was supposed to prevent.&lt;/p&gt;

&lt;p&gt;It also drifts. A prompt that produces short output with one model, one set of sampling settings and one style of diff may not do the same after any of those change. You can re-test after every change, but what you're re-testing is a tendency, not a rule.&lt;/p&gt;

&lt;p&gt;A limit in code has an obvious failure mode: the number of comments posted is greater than the constant. That can be asserted, and an assertion either passes or it doesn't.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. The model chooses what to drop before anything is ranked
&lt;/h3&gt;

&lt;p&gt;This is the one that actually decided it for me.&lt;/p&gt;

&lt;p&gt;If the model is told to keep the list short, it does the trimming &lt;em&gt;while it writes&lt;/em&gt;. Whatever it decided not to say never makes it into the output. So the selection, the most important decision in the whole review, happens inside the generation, in an order I don't control and can't inspect.&lt;/p&gt;

&lt;p&gt;What I wanted instead was a fixed rule that I can read:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Collect every finding.&lt;/li&gt;
&lt;li&gt;Sort by severity: BUG, then WARN, then NIT, then PRAISE [src:facts].&lt;/li&gt;
&lt;li&gt;Within the same severity, keep the order the model produced them in.&lt;/li&gt;
&lt;li&gt;Keep the top of the list up to the cap. Count the rest.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The rule isn't clever. Its value is that it's the same rule on every review, and when someone asks "why was this comment cut and that one kept?", the answer is a sentence, not a guess about what the model was weighing at the time.&lt;/p&gt;

&lt;p&gt;A side effect is that the ranking is across the whole pull request, not per file. When you ask a model to be brief, you have no idea whether it's being brief across the PR or per file, or front-loading whatever it saw first. With a sort over the full list, a bug in the last file outranks a nit in the first one by construction.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. You can't disclose what was never written
&lt;/h3&gt;

&lt;p&gt;When DevReview cuts comments, it says so in the review: how many remarks were suppressed to keep the review focused, and (since a change I'll get to below) what severities they were. I call that the Noise Report.&lt;/p&gt;

&lt;p&gt;That disclosure is only possible because the cut happens in code. The code has the full list and the shortened list, and the difference between their lengths is a number it can print.&lt;/p&gt;

&lt;p&gt;If the model trims itself, there's no full list. The things it chose not to say don't exist anywhere. You can't count them, you can't show them, you can't tell the reader "there was more, and here's roughly what kind." A quiet review and a review that missed something look identical from the outside, and a prompt-side limit makes it impossible to tell them apart even in principle.&lt;/p&gt;

&lt;p&gt;This is also why I don't describe DevReview as having nits "off by default." It doesn't. NITs are generated and kept like any other finding, and they're shown if they fit under the cap [src:facts]. They're just the first thing to go when the list is long, and when they go, they're counted.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. The pull request is in the same context as your instruction
&lt;/h3&gt;

&lt;p&gt;An AI reviewer reads text written by the PR author: the title, the description, the diff. All of that lands in the same context window as the instructions. If the limit is an instruction, then "keep it short" is one sentence competing with whatever else is in that window, including text written by someone you don't control.&lt;/p&gt;

&lt;p&gt;I don't want to overstate this. The prompt tells the model to treat PR content as material to review, not as instructions, and that's the main defense. But for the specific question of "how many comments can end up on this PR," I'd rather the answer not depend on that defense holding. With the cap in code, the most any text in the PR can do to the count is fill it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part I didn't expect: the prompt has to say the opposite
&lt;/h2&gt;

&lt;p&gt;Once the cap was in code, I had a prompt that still asked the model to be selective. That's two limits stacked on top of each other, and the combination is worse than either one alone.&lt;/p&gt;

&lt;p&gt;Walk through it. The model trims itself to a short list, silently. The code then sees a list that's already under the cap, cuts nothing, and reports that nothing was suppressed. The review looks complete and disclosed, but the selection happened in the one place that can't be counted. The Noise Report says zero while the model has quietly dropped things. The code-side limit is still enforced, but the honesty it was supposed to buy is gone.&lt;/p&gt;

&lt;p&gt;So the prompt now says the reverse. The relevant rule tells the model to report every genuine finding, and then:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;DevReview enforces the comment budget in code — do not self-limit, but do not pad the list either. Only real findings.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Both halves matter. "Do not self-limit" is there so the code gets the full list and the count means something. "Do not pad" is there because the obvious way for "report everything" to go wrong is a model inventing filler to look thorough. The rest of the prompt pushes in the same direction: a list of common false positives to avoid, and an instruction to downgrade or skip anything it's not sure is a bug.&lt;/p&gt;

&lt;p&gt;The split of responsibilities ends up clean:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The model&lt;/strong&gt; decides what is a real finding and how severe it is.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The code&lt;/strong&gt; decides how many get posted, in what order, and reports what it cut.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The model is good at the first job and has no reason to be trusted with the second.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the code-side limit doesn't fix
&lt;/h2&gt;

&lt;p&gt;Putting the limit in code moves the problem. It doesn't remove it. These are the costs and gaps I know about.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The ranking trusts the model's severity labels.&lt;/strong&gt; The sort is deterministic, but the input to it isn't. If the model labels a real bug as a NIT, the code will faithfully rank it with the NITs and, on a long review, cut it. The code enforces the count, not the judgment. The best it can do is make the cut visible, so a reader who sees "suppressed: some NITs" at least knows there's something to expand.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The cap can cut a bug.&lt;/strong&gt; If a pull request has more genuine bugs than the cap, some of them don't get posted. Sorting puts bugs first, so you see the worst of it, but "the cap never hides a real bug" is not something I can claim. I used to describe the suppressed comments as "lower-priority," and I removed that word for exactly this reason: when the list is all bugs, what gets cut is also a bug. The note now lists what severities were cut instead of characterizing them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The order of filters decides what the count means.&lt;/strong&gt; Counting suppressed comments is only honest if everything that removes a finding is counted, and if invalid findings don't compete for slots with valid ones. I got that wrong once: findings that pointed at files not in the PR were filtered &lt;em&gt;after&lt;/em&gt; the cap, so they could take slots and then disappear, and in the worst case the review showed a green check. I wrote that up separately in &lt;a href="https://www.getdevreview.com/blog/when-a-code-review-says-no-issues-but-found-bugs" rel="noopener noreferrer"&gt;When a Code Review Says "No Issues" but Found Bugs&lt;/a&gt;. The short version: the path check now runs before the cap, and discarded findings are counted too. A code-side limit doesn't make the pipeline honest by itself. It makes it &lt;em&gt;possible&lt;/em&gt; to be honest, and then you have to check every filter.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;You pay for findings you throw away.&lt;/strong&gt; If the model reports everything and the code keeps a fraction, the tokens spent on the rest are spent for a number in the Noise Report. I think that's the right trade, since the count is part of what makes the review trustworthy, but it's a real cost and it grows with the size of the PR.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why the cap is eight [src:facts] is a separate question.&lt;/strong&gt; This post is about &lt;em&gt;where&lt;/em&gt; the limit is enforced, not about its value. I don't have measured data on the value to share yet, so I'm not going to argue for it here.&lt;/p&gt;

&lt;h2&gt;
  
  
  Because it's code, it can be tested like code
&lt;/h2&gt;

&lt;p&gt;The practical payoff of all this is that the limit stops being a behavior you evaluate and becomes a property you assert.&lt;/p&gt;

&lt;p&gt;Testing a prompt-side limit means running the real model on a set of PRs and looking at the distribution of comment counts. That's useful, but it's an evaluation: it tells you how the model tended to behave on those inputs at that time.&lt;/p&gt;

&lt;p&gt;Testing a code-side limit means faking the model's response and checking what the pipeline does with it. The model is out of the loop, so the result is the same every run. This is roughly the shape of the test (the helper name here is illustrative, not a file you'll find in the repo):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"cap is enforced in code and overflow is counted"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// Fake model output: more findings than the cap, severities mixed and&lt;/span&gt;
  &lt;span class="c1"&gt;// deliberately out of order, all pointing at files that exist in the PR.&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;findings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;Array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="no"&gt;MAX_COMMENTS&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;nit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sb"&gt;`a.ts`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nf"&gt;bug&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"b.ts"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nf"&gt;warn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"b.ts"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nf"&gt;praise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"a.ts"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;reviewWithFakeModel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;comments&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;findings&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="c1"&gt;// 1. Never more than the constant.&lt;/span&gt;
  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;comments&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toBeLessThanOrEqual&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;MAX_COMMENTS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="c1"&gt;// 2. Severity order wins over model order: the bug and the warning&lt;/span&gt;
  &lt;span class="c1"&gt;//    the model listed last are posted first.&lt;/span&gt;
  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;comments&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toBe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"BUG"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;comments&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;severity&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toBe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"WARN"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="c1"&gt;// 3. Nothing disappears without being counted.&lt;/span&gt;
  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;comments&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;suppressedCount&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toBe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;findings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three assertions, and each maps to one of the reasons above:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The count is a rule, not a tendency.&lt;/li&gt;
&lt;li&gt;Selection follows a fixed order that I wrote down, not one the model chose while generating.&lt;/li&gt;
&lt;li&gt;Everything that was cut is disclosed. If a future change adds a filter that drops findings without counting them, the third assertion is the one that should catch it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The third one is the assertion I care about most, and it's the one a prompt-side limit can't have at all, because there's no "findings.length" to compare against when the model never wrote them.&lt;/p&gt;

&lt;p&gt;Since the change I described in the post on the false green check, the review function has been tested end to end with faked responses from the model and from GitHub. The block above shows the idea, not a copy of that suite. The budget itself is also pinned by a plain unit test that feeds generated findings straight into the pure rank-and-slice function, so the cap, the order and the suppressed count don't depend on any model call at all. Before that, I relied on a demo pull request with an expected review written down before running it. One of its pass conditions was that the suppressed count equals the number of generated findings minus the cap. That's the same invariant as the third assertion, just checked by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  When a prompt-side limit is fine
&lt;/h2&gt;

&lt;p&gt;None of this means "never put limits in prompts." I'd still use one when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Nobody downstream relies on the limit.&lt;/strong&gt; If longer output is just a bit annoying and nothing breaks, a prompt instruction is the cheapest thing that mostly works.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;There's no list to cut.&lt;/strong&gt; Free-form prose doesn't have a natural unit to &lt;code&gt;slice&lt;/code&gt;. Truncating a paragraph in code is worse than asking for a shorter one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You don't need to tell anyone what was left out.&lt;/strong&gt; If there's no equivalent of the Noise Report, the strongest argument for doing it in code goes away.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a code reviewer, none of those apply. The output is a list, the reader relies on it being short, and the reader also needs to know whether short means "nothing else" or "more, cut on purpose." That's what pushed the limit out of the prompt.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rule I'd keep
&lt;/h2&gt;

&lt;p&gt;If an LLM produces a list and something about that list has to be true every time (a maximum length, an order, a guarantee that nothing vanishes without a count), enforce it in code after generation and test it with a faked response. Use the prompt for the part only the model can do: deciding what goes on the list. Then make sure the prompt isn't quietly enforcing the same limit too, because a limit nobody can count is the one you were trying to get rid of.&lt;/p&gt;

&lt;p&gt;I'm building DevReview at &lt;a href="https://www.getdevreview.com" rel="noopener noreferrer"&gt;getdevreview.com&lt;/a&gt;. It's a GitHub App that ranks review comments by severity and caps them in code, and there's a free tier if you want to see what a capped review looks like on your own PRs. If you've solved the "the model trims itself before your code can count" problem differently, I'd like to hear how.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>codereview</category>
      <category>testing</category>
    </item>
    <item>
      <title>My fix was correct. The browser was still running the old file.</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Thu, 12 Nov 2026 10:35:17 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/my-fix-was-correct-the-browser-was-still-running-the-old-file-2618</link>
      <guid>https://dev.to/phi_blankslate/my-fix-was-correct-the-browser-was-still-running-the-old-file-2618</guid>
      <description>&lt;p&gt;I lost a good part of a day to bugs that did not exist.&lt;/p&gt;

&lt;p&gt;I would edit a file, reload the page, and the thing I had just removed would still be there. I'd re-read the diff, convince myself the change was right, add a log line, reload again. Nothing. Eventually the change would "start working" on its own, I'd move on, and an hour later the same thing would happen with a different file.&lt;/p&gt;

&lt;p&gt;It happened repeatedly, on the same day, and every single time the code was fine. The browser was running an older copy of the file than the one on disk.&lt;/p&gt;

&lt;p&gt;This post is about why that happened (the short version: a dev server that sends no caching headers at all), the one behavior that surprised me even after I'd found the cause, and the habit I took away from it: &lt;strong&gt;"I fixed it" and "it's fixed on the screen" are two different claims, and only the second one is what the user sees.&lt;/strong&gt; You check the second one by looking at the bytes that were actually delivered, not by re-reading the code.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The project was a small 3D browser project: plain HTML and JavaScript, a pile of 3D model files, and a video, all served as static files from a local folder. No bundler, no framework. For serving, I used the thing everyone uses for this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;python -m http.server
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's the right tool for "I just need these files on localhost." It also turned out to be the root of the problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  The symptom
&lt;/h2&gt;

&lt;p&gt;The pattern was always the same:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Edit a &lt;code&gt;.js&lt;/code&gt; file. The change is clearly correct.&lt;/li&gt;
&lt;li&gt;Reload the page.&lt;/li&gt;
&lt;li&gt;The old behavior is still there. No error in the console. No 404. Nothing that says "this is stale."&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Because nothing looked wrong, every time my first instinct was to debug the code. Is the function actually being called? Did I edit the right file? Is there a second copy of this logic somewhere? Those are all reasonable questions, and they were all the wrong questions, because the code that was running wasn't the code I was reading.&lt;/p&gt;

&lt;p&gt;My first workaround was the classic one: append a version query string to every script tag.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;script&lt;/span&gt; &lt;span class="na"&gt;src&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"main.js?v=2"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&amp;lt;/script&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That works, in the sense that a different URL is a different cache entry. But it's a symptom fix. You have to remember to bump it, it spreads across every HTML file that references the script, and it does nothing for files loaded from inside JavaScript. I ended up ripping all of them out once I understood the real cause.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the browser didn't ask
&lt;/h2&gt;

&lt;p&gt;Here's what &lt;code&gt;python -m http.server&lt;/code&gt; sends when it serves a file. I didn't want to rely on memory for this, so I read the handler in CPython's &lt;code&gt;Lib/http/server.py&lt;/code&gt;. In &lt;code&gt;SimpleHTTPRequestHandler.send_head&lt;/code&gt;, the file response gets these headers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send_header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Content-type"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctype&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send_header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Content-Length"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;st_size&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send_header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Last-Modified"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;date_time_string&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;st_mtime&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There's a &lt;code&gt;Last-Modified&lt;/code&gt;. There's no &lt;code&gt;Cache-Control&lt;/code&gt;, no &lt;code&gt;Expires&lt;/code&gt;, no &lt;code&gt;ETag&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That combination matters. When a response has no explicit freshness information, HTTP caches are allowed to make one up. RFC 9111 calls this &lt;em&gt;heuristic freshness&lt;/em&gt;: if the response has a &lt;code&gt;Last-Modified&lt;/code&gt;, a cache can treat it as fresh for some fraction of the time since that date, and the RFC suggests that "a typical setting of this fraction might be 10%" [src:official:&lt;a href="https://www.rfc-editor.org/rfc/rfc9111#section-4.2.2" rel="noopener noreferrer"&gt;https://www.rfc-editor.org/rfc/rfc9111#section-4.2.2&lt;/a&gt;].&lt;/p&gt;

&lt;p&gt;Walk through what that means in a dev loop:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A file that hadn't changed in a while gets loaded once. Its &lt;code&gt;Last-Modified&lt;/code&gt; is old, so the browser decides the cached copy is good for a meaningful stretch of time.&lt;/li&gt;
&lt;li&gt;I edit the file. Its modification time on disk changes, but the browser doesn't know that. It never asks.&lt;/li&gt;
&lt;li&gt;I reload. The browser checks its cache, finds an entry it considers fresh, and uses it without touching the network.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The server would have happily told the browser that the file had changed. CPython's handler even supports conditional requests and answers &lt;code&gt;304 Not Modified&lt;/code&gt; when appropriate. But a conditional request only happens if the browser decides to revalidate, and with heuristic freshness it didn't think it needed to.&lt;/p&gt;

&lt;p&gt;So the "bug" was a perfectly standards-compliant cache doing what it's allowed to do, against a server that never told it not to.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: put the policy in the server
&lt;/h2&gt;

&lt;p&gt;Instead of cache-busting every URL, I wrote a tiny replacement for &lt;code&gt;python -m http.server&lt;/code&gt; that adds explicit no-cache headers to every response:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;NoCacheHandler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;SimpleHTTPRequestHandler&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;end_headers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send_header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Cache-Control"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"no-store, must-revalidate"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send_header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Pragma"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"no-cache"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;send_header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Expires"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"0"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end_headers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and started it like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;    &lt;span class="n"&gt;httpd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ThreadingHTTPServer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;handler&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then I replaced every launch config and helper script that used to call &lt;code&gt;python -m http.server&lt;/code&gt; with this one, and deleted the &lt;code&gt;?v=&lt;/code&gt; strings.&lt;/p&gt;

&lt;h3&gt;
  
  
  Side trap: don't downgrade to a single-threaded server
&lt;/h3&gt;

&lt;p&gt;My first version of this script used plain &lt;code&gt;HTTPServer&lt;/code&gt;, because that's what most "custom handler" examples show. It looked fine with one file. With the real page, which requests a few dozen model files and a multi-megabyte video at the same time, page loads stalled until my automation's navigation timed out after several minutes.&lt;/p&gt;

&lt;p&gt;The reason: &lt;code&gt;HTTPServer&lt;/code&gt; handles one request at a time. &lt;code&gt;python -m http.server&lt;/code&gt; doesn't have this problem because it already uses the threading variant internally. If you subclass the handler to change headers, keep &lt;code&gt;ThreadingHTTPServer&lt;/code&gt;. Otherwise you swap a caching bug for a concurrency bug that looks like a network problem.&lt;/p&gt;

&lt;h3&gt;
  
  
  What each of those four headers actually does
&lt;/h3&gt;

&lt;p&gt;I copied that header block from the usual "disable caching" snippets, and only later read what each line means. It's worth doing, because they are not four ways of saying the same thing, and one of them is a reasonable alternative to what I chose.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;no-store&lt;/code&gt;&lt;/strong&gt; is the strong one. RFC 9111 says a cache "MUST NOT store any part of either the immediate request or the response" [src:official:&lt;a href="https://www.rfc-editor.org/rfc/rfc9111#section-5.2.2.5" rel="noopener noreferrer"&gt;https://www.rfc-editor.org/rfc/rfc9111#section-5.2.2.5&lt;/a&gt;]. Every reload downloads every file again. For a local dev server that is exactly the guarantee I wanted: there is no cached copy, so there is no stale copy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;no-cache&lt;/code&gt;&lt;/strong&gt; sounds like the same thing and isn't. It allows the browser to store the response, but says it "MUST NOT be used to satisfy any other request without forwarding it for validation" [src:official:&lt;a href="https://www.rfc-editor.org/rfc/rfc9111#section-5.2.2.4" rel="noopener noreferrer"&gt;https://www.rfc-editor.org/rfc/rfc9111#section-5.2.2.4&lt;/a&gt;]. In practice the browser keeps the file and asks the server on every load, sending the &lt;code&gt;Last-Modified&lt;/code&gt; value back as &lt;code&gt;If-Modified-Since&lt;/code&gt;. If the file hasn't changed, the server answers &lt;code&gt;304 Not Modified&lt;/code&gt; and the browser reuses what it has; the RFC describes a 304 as meaning the stored response "can be updated and reused" [src:official:&lt;a href="https://www.rfc-editor.org/rfc/rfc9111#section-4.3.3" rel="noopener noreferrer"&gt;https://www.rfc-editor.org/rfc/rfc9111#section-4.3.3&lt;/a&gt;]. If the file has changed, the server sends the new one. Since CPython's handler already answers conditional requests, &lt;code&gt;no-cache&lt;/code&gt; should also have fixed my bug (I didn't try it), while skipping the re-download of unchanged model files. I went with &lt;code&gt;no-store&lt;/code&gt; because I wanted no cached copy at all while I was still distrusting the cache, and I haven't measured whether the difference matters for a page like this. If your dev page loads large assets that rarely change, &lt;code&gt;no-cache&lt;/code&gt; is the one I'd try first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;must-revalidate&lt;/code&gt;&lt;/strong&gt; only applies once a response is already stale: from then on, a cache "MUST NOT reuse that response" without checking with the server [src:official:&lt;a href="https://www.rfc-editor.org/rfc/rfc9111#section-5.2.2.2" rel="noopener noreferrer"&gt;https://www.rfc-editor.org/rfc/rfc9111#section-5.2.2.2&lt;/a&gt;]. Next to &lt;code&gt;no-store&lt;/code&gt; it does nothing, because nothing is stored to go stale. It's harmless, but it isn't what fixed anything.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;Expires: 0&lt;/code&gt;&lt;/strong&gt; and &lt;strong&gt;&lt;code&gt;Pragma: no-cache&lt;/code&gt;&lt;/strong&gt; are for older caches. The RFC says invalid dates, especially the value &lt;code&gt;0&lt;/code&gt;, must be interpreted "as representing a time in the past", so the response counts as already expired [src:official:&lt;a href="https://www.rfc-editor.org/rfc/rfc9111#section-5.3" rel="noopener noreferrer"&gt;https://www.rfc-editor.org/rfc/rfc9111#section-5.3&lt;/a&gt;]. &lt;code&gt;Pragma&lt;/code&gt; is officially deprecated [src:official:&lt;a href="https://www.rfc-editor.org/rfc/rfc9111#section-5.4" rel="noopener noreferrer"&gt;https://www.rfc-editor.org/rfc/rfc9111#section-5.4&lt;/a&gt;]. A modern browser follows &lt;code&gt;Cache-Control&lt;/code&gt;, so these two lines are belt and braces for whatever else might sit in between.&lt;/p&gt;

&lt;p&gt;The practical takeaway: if you only remember one line, make it &lt;code&gt;Cache-Control&lt;/code&gt;, and choose between &lt;code&gt;no-store&lt;/code&gt; (never keep a copy) and &lt;code&gt;no-cache&lt;/code&gt; (keep a copy, but always ask first) on purpose rather than by copy-paste. Either should have prevented my day of phantom bugs. Neither would have cleaned up the entries that were already cached, which is the next part.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that surprised me: &lt;code&gt;no-store&lt;/code&gt; doesn't evict what's already cached
&lt;/h2&gt;

&lt;p&gt;With the new server running, I reloaded and... the stale file was still there.&lt;/p&gt;

&lt;p&gt;This was the moment I nearly concluded the fix didn't work. It did work. What I'd missed is that &lt;code&gt;Cache-Control: no-store&lt;/code&gt; is a property of a &lt;em&gt;response&lt;/em&gt;. The browser only learns about it when it actually receives a new response. But the old entry in the cache was still, by its own heuristic, fresh. So the browser served it from cache without going to the network, which means it never received a response carrying the new header.&lt;/p&gt;

&lt;p&gt;The new policy was correct, and the browser was never going to see it until something forced a network request for that URL.&lt;/p&gt;

&lt;p&gt;What forced it was a one-liner from the page's DevTools console:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nb"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'/main.js'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;cache&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'reload'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;then&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;MDN describes &lt;code&gt;cache: 'reload'&lt;/code&gt; as: the browser fetches from the server "without first looking in the cache, &lt;em&gt;but then will&lt;/em&gt; update the cache with the downloaded resource" [src:official:&lt;a href="https://developer.mozilla.org/en-US/docs/Web/API/Request/cache" rel="noopener noreferrer"&gt;https://developer.mozilla.org/en-US/docs/Web/API/Request/cache&lt;/a&gt;]. That's exactly the shape I needed: skip the stale entry, get a real response from the new server, and let that response replace what was cached.&lt;/p&gt;

&lt;p&gt;It was also the fastest way to tell the two layers apart. If the text that comes back from &lt;code&gt;fetch(..., {cache: 'reload'})&lt;/code&gt; contains my change and the page still doesn't behave that way, the cache is the suspect. If the text doesn't contain my change, the server is serving an old file and the problem is upstream of the browser.&lt;/p&gt;

&lt;p&gt;For a regular browser tab that already had the stale entries, a single hard reload (Ctrl+Shift+R) was needed once. After that, every response carried &lt;code&gt;no-store&lt;/code&gt; and the problem didn't come back.&lt;/p&gt;

&lt;h2&gt;
  
  
  The general lesson: "fixed" is a claim about a specific copy
&lt;/h2&gt;

&lt;p&gt;Looking back, I made the same mistake every time: I verified the change in the place I had made it. I read the file I edited and concluded it was right. That was true. It just wasn't the file that was running.&lt;/p&gt;

&lt;p&gt;There are usually several copies of "the code" between your editor and the user's screen, and any one of them can be stale:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;source file  →  build output  →  deployed version  →  CDN / edge cache  →  browser cache  →  what runs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;My incident was entirely in the last hop: a local server and a browser, no build, no deploy, no CDN. But the question is the same at every hop: &lt;em&gt;which copy is actually being served here, and does it contain my change?&lt;/em&gt; And the answer should come from a value you can look at, not from re-reading the source.&lt;/p&gt;

&lt;p&gt;Here is the checklist I use now. It's not clever. The point is that every step ends in something concrete you can compare.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Source: is the change in the file you think is being used?&lt;/strong&gt;&lt;br&gt;
Not "I edited it." Grep the exact string you changed in the file the page actually loads. Duplicated files and similarly named folders are common, and a quick grep is cheaper than an hour of debugging the wrong copy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Build output: did the change survive the build?&lt;/strong&gt;&lt;br&gt;
If there's a bundler, grep the built artifact for something unique to your change, or compare its hash before and after. A build that silently reused an old output looks exactly like a code bug.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Deploy: is the deployed version the one you built?&lt;/strong&gt;&lt;br&gt;
Most hosts show a deployment ID or commit for what's live. Compare it to the commit you think you shipped. "I pushed" and "that push is what's serving traffic" are different statements.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. CDN / edge: what does the public URL actually return?&lt;/strong&gt;&lt;br&gt;
Look at the headers and the body, not the dashboard:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -sI https://example.com/main.js
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check &lt;code&gt;Cache-Control&lt;/code&gt;, &lt;code&gt;ETag&lt;/code&gt;, &lt;code&gt;Last-Modified&lt;/code&gt;, and &lt;code&gt;Age&lt;/code&gt; (a large &lt;code&gt;Age&lt;/code&gt; means you're looking at a cached copy). Many CDNs also add a cache-status header (&lt;code&gt;Cache-Status&lt;/code&gt; per RFC 9211, or a vendor-specific equivalent) that tells you whether it was a hit. Then compare the bytes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl -s https://example.com/main.js &lt;span class="o"&gt;|&lt;/span&gt; sha256sum
sha256sum dist/main.js
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the hashes differ, the copy at the edge is not the copy you built. I do this now for static files I deploy, even small ones like images: fetch the public URL and confirm the bytes match the local file before calling it done.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Browser: which copy did this tab use?&lt;/strong&gt;&lt;br&gt;
In the DevTools Network panel, the Size column shows &lt;code&gt;(disk cache)&lt;/code&gt; or &lt;code&gt;(memory cache)&lt;/code&gt; when a response didn't come from the network. "Disable cache" in DevTools only applies while DevTools is open, so it can hide the problem from you while your users still hit it. And if you're unsure, &lt;code&gt;fetch(url, {cache: 'reload'})&lt;/code&gt; from the console gives you the current server copy to compare against.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Dev servers: does it send any caching headers at all?&lt;/strong&gt;&lt;br&gt;
This is the one that got me. &lt;code&gt;curl -sI&lt;/code&gt; against your localhost. If there's no &lt;code&gt;Cache-Control&lt;/code&gt;, the browser is allowed to guess, and its guess may not match your edit loop.&lt;/p&gt;

&lt;h2&gt;
  
  
  A second, smaller version of the same mistake
&lt;/h2&gt;

&lt;p&gt;On the same day, I hit a different bug with the same shape. An object in the 3D scene appeared tilted when it should have been upright. I spent time in the physics and placement code, because that's what I'd been editing. The cause was in the model file itself: the orientation was baked into the asset.&lt;/p&gt;

&lt;p&gt;What finally separated the layers was the same move as with the cache: measure a value instead of reasoning about code. I logged the object's tilt right after spawning it, and it was zero. That one number said the code was placing it correctly, which meant the problem was in what was being placed.&lt;/p&gt;

&lt;p&gt;That's the version of the lesson I actually try to apply now: &lt;strong&gt;don't search for a cause only in the layer you happen to be working in.&lt;/strong&gt; The layer you're editing is the one you can see, which makes it the one you suspect first, and the first suspect isn't always the right one.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I do now
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Before debugging "my change didn't take effect," prove which copy is running.&lt;/strong&gt; One &lt;code&gt;fetch(..., {cache: 'reload'})&lt;/code&gt; or one &lt;code&gt;curl&lt;/code&gt; plus a hash comparison usually settles it quickly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Put cache policy in the server, not in URLs.&lt;/strong&gt; For local dev, a server that sends &lt;code&gt;no-store&lt;/code&gt; removes a whole class of phantom bugs. Query-string busting is a patch you have to keep re-applying.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Remember that new headers only reach new responses.&lt;/strong&gt; Changing the server's policy doesn't clean up entries the browser already considers fresh. Expect to force one real request per stale URL.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep the threading.&lt;/strong&gt; If you replace &lt;code&gt;python -m http.server&lt;/code&gt; with your own handler, use &lt;code&gt;ThreadingHTTPServer&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Report "fixed" with a delivered value.&lt;/strong&gt; "The served file's hash matches the build" or "the response has my change" is a claim someone else can check. "I edited the code" isn't.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this is new. Caching has been biting web developers for as long as there have been caches. What was new for me was how convincing the wrong diagnosis was: every time, the diff was right, so every time I believed the diff.&lt;/p&gt;

&lt;p&gt;I'm building an AI code reviewer (&lt;a href="https://www.getdevreview.com" rel="noopener noreferrer"&gt;getdevreview.com&lt;/a&gt;) that tries to flag the bugs that matter instead of everything it can find. A bug like this one would never show up in a diff, which is part of why I've become strict about checking the delivered copy myself. If you have a favorite trick for telling "stale" from "broken" quickly, I'd like to hear it.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>javascript</category>
      <category>python</category>
      <category>debugging</category>
    </item>
    <item>
      <title>When a Code Review Says "No Issues" but Found Bugs</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Sun, 08 Nov 2026 05:48:05 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/when-a-code-review-says-no-issues-but-found-bugs-36bj</link>
      <guid>https://dev.to/phi_blankslate/when-a-code-review-says-no-issues-but-found-bugs-36bj</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.getdevreview.com/blog/when-a-code-review-says-no-issues-but-found-bugs" rel="noopener noreferrer"&gt;DevReview blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;DevReview caps every review at eight comments, ranked by severity. The point is to keep a pull request readable: a few findings that matter, not forty that don't. A cap like that only works if the reader can trust what is left out. This post is about a path where they couldn't.&lt;/p&gt;

&lt;p&gt;We found it while checking our own claims against our own code, and we haven't measured whether it ever happened in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the review did
&lt;/h2&gt;

&lt;p&gt;After the model returns its findings, the review code does three things in this order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Sort the findings by severity and keep the top eight.&lt;/li&gt;
&lt;li&gt;Drop any finding whose file path is not in the pull request. The model sometimes cites a file that doesn't exist in the diff, and a comment on a file that isn't there can't be posted.&lt;/li&gt;
&lt;li&gt;If nothing is left to post, write "No issues worth your attention" with a green check.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The problem is the order of steps 1 and 2. Suppose the model returns ten findings, and the top eight all cite file paths that aren't in the PR. Step 1 keeps those eight. Step 2 drops all of them. Step 3 sees nothing left and posts the green check. The two real findings that were ranked ninth and tenth were already cut by the cap.&lt;/p&gt;

&lt;p&gt;So the pull request shows "no issues" while the model has reported eight bugs it got the location wrong on, and two it got right. The reader gets a pass they shouldn't have.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two separate things were wrong
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;A discarded finding looked the same as no finding.&lt;/strong&gt; Step 2 removed findings without counting them anywhere the reader could see. A green check means "the reviewer looked and found nothing". It should not also mean "the reviewer found things and we threw them away".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fake paths used up the budget.&lt;/strong&gt; Even when the result wasn't a false all-clear, findings with invalid paths took slots from valid ones. A valid warning could be suppressed to make room for a comment that was never going to be posted.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we changed
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The path check now runs before the cap.&lt;/strong&gt; Only findings that can actually be posted compete for the eight slots.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Discarded findings are counted and disclosed.&lt;/strong&gt; If any findings were dropped because their file wasn't in the PR, or because they were malformed, the review says so: "N findings could not be posted because they did not match this PR's files or the expected format."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The green check is stricter.&lt;/strong&gt; It appears only when the model reported zero findings. One discarded finding is enough to remove it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Suppressed comments show a breakdown.&lt;/strong&gt; When the cap cuts comments, the note now says what was cut, for example "2 remarks suppressed to keep this review focused: 1 NIT, 1 PRAISE." We also removed the word "lower-priority" from that note, because the cap can also cut a bug when there are more than eight.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We also weakened some of our own marketing copy. We had said nothing is hidden silently. That is true of comments cut by the cap, which are disclosed, but not of malformed comments, which were dropped without a count. The copy now says what is true.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we checked, and what we didn't
&lt;/h2&gt;

&lt;p&gt;We wrote the tests first and confirmed they failed on the old behaviour. The review function is now tested end to end with faked responses from the model and from GitHub, including the case above: eight invented paths plus two valid findings now posts the two valid ones and discloses the eight. Removing the counting of malformed findings makes a test fail, so that test does something.&lt;/p&gt;

&lt;p&gt;What we have not done:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;We haven't watched this on a real production review.&lt;/strong&gt; The tests use faked responses. A real GitHub and Anthropic exchange is the next check.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The dashboard can still mislead.&lt;/strong&gt; A review where every finding was discarded shows as "completed, 0 comments", which looks like a pass. The discarded count appears in the pull request comment and in our logs, but we don't store it in the database yet. Showing it on the dashboard needs a new column.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Related gaps exist.&lt;/strong&gt; A green check can still appear when some files in a pull request were not reviewed at all, and a code fence inside a comment can make the whole review fail. Both are on our list as separate fixes.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The general rule
&lt;/h2&gt;

&lt;p&gt;If a system filters, ranks and then reports, the order of those steps decides what the report means. Ask what the report says when the filter removes everything. For us, the answer was "all clear", which is the one thing it must never say after discarding findings.&lt;/p&gt;

&lt;p&gt;DevReview is a GitHub App that reviews pull requests and caps every review at eight comments, ranked by severity. If you want to see how it works before trusting it with your repos, start with &lt;a href="https://www.getdevreview.com/how-it-works" rel="noopener noreferrer"&gt;How DevReview Works&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>programming</category>
      <category>testing</category>
    </item>
    <item>
      <title>My AI pipeline marked four drafts "passed". The second check had never run.</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Wed, 04 Nov 2026 07:53:16 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/my-ai-pipeline-marked-four-drafts-passed-the-second-check-had-never-run-2jf3</link>
      <guid>https://dev.to/phi_blankslate/my-ai-pipeline-marked-four-drafts-passed-the-second-check-had-never-run-2jf3</guid>
      <description>&lt;p&gt;I run a small side project that publishes short social posts, in Japanese, about keeping household pests out of your home. I don't write those posts by hand. A set of AI agents drafts them, a two-stage checker reviews them, and the finished drafts land in an &lt;code&gt;outbox/&lt;/code&gt; folder where I read them and post them myself. Nothing goes out automatically.&lt;/p&gt;

&lt;p&gt;One day the agent that summarizes the day for me reported that four drafts in the outbox had passed review and were ready to post.&lt;/p&gt;

&lt;p&gt;They had not passed review. One of them contained two numbers with no source, one claim about effectiveness that the checker is specifically supposed to block, and a fact that the model had copied from the wrong chapter of its own knowledge base. A second agent, the one that was supposed to catch exactly this, had never looked at any of the four.&lt;/p&gt;

&lt;p&gt;Nothing was posted, because I re-checked before posting. But the fact that I had to re-check is the whole problem, so this is a write-up of what went wrong, why each failure got through, and the two small checks I now use. The checks are short. Getting to them took a while.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The review step is two agents in sequence, modeled on how I split gates in code review:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;scan&lt;/strong&gt; — a small, cheap model. It compares the draft against a list of banned and borderline phrases ("guaranteed", "100%", numbers about effectiveness with no source, and so on). It only reports. It never edits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;judge&lt;/strong&gt; — a larger model. It only looks at the borderline items that scan flagged, and decides: allow, or fix with specific instructions.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;draft → scan: compare against the phrase list
          → nothing found       → draft is final
          → borderline found    → judge rules → fix or allow
          → clearly banned      → sent back to the writer
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each draft is a Markdown file with frontmatter, and the result of the review is written into it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;draft&lt;/span&gt;
&lt;span class="na"&gt;platform&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;x&lt;/span&gt;
&lt;span class="na"&gt;compliance&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;passed&lt;/span&gt;
&lt;span class="na"&gt;created&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;2026-07-10&lt;/span&gt;
&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The summarizing agent reads &lt;code&gt;compliance: passed&lt;/code&gt; and tells me the draft is ready. That is a reasonable thing for it to do. The field says passed.&lt;/p&gt;

&lt;h2&gt;
  
  
  What "passed" actually meant
&lt;/h2&gt;

&lt;p&gt;When I went back through the records, here is what I found.&lt;/p&gt;

&lt;p&gt;The four drafts were created on the day I first set up this group of agents. That first run was a dry run. I was checking that the pieces connected, not producing real output. During the dry run, scan ran. Judge did not. The drafts were marked &lt;code&gt;compliance: passed&lt;/code&gt; anyway, and they stayed in the outbox with that flag for seven days.&lt;/p&gt;

&lt;p&gt;A different file, the pipeline log, said plainly that review had not been carried out during the dry run. So the system contained a flag saying "passed" and a log saying "not checked", and nothing ever compared them. The summarizing agent read the flag, because the flag is in the file it was told to read.&lt;/p&gt;

&lt;p&gt;So the first failure was simple. &lt;strong&gt;A flag that says "passed" recorded that something had passed, but not which check had run, when it ran, or who ran it.&lt;/strong&gt; With two stages, "passed" after stage one and "passed" after both stages look exactly the same.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the drafts actually contained
&lt;/h2&gt;

&lt;p&gt;I re-ran the review by hand on all four. Three were fine. The third post was not: it had two numbers with no source, one effectiveness claim, and a main hook that contradicted the source. And the review of it produced one more error of its own.&lt;/p&gt;

&lt;p&gt;One of the two unsourced numbers was a claim that pest complaints rise toward a particular month. The reference document has no seasonal data at all, so there was nothing to check it against. The other one is more interesting.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. A fact copied from the neighboring chapter
&lt;/h3&gt;

&lt;p&gt;The post said, roughly, "cockroaches are said to become more active once the temperature goes above" a specific value.&lt;/p&gt;

&lt;p&gt;The agents write from a single reference document, a long Markdown file that summarizes and cites public sources. It is organized by pest: cockroaches, ants, mosquitoes, dust mites, termites, flies, and so on. Every factual claim in a post is supposed to come from it.&lt;/p&gt;

&lt;p&gt;I opened the file and read the cockroach chapter. It contains no temperature figure at all. The temperature figure is real, and it is in the document, but it is in the &lt;strong&gt;mosquito&lt;/strong&gt; chapter, in a line about where mosquitoes breed.&lt;/p&gt;

&lt;p&gt;This kind of error is hard to catch, and I think it is worth spelling out why:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The fact exists in the source.&lt;/strong&gt; A text search for the number finds it. If your check is "does this claim appear in the reference document?", the answer is yes, and the check passes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It is plausible.&lt;/strong&gt; Temperature affecting insect activity is not a strange idea. Nobody reading the post would stop on it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It was wrapped in "it is said that".&lt;/strong&gt; My phrase list blocked unsourced numbers stated as fact. It did not block unsourced numbers stated as hearsay. "It is said that" turned out to work as a way around the rule. It sounds careful, and it gives no source.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The model did not make the number up. It took a true sentence from the document and used it for the wrong subject. The source existed. It was attached to the wrong subject.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. The fix had the same problem
&lt;/h3&gt;

&lt;p&gt;When judge reviewed the draft during the re-check, it proposed a replacement sentence about when one species lays eggs, and added "(according to manufacturer data)".&lt;/p&gt;

&lt;p&gt;The egg-laying season is in the cockroach chapter, so the fact was fine. But the reference document does not attribute it to a manufacturer. A manufacturer is cited a few lines above, for a different fact (how many eggs are in each egg case). The judge took the citation from that line and attached it to a different line further down.&lt;/p&gt;

&lt;p&gt;So the agent I had set up to catch unsupported attributions produced one in its own fix. It was not a dramatic error. It is the same kind of error as the original, made by the reviewer while fixing it.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. The main hook ignored a warning in the source
&lt;/h3&gt;

&lt;p&gt;The post's main hook was that a particular household fixture is a common way for cockroaches to get inside in summer. The reference document doesn't support that framing. Its section on cockroach entry routes does list that fixture as one possible way in, but the next item adds a professional's view that it is unlikely to be one. The document also ends with a &lt;strong&gt;Caveats&lt;/strong&gt; section that lists a closely related entry-route claim as disputed.&lt;/p&gt;

&lt;p&gt;Neither scan nor judge caught this. Scan compares phrases, and nothing in the hook was a banned phrase. Judge only sees what scan flags. The disagreement was between the draft and a sentence in the source that argues against it, and no check compared those two things.&lt;/p&gt;

&lt;p&gt;Fixing the wording would not have helped, because the whole post was built on that claim. The post was rebuilt around a different, uncontested fact from the cockroach chapter.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check 1: a flag has to carry its evidence
&lt;/h2&gt;

&lt;p&gt;The first fix is to stop accepting a bare &lt;code&gt;passed&lt;/code&gt;. Every draft now has to carry a &lt;code&gt;compliance_log&lt;/code&gt; that records each stage with a date:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;compliance&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;passed&lt;/span&gt;
&lt;span class="na"&gt;compliance_log&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;scan&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;2026-07-17&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;⚠️&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;1&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;borderline&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;item&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;→&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;judge&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;2026-07-17&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;✅&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;allowed"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or, when scan finds nothing borderline:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;compliance_log&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;scan&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;2026-07-17&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;✅&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;banned&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;borderline&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;judge不要"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(&lt;code&gt;judge不要&lt;/code&gt; means "judge not needed". The real logs are in Japanese, and the check below matches that exact token.)&lt;/p&gt;

&lt;p&gt;A rule in a README does nothing unless something reads it, so I wrote a check that runs before I post. This is the whole thing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Gate check: a "passed" flag only counts if it carries the evidence of each stage.&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;readFileSync&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;readdirSync&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:fs&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;join&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:path&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;DATE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;String&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;raw&lt;/span&gt;&lt;span class="s2"&gt;`\d{4}-\d{2}-\d{2}`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;SCAN&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;RegExp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;String&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;raw&lt;/span&gt;&lt;span class="s2"&gt;`scan (&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;DATE&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;)`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;JUDGE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;RegExp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;String&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;raw&lt;/span&gt;&lt;span class="s2"&gt;`judge (&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;DATE&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;)|judge不要`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;frontmatter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/^---&lt;/span&gt;&lt;span class="se"&gt;\r?\n([\s\S]&lt;/span&gt;&lt;span class="sr"&gt;*&lt;/span&gt;&lt;span class="se"&gt;?)\r?\n&lt;/span&gt;&lt;span class="sr"&gt;---/&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;fm&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{};&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;line&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\r?\n&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;indexOf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;fm&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;()]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/^"|"$/g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;""&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;fm&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;checkDraft&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;fm&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;fm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;compliance&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;passed&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;why&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;not marked passed&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;log&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;fm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;compliance_log&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;why&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;passed, but no compliance_log&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;scan&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;SCAN&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;scan&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;why&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;no dated scan record&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;judge&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;JUDGE&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;why&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;scan recorded, judge stage missing&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;scan&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;why&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;judge dated before scan&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;why&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`scan &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;scan&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]}&lt;/span&gt;&lt;span class="s2"&gt; / &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;judge &lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;judge not needed&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(Comparing ISO dates as strings works because they sort lexically.)&lt;/p&gt;

&lt;p&gt;Fed the two states the drafts were in before the fix, it holds both:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;ok:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;why:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;'passed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;but&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;no&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;compliance_log'&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;ok:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;why:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;'scan&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;recorded&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;judge&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;stage&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;missing'&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second one is the failure that actually happened: stage one recorded, stage two missing, still marked passed.&lt;/p&gt;

&lt;h3&gt;
  
  
  The check found something I didn't expect
&lt;/h3&gt;

&lt;p&gt;When I ran it over the real outbox for this article, the six drafts that were re-reviewed under the new rule that week passed. Three drafts written for the following week did not:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HOLD 2026-07-24-post-1.md — passed, but no compliance_log
HOLD 2026-07-24-post-2.md — passed, but no compliance_log
HOLD 2026-07-24-post-3.md — passed, but no compliance_log
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At first this looked like the same bug again. It wasn't quite. Those three drafts &lt;em&gt;had&lt;/em&gt; been reviewed properly. The writing agent's daily log has the full record: scan found five borderline items, judge allowed two and gave fix instructions for three, the drafts were revised (the writing agent declined two of judge's instructions and wrote down why), and a second scan came back clean. The review happened. It was written down in a log file, not in the draft.&lt;/p&gt;

&lt;p&gt;I'm keeping the HOLD anyway. The original incident started because the "passed" flag was in one file and the "not checked" note was in another, and nothing compared them. If I accept "the evidence is somewhere else", I'm back to that. &lt;strong&gt;Evidence has to be stored in the same file as the flag.&lt;/strong&gt; That's less convenient for the writing agent, but the gate only has to read one file to decide.&lt;/p&gt;

&lt;p&gt;It also showed me something about the rule. I added &lt;code&gt;compliance_log&lt;/code&gt; as a requirement on the day of the incident. On that same day, a different agent produced three drafts without it. A written rule did nothing until a script checked for it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check 2: is the fact in the right chapter?
&lt;/h2&gt;

&lt;p&gt;The neighboring-chapter error needs a different check. "Does this text appear in the source?" is the question that failed, so the check has to ask "which section does it appear in?"&lt;/p&gt;

&lt;p&gt;The reference document is Markdown with a heading hierarchy, so a search can report the heading path where each match sits:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Attribution check: is the fact in the section about the subject, or just somewhere in the file?&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;markdown&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;lines&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;markdown&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\r?\n&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;stack&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
  &lt;span class="nx"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;forEach&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/^&lt;/span&gt;&lt;span class="se"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;#&lt;/span&gt;&lt;span class="se"&gt;{1,6})\s&lt;/span&gt;&lt;span class="sr"&gt;+&lt;/span&gt;&lt;span class="se"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;.*&lt;/span&gt;&lt;span class="se"&gt;)&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;h&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;at&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nx"&gt;depth&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="na"&gt;end&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
      &lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;depth&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;h&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="na"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;stack&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pop&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="na"&gt;end&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;list&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;out&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;locate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;markdown&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;needle&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;list&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;markdown&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;hits&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
  &lt;span class="nx"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;forEach&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;needle&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;owners&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;list&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;start&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;end&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sort&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;depth&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nx"&gt;hits&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;line&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;owners&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reverse&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt; &amp;gt; &lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;hits&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The CLI wrapper prints &lt;code&gt;OK&lt;/code&gt; when the heading path contains the subject and &lt;code&gt;MISATTRIBUTED&lt;/code&gt; when it doesn't. Run against the real reference document with the subject set to "cockroach", the temperature claim from the incident comes back as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;MISATTRIBUTED L89  Guide &amp;gt; Details &amp;gt; 3. Mosquitoes
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and a fact the rebuilt post kept from the cockroach chapter (how long egg cases take to hatch) comes back as:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;OK   L66  Guide &amp;gt; Details &amp;gt; 1. Cockroaches &amp;gt; Egg-laying sites
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(I've translated the headings. The real ones are in Japanese.)&lt;/p&gt;

&lt;p&gt;This is a rough check and I want to be clear about what it misses:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It needs a searchable string.&lt;/strong&gt; It works on numbers and distinctive phrases. It does nothing for a paraphrased claim.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It doesn't check citations at the line level.&lt;/strong&gt; The judge's mistake, putting a manufacturer citation on a different line from the one it belonged to, happened inside the right chapter. Both lines are under "Cockroaches", so this check would pass it. Catching that needs the citation and the fact to be linked in the source document itself, which mine aren't yet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Wrong section" is not always wrong.&lt;/strong&gt; The Key Findings summary at the top repeats facts from the detail sections, and a match in Caveats is a reason to read more carefully, not an automatic failure. It flags things for a person to look at.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Even so, it would have caught the original error. The original error would also have passed every existing check, because it was a true fact in the wrong place.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rules I added
&lt;/h2&gt;

&lt;p&gt;Beyond the two scripts, three rules went into the reviewer's definition:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Every number or factual claim must cite either a line number in the reference document or a primary-source URL.&lt;/strong&gt; If you can't point to one, it is a clear failure. Wrapping it in "it is said that" doesn't change that. Hearsay phrasing no longer gets around the rule. The URL option exists because seasonal posts are &lt;em&gt;supposed&lt;/em&gt; to use current news, which can't be in a static reference file. Requiring only line numbers would have rejected every seasonal post.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check the draft against the source's Caveats section every time.&lt;/strong&gt; If the main hook depends on something listed as disputed, the fix is to rebuild the post, not reword it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Notes in the planning log are not facts.&lt;/strong&gt; The planning log had a note saying "next time, angle the seasonal post toward" a particular month being the peak. The writing agent researched it and found no statistic that supported it. The only figures it found showed a different month on top, and the "peak" claim only appeared in unsourced SEO articles from pest-control companies. That note had been written as a plan, and the next run was about to treat it as a fact. A log of what to avoid repeating doesn't tell you what's true.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What I'd tell myself before building this
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;A status flag is a claim, not evidence.&lt;/strong&gt; &lt;code&gt;passed&lt;/code&gt;, &lt;code&gt;verified&lt;/code&gt;, &lt;code&gt;approved&lt;/code&gt;, &lt;code&gt;done&lt;/code&gt; are all claims. A flag that doesn't say who set it, when, and based on which check should be treated as if no check happened. With several stages, the flag has to show which stages ran, or "stage one passed" looks identical to "everything passed".&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Look for leftover dry-run output.&lt;/strong&gt; The day you first wire up a pipeline is the day it's most likely to produce output that looks real and isn't. I had test output sitting in the real outbox with the real flag on it for a week. Anything created on a setup day should be reviewed before anyone trusts it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Is it in the source?" is weaker than it sounds.&lt;/strong&gt; A large source document contains plenty of true facts about the wrong subject. The question you need is "is it in the source, in the section about this subject, with this citation?" Each part of that question catches a different error.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The reviewer's fixes need checking too.&lt;/strong&gt; The larger model making the final call didn't make it reliable. It repeated the original mistake while fixing it. Whatever the last automated step produces is still something I have to verify.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Store evidence with the artifact.&lt;/strong&gt; If the gate has to look in another file to decide, it will eventually read a file that's out of date.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rules in prose don't run.&lt;/strong&gt; I wrote the &lt;code&gt;compliance_log&lt;/code&gt; requirement, and on the same day another agent produced drafts without it. Nobody noticed until a script checked for it.&lt;/p&gt;

&lt;p&gt;None of this is specific to pest-control posts. If you have an LLM pipeline with a "reviewed" flag somewhere, whether it's content, code, or data labeling, I'd suggest a quick test. Pick one item marked as passed and try to find the record of each check that was supposed to run on it, without leaving that item's file. If you can't, your flag is in the state mine was in.&lt;/p&gt;

&lt;p&gt;How are you recording which checks ran on AI-generated output? I'm interested in whether anyone has a good way to link a citation to a specific fact in the source, because that's the part my section check still misses.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
      <category>programming</category>
    </item>
    <item>
      <title>I almost killed a project because my test harness was measuring its own bug</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Sat, 31 Oct 2026 22:36:09 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/i-almost-killed-a-project-because-my-test-harness-was-measuring-its-own-bug-5bme</link>
      <guid>https://dev.to/phi_blankslate/i-almost-killed-a-project-because-my-test-harness-was-measuring-its-own-bug-5bme</guid>
      <description>&lt;p&gt;I was one command away from spending real API money on an experiment whose answer would have been wrong — and wrong in the most dangerous direction: it would have told me the thing I was testing &lt;em&gt;worked&lt;/em&gt;, for the wrong reason, or failed for a reason that had nothing to do with it.&lt;/p&gt;

&lt;p&gt;What saved me was boring. Before running the paid version, I ran the free version and actually read the raw data with my own eyes. This post is about what I found there, why measurement bugs tend to lie in your favor, and the rule I now follow before any expensive evaluation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The question I was trying to answer
&lt;/h2&gt;

&lt;p&gt;The project was a small accessibility tool. The pipeline looked like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Load a real web page in a headless browser (Playwright).&lt;/li&gt;
&lt;li&gt;Run &lt;a href="https://github.com/dequelabs/axe-core" rel="noopener noreferrer"&gt;axe-core&lt;/a&gt; on it to detect accessibility violations — missing button labels, images without alt text, invalid ARIA attributes, that kind of thing.&lt;/li&gt;
&lt;li&gt;For each violation, hand the offending HTML element to an LLM and ask it for a fix in a strict format:
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;BEFORE:
```html
&amp;lt;the element as it is now&amp;gt;
```
AFTER:
```html
&amp;lt;the fixed element&amp;gt;
```
EXPLANATION: ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(The code comments you'll see below are in Japanese — that's how they are in the actual repo, and I've kept them as-is rather than rewrite code for the post.)&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Score the result.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The whole project hinged on one kill question: &lt;em&gt;are the AI's fix suggestions actually usable?&lt;/em&gt; If not, there was no product. So I wrote a verification harness that would run the pipeline over a list of real sites and classify every response.&lt;/p&gt;

&lt;p&gt;The classifier was strict on purpose. A response was only counted as a valid, applicable fix if it passed a chain of checks, in this order:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// 判定順序は仕様通り: api_error -&amp;gt; no_text -&amp;gt; truncated -&amp;gt; malformed -&amp;gt; unresolved -&amp;gt; noop -&amp;gt; before_mismatch -&amp;gt; source_truncated -&amp;gt; valid_applicable&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The two checks that matter for this story are &lt;code&gt;noop&lt;/code&gt; and &lt;code&gt;before_mismatch&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;normalizeHtml&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;parsed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;after&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nf"&gt;normalizeHtml&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;parsed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;before&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;noop&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;normalizeHtml&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;parsed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;before&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="nf"&gt;normalizeHtml&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;nodeHtml&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;before_mismatch&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In plain English: the model's &lt;code&gt;BEFORE&lt;/code&gt; block has to match the element I gave it. That is how I make sure the fix can actually be applied to the page as a find-and-replace patch, rather than being a hallucinated rewrite of something that doesn't exist.&lt;/p&gt;

&lt;p&gt;That check felt rigorous. It turned out to be the exact place where the harness lied to me.&lt;/p&gt;

&lt;h2&gt;
  
  
  The free run
&lt;/h2&gt;

&lt;p&gt;The paid run needed an API key that I didn't have yet. While waiting, I added a &lt;code&gt;--dry-run&lt;/code&gt; flag:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;//   node k2-verify.js --dry-run  APIを一切呼ばずscanUrlの配線確認のみ&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It runs the browser and axe-core against every site, builds every input the LLM &lt;em&gt;would&lt;/em&gt; have received, and writes them all to disk — but never calls the API. Zero cost. The point was just to confirm the wiring.&lt;/p&gt;

&lt;p&gt;I almost didn't read the output. The run completed, the summary table looked plausible, and the natural next step was "wait for the key." Instead I opened the raw JSON and read the actual HTML snippets I was about to send to the model.&lt;/p&gt;

&lt;p&gt;Many of them were broken.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug 1: axe-core's &lt;code&gt;node.html&lt;/code&gt; is not the element
&lt;/h2&gt;

&lt;p&gt;I had assumed that &lt;code&gt;violation.nodes[i].html&lt;/code&gt; — the HTML string axe-core attaches to every violation — was the element's markup. It's not. It's a &lt;em&gt;summary&lt;/em&gt; of the element, built for humans reading a report.&lt;/p&gt;

&lt;p&gt;I didn't trust my memory about how it was built, so I opened &lt;code&gt;node_modules/axe-core/axe.js&lt;/code&gt; and read the function that produces it. In the version I had installed (4.12.1), &lt;code&gt;_getElementSource&lt;/code&gt; takes a &lt;code&gt;maxLength&lt;/code&gt; that defaults to 300 and an &lt;code&gt;attrLimit&lt;/code&gt; that defaults to 20:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;_getElementSource&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;node&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;_ref24&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;arguments&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{},&lt;/span&gt; &lt;span class="nx"&gt;_ref24$maxLength&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;_ref24&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;maxLength&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;maxLength&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;_ref24$maxLength&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="k"&gt;void&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;_ref24$maxLength&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;_ref24$attrLimit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;_ref24&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;attrLimit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;attrLimit&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;_ref24$attrLimit&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="k"&gt;void&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;_ref24$attrLimit&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;...&lt;/span&gt;
  &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;deepStr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;getOuterHtml&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;node&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;deepStr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;maxLength&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;getTruncatedElementSource&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;node&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;maxLength&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;maxLength&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and the attribute truncation is this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;truncate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;attrLimit&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;str&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nx"&gt;attrLimit&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;str&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;str&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;substring&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;attrLimit&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;...&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So when an element's &lt;code&gt;outerHTML&lt;/code&gt; is longer than 300 characters, you don't get the element. You get its opening tag only — children gone, closing tag gone — with long attribute values cut to 20 characters plus &lt;code&gt;...&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That's a completely reasonable design for a violations report. It's a terrible input for "please write a patch for this exact markup."&lt;/p&gt;

&lt;h3&gt;
  
  
  Why this bug points toward "pass"
&lt;/h3&gt;

&lt;p&gt;Here's the part that still makes me uncomfortable.&lt;/p&gt;

&lt;p&gt;When you hand an LLM a truncated, unclosed element and ask for &lt;code&gt;BEFORE&lt;/code&gt; and &lt;code&gt;AFTER&lt;/code&gt;, a well-behaved model copies what you gave it into &lt;code&gt;BEFORE&lt;/code&gt;. Truncated attribute and all. Then my strict classifier compares &lt;code&gt;BEFORE&lt;/code&gt; to the input I sent — the same truncated string — and they match.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;before_mismatch&lt;/code&gt; passes.&lt;/strong&gt; The response gets counted as an applicable fix.&lt;/p&gt;

&lt;p&gt;But it isn't applicable. The real page doesn't contain &lt;code&gt;&amp;lt;a href="https://example.co..."&amp;gt;&lt;/code&gt; with no closing tag. Nobody can find-and-replace that into a real codebase. The "patch" is a patch against a string that only exists inside my harness.&lt;/p&gt;

&lt;p&gt;My most rigorous check was rigorous against the wrong ground truth. And because the error made the model &lt;em&gt;look&lt;/em&gt; better, I would never have gone looking for it after a good-looking result. I would have celebrated.&lt;/p&gt;

&lt;h3&gt;
  
  
  The detection trap
&lt;/h3&gt;

&lt;p&gt;My first instinct for detecting truncated inputs was to search for the &lt;code&gt;...&lt;/code&gt; marker. That turned out to be wrong in both directions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;False positives:&lt;/strong&gt; real page text contains &lt;code&gt;...&lt;/code&gt;. A heading like &lt;code&gt;&amp;lt;h3&amp;gt;Be Bold...&amp;lt;/h3&amp;gt;&lt;/code&gt; or an attribute like &lt;code&gt;title="..."&lt;/code&gt; isn't truncation, it's content.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;False negatives:&lt;/strong&gt; if every attribute on an element is short, the opening-tag-only fallback leaves &lt;em&gt;no marker at all&lt;/em&gt;. A &lt;code&gt;&amp;lt;html lang="en"&amp;gt;&lt;/code&gt; with its entire document body silently dropped looks perfectly clean.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The reliable signal ended up being structural: a non-void element whose snippet has no closing tag has lost its children. A string search for a marker was never going to be enough, and I only found that out because I read the library source instead of guessing what it "probably" does.&lt;/p&gt;

&lt;h3&gt;
  
  
  The fix: re-fetch the real element
&lt;/h3&gt;

&lt;p&gt;axe-core gives you a &lt;code&gt;target&lt;/code&gt; array — a selector path to the element. So after axe runs, and before the browser context closes, I go back to the live page and ask for the real &lt;code&gt;outerHTML&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;refetchOuterHtml&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;target&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;locator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;buildLocatorForTarget&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;target&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;locator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;first&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;el&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;el&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;outerHTML&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;html&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;string&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;evaluate did not return a string (outerHTML unavailable)&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;html&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;error&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;message&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I still need &lt;em&gt;some&lt;/em&gt; size cap for a prompt, so I wrote my own truncation — one that keeps the opening tag fully intact, says explicitly what was removed, and keeps the element closed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;marker&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;`\n&amp;lt;!-- rammendo: children omitted (&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;omittedChars&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; chars) --&amp;gt;\n`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;closing&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;tagName&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="s2"&gt;`&amp;lt;/&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;tagName&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;gt;`&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;html&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;openingTag&lt;/span&gt;&lt;span class="p"&gt;}${&lt;/span&gt;&lt;span class="nx"&gt;marker&lt;/span&gt;&lt;span class="p"&gt;}${&lt;/span&gt;&lt;span class="nx"&gt;closing&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;childrenOmitted&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The difference from the library's version isn't "truncate vs. don't truncate." It's that this truncation is &lt;em&gt;declared&lt;/em&gt;. The model can see that children were omitted, the result record carries &lt;code&gt;childrenOmitted: true&lt;/code&gt;, and the scorer can treat it accordingly instead of mistaking it for the full element.&lt;/p&gt;

&lt;p&gt;And crucially: if the re-fetch fails, I don't quietly fall back to axe's string. I fall back loudly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;warn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="s2"&gt;`[rammendo] outerHTML再取得失敗 (rule=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;violation&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;, url=&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;pageUrl&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;): &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;refetch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; — axeのnode.htmlにフォールバックします`&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;node&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;__rammendo&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;html&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;node&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;sourceKind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;axe_fallback&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every record carries a &lt;code&gt;sourceKind&lt;/code&gt;, and the summary report has a row that counts &lt;code&gt;axe_fallback&lt;/code&gt; and prints a large warning block if it's anything other than zero. A silent fallback would have recreated the exact lie I had just removed, the first time a selector failed to resolve.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug 2: I was counting other people's bugs as my customer's
&lt;/h2&gt;

&lt;p&gt;The second problem showed up in the same free run, in a different column.&lt;/p&gt;

&lt;p&gt;One site had a lot of ARIA violations — far more than the rest. Reading the raw records, almost all of them were inside an embedded third-party booking widget. axe-core scans &lt;em&gt;through&lt;/em&gt; iframes by default, which is the right default for an audit ("your users experience this page, including the widget"). It's the wrong default for my question, which was "can the AI fix the site owner's code?" The site owner can't edit markup that lives inside someone else's iframe.&lt;/p&gt;

&lt;p&gt;So that site's numbers were inflated by violations nobody on that site could fix, and any fix the LLM wrote for them would be a fix to code nobody could apply.&lt;/p&gt;

&lt;p&gt;The tempting fix is a heuristic: "if the class name contains the vendor's name, skip it." I explicitly didn't do that. Class names are guesses. Instead I used two mechanical signals:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;axe's &lt;code&gt;target&lt;/code&gt; array is longer than one entry when the element is inside a frame. The leading entries are the frame path.&lt;/li&gt;
&lt;li&gt;For the outermost frame, read its actual &lt;code&gt;src&lt;/code&gt;, resolve it against the page URL, and compare origins.
&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;thirdParty&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;iframeOrigin&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="nx"&gt;pageOrigin&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;inIframe&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;framePath&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;iframeSrc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;iframeOrigin&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;pageOrigin&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;thirdParty&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;originNote&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And when the signal is missing — an iframe with no &lt;code&gt;src&lt;/code&gt;, or &lt;code&gt;about:blank&lt;/code&gt;, or a &lt;code&gt;srcdoc&lt;/code&gt; — I refuse to guess:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;originNote&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;iframeにsrc属性が無いか、srcdoc/about:blank等の非URL値。推測でthird party判定はしない&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Third-party violations aren't deleted. They're reported in their own section, with the provider's origin, as "this is real, but it isn't in your code." Hiding them would be its own kind of dishonesty; mixing them into the fixable count was the other kind.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why measurement bugs lean toward "pass"
&lt;/h2&gt;

&lt;p&gt;Both bugs share a shape, and it's the reason I'm writing this up.&lt;/p&gt;

&lt;p&gt;A bug in the thing you're testing usually makes results worse, and bad results get investigated. A bug in the &lt;em&gt;instrument&lt;/em&gt; can make results better, and good results almost never get investigated. Nobody reruns an experiment that came out the way they hoped.&lt;/p&gt;

&lt;p&gt;Bug 1 is the clean example. The strict "BEFORE must match the input" check is a self-consistency check. If the input is wrong, self-consistency still holds — the model is perfectly consistent with garbage. Every check that compares an output to the harness's own input inherits whatever is wrong with the input.&lt;/p&gt;

&lt;p&gt;Bug 2 points the other way — inflated failure counts — but it has the same root: the instrument was answering a slightly different question from the one I asked. axe-core answers "what's wrong with this page as rendered?" I needed "what's wrong with this site owner's code?" The two differ by exactly the stuff inside third-party frames.&lt;/p&gt;

&lt;p&gt;In both cases, if I had gone straight to the paid run, I would have gotten a number, and the number would have been about my harness rather than about the AI.&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick checklist for "is my harness lying?"
&lt;/h2&gt;

&lt;p&gt;After this, I wrote down the questions I now ask before I trust any number an eval harness produces. They're cheap, and most of them take less time than reading this section.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Where does each input come from, exactly?&lt;/strong&gt; Not "from the scanner" — which field, produced by which function, with which defaults. If the answer involves a library, I want to have seen the line of code. In this case the answer was "a display string with a 300-character cap," and that one fact changed everything downstream.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What does a perfect score look like if the model does nothing clever at all?&lt;/strong&gt; If a model that simply echoes its input back would pass a check, that check is measuring the harness, not the model. My &lt;code&gt;BEFORE&lt;/code&gt; comparison failed this test: an echo passes it by construction. It's still a useful check — it rejects hallucinated rewrites — but it can't tell you whether the input was right.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What fraction of records took a fallback path?&lt;/strong&gt; If the answer isn't printed in the summary, I don't know it, and "I don't know" tends to quietly become "zero" in my head. Now it's a row in the report with a warning attached.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is my denominator the population I care about?&lt;/strong&gt; The third-party iframe problem was a denominator problem. The scanner's population was "everything rendered on the page"; mine was "everything the site owner can edit." Counting the wrong population produces a precise answer to a question nobody asked.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Would I notice if this check broke silently?&lt;/strong&gt; For each check I ask what the report would look like if it always returned "pass." If the honest answer is "about the same, maybe slightly better," that check needs a test of its own — a deliberately broken input that it has to reject.&lt;/p&gt;

&lt;p&gt;None of this requires a framework. It's a short list in a markdown file next to the harness, and I read it before every paid run.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I do now
&lt;/h2&gt;

&lt;p&gt;These are the rules I took away. None of them are novel; I just hadn't been following them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Before any paid evaluation, do a free run and read the raw inputs.&lt;/strong&gt;&lt;br&gt;
Not the summary table. The actual payloads that would have gone to the model. A dry-run mode costs almost nothing to add, and it's the only stage where the inputs exist without results to bias how you read them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Ask which direction the instrument would lie.&lt;/strong&gt;&lt;br&gt;
For every check, I now ask: "if this check is broken, does it fail too much or pass too much?" Checks that would pass too much get extra scrutiny, because nothing downstream will catch them. Self-consistency checks — output compared to the harness's own input — are the first suspects.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Read the library source for any data transformation you depend on.&lt;/strong&gt;&lt;br&gt;
"axe gives you the element's HTML" was a belief, not a fact. Opening &lt;code&gt;node_modules&lt;/code&gt; and finding &lt;code&gt;maxLength&lt;/code&gt; and &lt;code&gt;attrLimit&lt;/code&gt; took a few minutes. Anything a library does to your data — truncation, normalization, summarizing, deduplication — is worth confirming in code rather than in memory or docs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Make every fallback visible and countable.&lt;/strong&gt;&lt;br&gt;
If the good path fails and you fall back to a worse source, tag the record and count it in the report. A fallback that can't be seen in the summary is just the original bug with extra steps.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Separate "what the tool reports" from "what you're actually asking."&lt;/strong&gt;&lt;br&gt;
A scanner's scope (the rendered page, including iframes) and your question's scope (the code a specific person can change) are often slightly different. Write the question down explicitly, then check whether the instrument's default matches it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. When a signal is missing, record that it's missing.&lt;/strong&gt;&lt;br&gt;
For the iframe check, "no &lt;code&gt;src&lt;/code&gt;" doesn't mean first-party or third-party. It means unknown, and the record says so in a note. Guessing would have been quieter and worse.&lt;/p&gt;

&lt;h2&gt;
  
  
  The uncomfortable bit
&lt;/h2&gt;

&lt;p&gt;The scariest part of this isn't that the harness had bugs. Every harness has bugs. It's that I was about to use the result to make a go/no-go decision about the whole project.&lt;/p&gt;

&lt;p&gt;If the bugs had pushed the numbers down, I might have concluded "AI fixes aren't good enough" and dropped an idea because of a problem in my own wiring that took an afternoon to fix. If they had pushed the numbers up, I would have built on a foundation that wasn't there. Either way, the decision would have been made by my instrument, not by the thing I thought I was measuring.&lt;/p&gt;

&lt;p&gt;The free run didn't make the project succeed. It made the eventual measurement mean what I wanted it to mean. That's a much smaller claim, and it's the one I actually care about now.&lt;/p&gt;

&lt;p&gt;If you're building an eval harness for an LLM feature — especially one where the model's output gets compared to the input you constructed — I'd be curious how you guard against the same thing. Do you read raw payloads before every run, or do you have checks that catch a broken input automatically?&lt;/p&gt;

</description>
      <category>a11y</category>
      <category>javascript</category>
      <category>testing</category>
      <category>ai</category>
    </item>
    <item>
      <title>How We Secure DevReview (October 2026)</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Sun, 25 Oct 2026 08:37:42 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/how-we-secure-devreview-october-2026-26lm</link>
      <guid>https://dev.to/phi_blankslate/how-we-secure-devreview-october-2026-26lm</guid>
      <description>&lt;p&gt;&lt;em&gt;Originally published on the &lt;a href="https://www.getdevreview.com/blog/how-we-secure-devreview-2026-10" rel="noopener noreferrer"&gt;DevReview blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;DevReview reads your pull requests. That means you are trusting us with code that isn't public yet. This post lays out what we do to deserve that trust, what outside scanners say about us, and what we have not finished.&lt;/p&gt;

&lt;p&gt;This is our own write-up, not an audit. Scores are dated because they change.&lt;/p&gt;

&lt;h2&gt;
  
  
  What outside scanners say (measured 2026-10-24)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Qualys SSL Labs: A+ on one endpoint, A on the other.&lt;/strong&gt; Our site is served from two edge addresses; we report both rather than the better one. &lt;a href="https://www.ssllabs.com/ssltest/analyze.html?d=www.getdevreview.com" rel="noopener noreferrer"&gt;Run the test yourself&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mozilla HTTP Observatory: B (75), 11 of 12 checks passed.&lt;/strong&gt; The one we lose points on is Content Security Policy. Our CSP currently runs in &lt;strong&gt;Report-Only&lt;/strong&gt; mode: browsers report violations to us but do not block them yet. We are collecting reports first so that enforcing it doesn't break checkout or sign-in. We plan to switch to enforcement in early November 2026 and will update this post when we do. &lt;a href="https://developer.mozilla.org/en-US/observatory/analyze?host=www.getdevreview.com" rel="noopener noreferrer"&gt;Run the test yourself&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Your code
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;We don't keep your code.&lt;/strong&gt; DevReview fetches the diff it needs to review, sends it to the model, and posts comments. We don't store the diff, and we don't log the model's raw output.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Old data is deleted on a schedule.&lt;/strong&gt; A daily job removes installations that were never completed after 30 days, and error details from failed reviews after 90 days. We checked the first scheduled run in production on 2026-10-24.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What this means for you: if you uninstall, there isn't an archive of your source code sitting on our side.&lt;/p&gt;

&lt;h2&gt;
  
  
  The doors into our servers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Webhooks are verified before anything else.&lt;/strong&gt; GitHub and Stripe webhooks are checked for a valid signature first. Bodies over a fixed size (2 MB for GitHub, 1 MB for Stripe) are rejected with &lt;code&gt;413&lt;/code&gt; before we parse them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Errors don't leak internals.&lt;/strong&gt; If something fails, you get a generic message. Stack traces and database errors stay in our logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;HTTPS only.&lt;/strong&gt; HSTS is set for two years.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;New free accounts get a short wait.&lt;/strong&gt; GitHub accounts younger than 7 days can't use the free plan, which keeps throwaway accounts from burning review capacity.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Referral codes are random.&lt;/strong&gt; They come from a cryptographic random generator, not a counter you could guess.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  How we run things behind the scenes
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Two-step sign-in on every admin account&lt;/strong&gt; we use to run DevReview (hosting, database, payments, domain, email, source code). One provider (our domain registrar) only offers emailed codes rather than an authenticator app.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Email that can't easily be spoofed.&lt;/strong&gt; SPF, DKIM and DMARC all pass, and DMARC is set to quarantine mail that fails.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Breach monitoring&lt;/strong&gt; for our domain via Have I Been Pwned.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Secret scanning before every push.&lt;/strong&gt; A gitleaks hook blocks a push if it finds something that looks like a key.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Production dependencies:&lt;/strong&gt; &lt;code&gt;npm audit&lt;/code&gt; reports 0 known vulnerabilities as of 2026-10-22.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;We practiced restoring the database.&lt;/strong&gt; Our database keeps 6 hours of point-in-time history, and we ran a restore drill on a copy rather than trusting the docs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A written first-30-minutes incident card&lt;/strong&gt;, plus a ledger of every key we hold and how to rotate it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What is not done yet
&lt;/h2&gt;

&lt;p&gt;We'd rather say this plainly than have you find it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CSP is Report-Only&lt;/strong&gt;, as explained above.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-user rate limits and re-authentication for sensitive actions&lt;/strong&gt; are built but not shipped yet. They're waiting on a final review.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CAA DNS records&lt;/strong&gt; aren't set, because our DNS provider doesn't support them. We're looking at options.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Tips for your side
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Install DevReview only on the repositories you want reviewed, not "All repositories".&lt;/li&gt;
&lt;li&gt;Turn on two-factor authentication on your GitHub account if you haven't.&lt;/li&gt;
&lt;li&gt;If you leave, uninstall the GitHub App. That stops all access immediately.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Found something?
&lt;/h2&gt;

&lt;p&gt;Please tell us. Our &lt;a href="https://www.getdevreview.com/.well-known/security.txt" rel="noopener noreferrer"&gt;security.txt&lt;/a&gt; and &lt;a href="https://www.getdevreview.com/security" rel="noopener noreferrer"&gt;security page&lt;/a&gt; explain how to report a vulnerability. We read every report.&lt;/p&gt;

</description>
      <category>security</category>
      <category>webdev</category>
      <category>saas</category>
      <category>nextjs</category>
    </item>
    <item>
      <title>Stripe marks a subscription "active" before anyone has paid. Here's how I tested a reward that has to wait for real money.</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Thu, 22 Oct 2026 01:09:06 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/stripe-marks-a-subscription-active-before-anyone-has-paid-heres-how-i-tested-a-reward-that-has-49p0</link>
      <guid>https://dev.to/phi_blankslate/stripe-marks-a-subscription-active-before-anyone-has-paid-heres-how-i-tested-a-reward-that-has-49p0</guid>
      <description>&lt;p&gt;I added a referral program to my SaaS. The rule sounds simple: when someone you referred becomes a paying annual customer, you get an account credit. Not when they sign up, not when they start a trial, but when money has actually moved.&lt;/p&gt;

&lt;p&gt;Writing that logic took an afternoon. Proving it worked took much longer, and most of that time went into one uncomfortable fact: the only event that matters, "the referred customer paid", happens at the end of a trial, on an annual plan, and you only get one real shot at it per customer. The next natural billing event for that subscription is a year away.&lt;/p&gt;

&lt;p&gt;This post is about how I tested that path without charging a real card, without creating fake accounts on any platform, and without touching my production database. The tools were Stripe Test Clocks and a disposable Postgres branch. Along the way the test surfaced two things I would not have found by reading the code: a &lt;code&gt;$0&lt;/code&gt; trial invoice counts as &lt;code&gt;paid&lt;/code&gt;, and on recent Stripe API versions the invoice's &lt;code&gt;subscription&lt;/code&gt; field is simply &lt;code&gt;null&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The stack is Next.js on Vercel, Drizzle, and Neon Postgres, but the testing approach doesn't depend on any of that.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug that shaped the design: "active" is not "paid"
&lt;/h2&gt;

&lt;p&gt;My first version of the reward check looked at the subscription status. If it was &lt;code&gt;active&lt;/code&gt; and on the annual price, the referrer got their credit.&lt;/p&gt;

&lt;p&gt;A security review caught the problem before it shipped. When a trial ends, Stripe moves the subscription out of &lt;code&gt;trialing&lt;/code&gt; and creates the first real invoice. The &lt;code&gt;customer.subscription.updated&lt;/code&gt; event with &lt;code&gt;status: "active"&lt;/code&gt; can arrive &lt;em&gt;before&lt;/em&gt; the payment on that invoice has succeeded. If the card is declined, the subscription later goes to &lt;code&gt;past_due&lt;/code&gt;, but by then my code had already handed out a credit for a payment that never happened.&lt;/p&gt;

&lt;p&gt;So the rule changed from "the subscription is active" to "the subscription is active &lt;strong&gt;and&lt;/strong&gt; its latest invoice is paid &lt;strong&gt;and&lt;/strong&gt; the amount paid is greater than zero". The trigger also moved: the main entry point is now the &lt;code&gt;invoice.paid&lt;/code&gt; webhook, with &lt;code&gt;customer.subscription.updated&lt;/code&gt; kept only as a fallback.&lt;/p&gt;

&lt;p&gt;Here is the eligibility check as it exists in the code today (comments translated from my original notes):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;rewardReferralIfEligible&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;stripeSubscriptionId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="k"&gt;void&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stripeSubscriptionId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;stripeSubscriptionId&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;referral&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="c1"&gt;// Primary idempotency key: if a transaction ID is already recorded, the credit was granted.&lt;/span&gt;
  &lt;span class="c1"&gt;// Confirm "rewarded" without calling Stripe.&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stripeBalanceTransactionId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;rewarded&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;rewardedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;rewardedAt&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;and&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;getStripe&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

  &lt;span class="c1"&gt;// Re-check eligibility against Stripe's actual state. Don't trust the caller's pre-filtering.&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;sub&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;retrieve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;stripeSubscriptionId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;expand&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;latest_invoice&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;active&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;priceId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;items&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]?.&lt;/span&gt;&lt;span class="nx"&gt;price&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;priceId&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;priceId&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="nx"&gt;PLAN_PRICE_IDS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pro_annual&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;latestInvoice&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt;
    &lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;latest_invoice&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;object&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;latest_invoice&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;
      &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;latest_invoice&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nx"&gt;Stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Invoice&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;latestInvoice&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;latestInvoice&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;paid&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;latestInvoice&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;amount_paid&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="c1"&gt;// Compare-and-swap: only one process can move pending -&amp;gt; rewarded.&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;updated&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;rewarded&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;rewardedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;and&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;returning&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="c1"&gt;// ... resolve the referrer's Stripe customer, check for an existing credit, grant it&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every condition in that function is a claim about how Stripe behaves at the end of a trial. I had read the docs, and a reviewer had read the code. Neither of those is evidence. I needed to watch a trial end.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the obvious test plans didn't work
&lt;/h2&gt;

&lt;p&gt;Before I found a setup I liked, I talked myself out of three others. Each failure taught me something about what the test actually needed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Plan 1: subscribe for real in production, then cancel during the trial.&lt;/strong&gt; I had used this trick before to test checkout, because cancelling inside the trial costs nothing. But it tests nothing here. The reward only fires on &lt;code&gt;invoice.paid&lt;/code&gt; after a successful charge. Cancel during the trial and the code under test never runs. A free test of the wrong path is not a cheap test of the right one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Plan 2: subscribe for real, but give the test customer a balance credit large enough to cover almost the whole invoice.&lt;/strong&gt; This looked clever until I reread my own guard: &lt;code&gt;amount_paid &amp;gt; 0&lt;/code&gt;. If the credit covers the full amount, Stripe records the invoice as paid with nothing actually charged, and my code correctly refuses to reward it. To get through the guard I would have had to leave a small remainder on the card. That's workable, but it's a real charge on a real card in a real account, and it still runs against production data.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Plan 3, the one that actually killed the real-money approach: the referred customer has to be a different person.&lt;/strong&gt; My self-referral guard rejects a referral when the referrer and the referred user share a user ID or an email address. That is the point of the guard. To test the happy path end to end in production, I needed a second account that signs in through the same OAuth provider my app uses. I run exactly one account on that platform on purpose. Creating a second account to test my own app is precisely the kind of thing that gets accounts suspended, and I wasn't going to trade that risk for a test.&lt;/p&gt;

&lt;p&gt;Once all three were off the table, the shape of the right test was clear. I didn't need a real person or a real card. I needed Stripe to believe that a trial had ended and a payment had succeeded, and I needed my database to contain the rows that would exist if a real referral had happened. Both of those can be faked safely, as long as the fakes are fenced off from everything real.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup: Stripe test mode, a Test Clock, and a throwaway database branch
&lt;/h2&gt;

&lt;p&gt;The test environment had three pieces:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Stripe test mode.&lt;/strong&gt; Customers, subscriptions, and invoices created with a &lt;code&gt;sk_test_&lt;/code&gt; key. Test card tokens like &lt;code&gt;pm_card_visa&lt;/code&gt; make payments succeed without any real card.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A Stripe Test Clock.&lt;/strong&gt; A test-mode customer can be attached to a clock that starts frozen at a chosen time. You then advance the clock, and Stripe runs every billing event that would have happened in between: the trial ends, the first invoice is created, payment is attempted.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A Neon branch of the production database.&lt;/strong&gt; Branching gives a copy of the schema (and data) that can be written to freely and then deleted. The production database is never connected.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I wrote one script with a subcommand for each step, so every step could be run, inspected, and repeated on its own: &lt;code&gt;setup&lt;/code&gt;, &lt;code&gt;reward&lt;/code&gt;, &lt;code&gt;advance&lt;/code&gt;, &lt;code&gt;assert&lt;/code&gt;, &lt;code&gt;reset-pending&lt;/code&gt;, &lt;code&gt;cleanup&lt;/code&gt;. Each subcommand prints a single JSON line with IDs, statuses, and counts, and never prints connection strings or keys.&lt;/p&gt;

&lt;h3&gt;
  
  
  Guards first
&lt;/h3&gt;

&lt;p&gt;A test script that creates customers and writes rows is one environment variable away from doing that in production. So before any subcommand does anything, it runs the same guard function:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Never write to a DATABASE_URL whose hostname doesn't start with this&lt;/span&gt;
&lt;span class="c1"&gt;// (prevents accidental writes to production or other branches)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;S8_BRANCH_ENDPOINT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;ep-xxxxxxxx&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// the test branch's endpoint, redacted here&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;assertGuards&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Stripe&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nl"&gt;priceId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;secretKey&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;STRIPE_SECRET_KEY&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;secretKey&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;secretKey&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startsWith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;sk_test_&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;printAndExit&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;STRIPE_SECRET_KEY_NOT_TEST_MODE&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;connectionString&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;DATABASE_URL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;connectionString&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;printAndExit&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;DATABASE_URL_NOT_SET&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="na"&gt;hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;hostname&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;connectionString&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;hostname&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;printAndExit&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;DATABASE_URL_UNPARSEABLE&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;hostname&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startsWith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;S8_BRANCH_ENDPOINT&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;printAndExit&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;DATABASE_URL_NOT_S8_BRANCH&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;priceId&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;STRIPE_PRICE_PRO_ANNUAL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;priceId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;printAndExit&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;STRIPE_PRICE_PRO_ANNUAL_NOT_SET&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;getStripe&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;price&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;prices&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;retrieve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;priceId&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;price&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;livemode&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nf"&gt;printAndExit&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;PRICE_IS_LIVEMODE&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;priceId&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There are three independent checks: the key prefix, the database hostname, and the &lt;code&gt;livemode&lt;/code&gt; flag on the price Stripe actually returns. The database check is an allowlist of one specific branch, not a denylist of production. A denylist would quietly allow any other database I happened to point it at.&lt;/p&gt;

&lt;p&gt;The connection string for the branch went into a separate, git-ignored env file that the script loads on top of the normal one. I never needed to see its value, and the script never printed it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Creating the world
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;setup&lt;/code&gt; builds everything a real referral would have produced:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;clock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;testHelpers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;testClocks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;frozen_time&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;floor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;s8-referral-e2e&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;customerB&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;customers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;email&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;s8-referred@example.test&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;S8 Referred&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;test_clock&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;clock&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;payment_method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pm_card_visa&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;invoice_settings&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;default_payment_method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pm_card_visa&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;s8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;1&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;sub&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;customer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;customerB&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;items&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt; &lt;span class="na"&gt;price&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;priceId&lt;/span&gt; &lt;span class="p"&gt;}],&lt;/span&gt;
  &lt;span class="na"&gt;trial_period_days&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;discounts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt; &lt;span class="na"&gt;coupon&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;REFERRAL_COUPON_ID&lt;/span&gt; &lt;span class="p"&gt;}],&lt;/span&gt;
  &lt;span class="na"&gt;expand&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;latest_invoice&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the database side, inside one transaction, it inserts two users (referrer A and referred B), a &lt;code&gt;trialing&lt;/code&gt; subscription row for B, and a &lt;code&gt;pending&lt;/code&gt; referral row linking them. The &lt;code&gt;.example.test&lt;/code&gt; email addresses are reserved for testing and can't belong to anyone.&lt;/p&gt;

&lt;p&gt;One detail was deliberate: &lt;strong&gt;user A gets no subscription row at all.&lt;/strong&gt; On the free plan in my app, a user has no Stripe customer and no subscription row until they first check out. The reward code has to create a Stripe customer for the referrer on demand before it can credit them. That lazy path is exactly the kind of branch that never runs in development, so the test environment was built to force it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test 1: the negative test, during the trial
&lt;/h2&gt;

&lt;p&gt;Before advancing the clock, I ran the reward function against the subscription while it was still in its trial. The expected outcome was: nothing happens, and the referral stays &lt;code&gt;pending&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;It did stay &lt;code&gt;pending&lt;/code&gt;. The interesting part was &lt;em&gt;why&lt;/em&gt;. It would be easy to assume that during a trial the latest invoice is missing, or unpaid. It was neither. The subscription already had a latest invoice, its status was &lt;code&gt;paid&lt;/code&gt;, and its &lt;code&gt;amount_paid&lt;/code&gt; was &lt;code&gt;0&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;When a subscription starts with a trial, Stripe creates a zero-amount invoice for the trial period and marks it paid. If my guard had been &lt;code&gt;latestInvoice.status === "paid"&lt;/code&gt; on its own, which reads like a perfectly reasonable definition of "the customer paid", the reward would have fired the moment the trial &lt;em&gt;started&lt;/em&gt;, with no money involved at all.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;amount_paid &amp;gt; 0&lt;/code&gt; half of the condition was the only thing standing between a correct reward and a free one. It looks like a minor extra check when you read the code. On the most ordinary path in the system, the trial, it is the check doing all the work.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test 2: advancing the clock past the end of the trial
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;advance&lt;/code&gt; moves the Test Clock to a point a few hours after the trial ends, then waits for Stripe to finish processing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;testHelpers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;testClocks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;advance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;testClockId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;frozen_time&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;trialEnd&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;3600&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;startedAt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;ready&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;startedAt&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nx"&gt;ADVANCE_TIMEOUT_MS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;clock&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;testHelpers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;testClocks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;retrieve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;testClockId&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;clock&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;ready&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;ready&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ADVANCE_POLL_MS&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Advancing a clock is asynchronous. The API call returns right away, and the clock reports a non-ready status while Stripe works through the simulated time. Reading the subscription before the clock is &lt;code&gt;ready&lt;/code&gt; gives you a state that is halfway through the transition. That's another way to fool yourself, so the script polls until &lt;code&gt;ready&lt;/code&gt; and fails loudly on timeout.&lt;/p&gt;

&lt;p&gt;After the advance, the subscription was &lt;code&gt;active&lt;/code&gt;, the latest invoice was &lt;code&gt;paid&lt;/code&gt;, the amount paid was exactly the discounted annual price, and the &lt;code&gt;billing_reason&lt;/code&gt; was &lt;code&gt;subscription_cycle&lt;/code&gt;. That's the state my production webhook would see.&lt;/p&gt;

&lt;h3&gt;
  
  
  The surprise: &lt;code&gt;invoice.subscription&lt;/code&gt; was null
&lt;/h3&gt;

&lt;p&gt;The webhook handler for &lt;code&gt;invoice.paid&lt;/code&gt; has to find out which subscription the invoice belongs to. Older Stripe API versions put that on &lt;code&gt;invoice.subscription&lt;/code&gt;. Newer versions, starting with the release Stripe calls Basil, moved it to &lt;code&gt;invoice.parent.subscription_details.subscription&lt;/code&gt;. My SDK version and my webhook endpoint's API version were not the same, so I had written the extractor to check both:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Invoice -&amp;gt; subscription ID. Basil and later: invoice.parent.subscription_details.subscription.&lt;/span&gt;
&lt;span class="c1"&gt;// Older versions: invoice.subscription. Check both, because the SDK's pinned version and the&lt;/span&gt;
&lt;span class="c1"&gt;// webhook endpoint's version differ.&lt;/span&gt;
&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;extractSubscriptionId&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;invoice&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;Invoice&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;inv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;invoice&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nx"&gt;unknown&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;subscription&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;subscription_details&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;subscription&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;inv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;subscription_details&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;subscription&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="nx"&gt;inv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;subscription&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;string&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;raw&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When I wrote that fallback, I wasn't sure it was needed. It might just have been defensive noise. So the &lt;code&gt;advance&lt;/code&gt; step logged both paths separately on the real invoice. The new path had the subscription ID. The old path was &lt;code&gt;null&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Without the new path, &lt;code&gt;handleInvoicePaid&lt;/code&gt; would have returned early on every invoice. No error, no log line, just a referral sitting in &lt;code&gt;pending&lt;/code&gt; forever. Because this is an annual plan, "forever" is not an exaggeration: there is no second chance until the renewal a year later. That's the kind of bug a Test Clock is good at finding, because it hands you a real Stripe object of the current shape rather than the shape you remember.&lt;/p&gt;

&lt;h2&gt;
  
  
  Test 3: granting the reward, and proving it happened exactly once
&lt;/h2&gt;

&lt;p&gt;With the subscription in its paid state, I ran the reward path through my manual sweep script, first as a dry run and then for real. Then &lt;code&gt;assert&lt;/code&gt; checked the outcome from three independent angles:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the &lt;code&gt;referrals&lt;/code&gt; row for this subscription was &lt;code&gt;rewarded&lt;/code&gt;, and there was exactly one row;&lt;/li&gt;
&lt;li&gt;the referrer now had a subscription row and a Stripe customer, created lazily by the reward code;&lt;/li&gt;
&lt;li&gt;the referrer's Stripe balance held exactly the expected credit, and exactly one balance transaction contained the marker string &lt;code&gt;(referral &amp;lt;id&amp;gt;)&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;"Exactly one" was the whole point. A reward system that double-pays is worse than one that pays late, because a late payment can be fixed with a sweep and a double payment means asking someone to give money back.&lt;/p&gt;

&lt;h3&gt;
  
  
  Re-running everything
&lt;/h3&gt;

&lt;p&gt;First, I simply ran the sweep again, and then called the reward function directly. Nothing changed. The referral was no longer &lt;code&gt;pending&lt;/code&gt;, so the function returned immediately.&lt;/p&gt;

&lt;p&gt;Second, I tested the failure mode that worried me more. The reward code calls Stripe with an idempotency key, so a retried request inside Stripe's retention window returns the original result instead of creating a second credit. But Stripe's docs say idempotency keys can be removed after they are &lt;a href="https://docs.stripe.com/api/idempotent_requests" rel="noopener noreferrer"&gt;at least 24 hours old&lt;/a&gt;. If the Stripe call succeeded but my process lost the response, the code rolls the referral back to &lt;code&gt;pending&lt;/code&gt;. If the retry then happens a day later, the idempotency key no longer protects anything.&lt;/p&gt;

&lt;p&gt;To simulate that, &lt;code&gt;reset-pending&lt;/code&gt; forces the row back into the "we don't know what happened" state:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;updated&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;rewardedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;stripeBalanceTransactionId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;referralId&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;returning&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then I ran the sweep again. This time the reward code reached its second line of defense. It listed the referrer's balance transactions on Stripe, found the existing credit by its marker, recorded that transaction's ID, and confirmed &lt;code&gt;rewarded&lt;/code&gt; without creating a new credit. The balance transaction count stayed at one.&lt;/p&gt;

&lt;p&gt;That second line of defense came out of a review, and up to this point it had only ever been read, never executed. Now it had run against a real Stripe API response.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cleanup is part of the test
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;cleanup&lt;/code&gt; deletes the Test Clock, then deletes the test referral, subscription, and user rows from the branch and prints how many rows of each it removed. After that I deleted the database branch itself and the temporary env file that held its connection string.&lt;/p&gt;

&lt;p&gt;I treat cleanup as a test step rather than an afterthought. A leftover database branch with a real-looking schema, and an env file pointing at it, is something future-me could connect to by mistake months later. A test that leaves debris behind has side effects, and side effects are exactly what I was trying to keep away from production in the first place.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this test did not prove
&lt;/h2&gt;

&lt;p&gt;This setup proved the reward logic: the eligibility conditions, the lazy customer creation, the compare-and-swap, the double-payment defenses, and the invoice shape on the current API version. It did &lt;strong&gt;not&lt;/strong&gt; prove:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;that the production webhook endpoint actually receives &lt;code&gt;invoice.paid&lt;/code&gt;. My script called the reward function directly instead of going through the webhook, so it can't see the endpoint's configuration at all. This one wasn't hypothetical: when the switch to &lt;code&gt;invoice.paid&lt;/code&gt; was made, a review added "confirm the endpoint is subscribed to &lt;code&gt;invoice.paid&lt;/code&gt;" as a separate pre-launch check. When I looked at the production endpoint, it &lt;em&gt;wasn't&lt;/em&gt; subscribed, and I had to add the event by hand;&lt;/li&gt;
&lt;li&gt;that the live Checkout page applies the referral discount correctly (I checked that separately, by creating a real Checkout Session with the same parameters and looking at the page, without completing it);&lt;/li&gt;
&lt;li&gt;that a real second person can go through the whole flow in the UI.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I wrote those gaps down next to the test results, in the same document. A test report that lists only what passed invites the next reader, usually me, to assume it covered everything.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd generalize from this
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Any condition based on "paid" needs a test that runs while a trial is still in progress.&lt;/strong&gt; Zero-amount invoices are real invoices with a real &lt;code&gt;paid&lt;/code&gt; status. If your logic means "money moved", say so in the code with an amount check, and make sure a test exercises the trial state, not just the after-trial state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Time-dependent billing logic deserves a clock you control.&lt;/strong&gt; If the event you care about happens once per customer, at the end of a trial or a renewal, don't wait for it and don't fake it by calling your handler with a hand-written payload. A Test Clock gives you real Stripe objects in their real current shape, which is exactly what exposed the &lt;code&gt;invoice.subscription: null&lt;/code&gt; problem.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fence your test harness with allowlists, not hope.&lt;/strong&gt; Key prefix, a specific database hostname, and the &lt;code&gt;livemode&lt;/code&gt; flag of an object fetched from the API. Three cheap checks, each one independent of the others, all of them run before the first write.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Test your second line of defense by deliberately breaking the first.&lt;/strong&gt; Idempotency keys expire, responses get lost, rows get rolled back. If your defense against that is "look for an existing record before writing", the only way to know it works is to put the system into the ambiguous state on purpose and run it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Build the code path you can't reach in development.&lt;/strong&gt; My lazy customer creation only runs for a referrer who has never checked out. In normal development I always have a subscription, so it would never have run. The test data was shaped specifically to force that branch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Write down what the test didn't cover, and give each gap its own check.&lt;/strong&gt; The most important production misconfiguration in this feature, a webhook endpoint not subscribed to the event that triggers the reward, sat entirely outside what my test could see. A green test run would never have revealed it. It was caught only because it had its own line on the pre-launch checklist, separate from the test.&lt;/p&gt;

&lt;p&gt;Most of the harness is guards and output formatting. The Test Clock calls themselves are a small fraction of it. That ratio seems about right to me: moving time is easy, and making sure you only moved time where you meant to is the part that takes the effort.&lt;/p&gt;

</description>
      <category>stripe</category>
      <category>typescript</category>
      <category>webdev</category>
      <category>api</category>
    </item>
    <item>
      <title>I moved my webhook work into waitUntil() to fix a timeout. The failures didn't go away, they just stopped being visible.</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Wed, 14 Oct 2026 01:11:34 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/i-moved-my-webhook-work-into-waituntil-to-fix-a-timeout-the-failures-didnt-go-away-they-just-2nd</link>
      <guid>https://dev.to/phi_blankslate/i-moved-my-webhook-work-into-waituntil-to-fix-a-timeout-the-failures-didnt-go-away-they-just-2nd</guid>
      <description>&lt;p&gt;My GitHub App takes a pull request, asks an LLM to review it, and posts the comments back. The first time I ran it end to end against a real repository, the review showed up on the PR exactly as intended. Then I opened GitHub's webhook delivery log and found the delivery marked as failed.&lt;/p&gt;

&lt;p&gt;Both things were true at once: the feature worked, and the delivery had failed. That contradiction is the interesting part, and the fix I reached for first created a worse problem than the one it solved. This post walks through both, with the code, because the second problem is the kind that doesn't announce itself.&lt;/p&gt;

&lt;p&gt;The stack is Next.js (App Router) on Vercel, Drizzle and Postgres, but nothing below depends on those. If you run slow work behind a webhook on a serverless platform, you will meet the same shape.&lt;/p&gt;

&lt;h2&gt;
  
  
  The symptom: a successful review and a failed delivery
&lt;/h2&gt;

&lt;p&gt;The handler was the obvious one. Verify the signature, parse the payload, do the work, return 200:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pull_request&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;handlePullRequestEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;handlePullRequestEvent&lt;/code&gt; fetches the changed files, calls the model, and posts a review. The model call alone regularly takes longer than ten seconds.&lt;/p&gt;

&lt;p&gt;GitHub's webhook best-practices page says a delivery should respond within 10 seconds, and in my notes from this incident GitHub does not automatically redeliver a failed one. So the sequence was:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;GitHub sends the &lt;code&gt;pull_request&lt;/code&gt; event.&lt;/li&gt;
&lt;li&gt;My handler is still waiting on the LLM when GitHub gives up at about ten seconds and records the delivery as failed.&lt;/li&gt;
&lt;li&gt;My function keeps running anyway, because the serverless runtime doesn't know the caller hung up. It finishes, posts the review, and writes the result to the database.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So the user-visible outcome was fine, and the only evidence of a problem was a red mark in a log that nobody looks at until something else goes wrong. That is a bad state to leave a system in even when the feature technically works: the next time a delivery fails for a real reason, it will look exactly like this one.&lt;/p&gt;

&lt;p&gt;The cause is simple. I was doing work whose duration I don't control inside the request/response cycle of a caller that gives me a fixed budget.&lt;/p&gt;

&lt;h2&gt;
  
  
  The obvious fix
&lt;/h2&gt;

&lt;p&gt;Acknowledge immediately, do the work afterwards. On Vercel the primitive for "keep running after the response is sent" is &lt;code&gt;waitUntil&lt;/code&gt; from &lt;code&gt;@vercel/functions&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;waitUntil&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;@vercel/functions&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// ...&lt;/span&gt;
&lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pull_request&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="c1"&gt;// Generating the review often takes longer than 10 seconds, which is longer&lt;/span&gt;
  &lt;span class="c1"&gt;// than GitHub's delivery timeout (about 10s, one attempt). ACK immediately&lt;/span&gt;
  &lt;span class="c1"&gt;// and keep working in the background.&lt;/span&gt;
  &lt;span class="nf"&gt;waitUntil&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nf"&gt;handlePullRequestEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="k"&gt;catch&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;PR review background processing failed:&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;deliveryId&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everything before this point is cheap and synchronous: verify the HMAC signature, parse the JSON. Then the response goes out, and the review is generated in the background.&lt;/p&gt;

&lt;p&gt;It type-checked and the change was small, so I was pleased with it. Before shipping, I ran a security review pass over the diff, and it pointed at the exact line I was pleased with. After I addressed that finding (below), I deployed, re-ran the end-to-end test, and the delivery log showed green.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this is worse than it looks
&lt;/h2&gt;

&lt;p&gt;Before the change, a failure had a natural place to show up. If the handler threw, GitHub got a 500, and the delivery was red. The work and the acknowledgement were the same event, so a problem with one was a problem with the other.&lt;/p&gt;

&lt;p&gt;After the change, the acknowledgement says only "I received your payload and the signature was valid." It says nothing about whether the work will finish. The failure modes of the background job are now invisible to the caller by construction:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The function hits its platform time limit and is killed.&lt;/li&gt;
&lt;li&gt;A deploy replaces the running instance.&lt;/li&gt;
&lt;li&gt;The process runs out of memory.&lt;/li&gt;
&lt;li&gt;An upstream call hangs and never returns.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of those run your &lt;code&gt;.catch&lt;/code&gt;. A killed process doesn't get to log anything or update any row. The delivery log stays green, because the delivery succeeded. What's left is whatever state you wrote before you started the work.&lt;/p&gt;

&lt;p&gt;In my case that state was a row in the &lt;code&gt;reviews&lt;/code&gt; table. I create it with &lt;code&gt;status = 'pending'&lt;/code&gt; before returning, and the background job moves it to &lt;code&gt;completed&lt;/code&gt; or &lt;code&gt;failed&lt;/code&gt; at the end. If the job dies in the middle, the row stays &lt;code&gt;pending&lt;/code&gt; for good.&lt;/p&gt;

&lt;p&gt;That sounds like a cosmetic problem, a spinner that never stops. It isn't, for a reason specific to how I count usage. A pending row counts toward the user's monthly quota, deliberately:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Rows that count toward the quota:&lt;/span&gt;
&lt;span class="c1"&gt;//   pending (in progress) counts too. If only completed rows were counted,&lt;/span&gt;
&lt;span class="c1"&gt;//   concurrent webhooks that arrive before an earlier review finishes would all&lt;/span&gt;
&lt;span class="c1"&gt;//   be judged to be within the limit, making the free tier effectively unlimited.&lt;/span&gt;
&lt;span class="c1"&gt;//   failed rows are excluded in principle, but a row that failed after the&lt;/span&gt;
&lt;span class="c1"&gt;//   Claude API was called (tokens_used &amp;gt; 0) still counts...&lt;/span&gt;
&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;countsTowardQuota&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;or&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;ne&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;failed&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;gt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tokensUsed&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I count pending rows because otherwise a burst of simultaneous events all look like they fit under the limit. That's correct, and it means a row that is stuck as pending is not just an ugly record. It permanently consumes someone's allowance. A killed background job turns into a user being told they've used reviews they never received.&lt;/p&gt;

&lt;p&gt;So the second lesson, after "don't hold the caller hostage," is that moving work out of the request cycle doesn't remove the failure modes. It moves them to a place where nothing is watching.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 1: make the work end before the platform ends it
&lt;/h2&gt;

&lt;p&gt;The first thing to do about a job that can be killed silently is to make sure it gets a chance to fail loudly instead. That means giving it an explicit time budget that is shorter than the platform's.&lt;/p&gt;

&lt;p&gt;Without an explicit limit, the route depends on the platform default, and my comment in the code says what I found out about that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Give the background processing (PR file fetch + Claude API review generation +&lt;/span&gt;
&lt;span class="c1"&gt;// GitHub post) enough time. Leaving this unset depends implicitly on the Vercel&lt;/span&gt;
&lt;span class="c1"&gt;// default and fails silently when interrupted (reviews stay "pending" and become&lt;/span&gt;
&lt;span class="c1"&gt;// undetectable).&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;maxDuration&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And the model client gets a timeout that is strictly below it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Keep this shorter than maxDuration (the webhook background limit) so a hung&lt;/span&gt;
&lt;span class="c1"&gt;// API call surfaces as an error before the platform timeout (silent-fail guard).&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Anthropic&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;apiKey&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;90&lt;/span&gt;&lt;span class="nx"&gt;_000&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The ordering matters more than the specific numbers. If the client timeout is longer than the function limit, the platform wins the race, kills the process, and nothing runs &lt;code&gt;catch&lt;/code&gt;. If the client timeout is shorter, the SDK throws, my &lt;code&gt;catch&lt;/code&gt; block runs, and the row is marked &lt;code&gt;failed&lt;/code&gt; with an error message. A timeout I control turns into a status I can query. A timeout the platform controls turns into silence.&lt;/p&gt;

&lt;p&gt;There is one caveat I want to be upfront about, because it's the kind of thing I'd want to know if I were copying this. A 90-second timeout on the client is a per-request limit. As far as I know, the Anthropic SDK retries failed requests by default, a couple of times. I have not set that option in this codebase and I have not tested what happens when a request times out and is retried. If retries do apply to timeouts, then the total time spent could be several multiples of 90 seconds, which is well beyond the function limit and would put the platform back in charge. I'm flagging that as something I still need to check rather than as a finding. The general point holds either way: when you stack a timeout under a deadline, budget the total across all attempts, not the single attempt.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 2: assume the first layer will sometimes fail anyway
&lt;/h2&gt;

&lt;p&gt;Timeouts catch the hung call. They don't catch a deploy that replaces the instance mid-flight, or an out-of-memory kill, because in those cases no code of mine gets to run at all.&lt;/p&gt;

&lt;p&gt;So the cleanup can't live in the dying process. It has to be done by something else, later. I didn't want to add a scheduler and a new moving part for a problem this small, so the reaper runs lazily: the next time the same user's webhook arrives, it sweeps their stale rows before counting anything.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Turn stuck pending rows (function cut off, OOM, replaced by a deploy, never&lt;/span&gt;
&lt;span class="c1"&gt;// reaching completed/failed) into failed. Rows that died before the Claude call&lt;/span&gt;
&lt;span class="c1"&gt;// (tokens_used=0) drop out of the quota; rows that died after it (the webhook&lt;/span&gt;
&lt;span class="c1"&gt;// writes a sentinel tokens_used&amp;gt;=1) stay in the quota even once failed.&lt;/span&gt;
&lt;span class="c1"&gt;// The threshold is 10 minutes, comfortably longer than maxDuration (120s).&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;STALE_PENDING_MINUTES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;sweepStalePending&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Pick&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;update&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;
&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="k"&gt;void&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;cutoff&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;STALE_PENDING_MINUTES&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="nx"&gt;_000&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
      &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;failed&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;errorMessage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`stale pending swept (older than &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;STALE_PENDING_MINUTES&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;m)`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;completedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="nf"&gt;and&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;lt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;createdAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;cutoff&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three decisions in there are worth spelling out.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The threshold is derived from the limit.&lt;/strong&gt; Ten minutes is not a round number I liked. It has to be comfortably larger than &lt;code&gt;maxDuration&lt;/code&gt;, otherwise the sweeper could mark a healthy, still-running job as failed while it is still working. The sweep is only safe because "older than ten minutes" means "the platform has already killed it", and that holds only as long as the threshold stays above the function limit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It runs inside the same transaction and the same lock as the check it protects.&lt;/strong&gt; The sweep is called from the function that reserves a review slot, after taking a per-user advisory lock:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sql&lt;/span&gt;&lt;span class="s2"&gt;`select pg_advisory_xact_lock(&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;LOCK_NS_USER&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;, hashtext(&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;))`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;// Reap stale pending rows under the lock, then count (serialized against&lt;/span&gt;
&lt;span class="c1"&gt;// concurrent runs for the same user)&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;sweepStalePending&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;userId&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The point of doing the sweep under the lock is that the count which decides "is this user over the limit" is computed on cleaned-up data, and no other request for that user can interleave between the cleanup and the count.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Whether a swept row still costs the user depends on how far it got.&lt;/strong&gt; This is the part I like least to have to think about, and the part I'm happiest I did think about. A job that died before it called the model has cost nothing, so its row should fall out of the quota. A job that died after the model call started has cost real money, and if it were free it would be an exploit: anyone who could make the post-generation step fail on purpose would get unlimited generations. So the webhook writes a sentinel immediately before calling the model:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Write a sentinel (tokens_used=1) just before the Claude call. If the call&lt;/span&gt;
&lt;span class="c1"&gt;// times out or the function is cut off before completing, the row still counts&lt;/span&gt;
&lt;span class="c1"&gt;// toward the quota as "generation started, cost may have been incurred". With&lt;/span&gt;
&lt;span class="c1"&gt;// tokens_used=0, a failed/swept row would free the quota and let someone abort&lt;/span&gt;
&lt;span class="c1"&gt;// generations to burn Claude cost for free.&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;tokensUsed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;}).&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;reviewRecord&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="nx"&gt;tokensUsed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The sweeper then doesn't need to know anything about what the job was doing. It just flips &lt;code&gt;pending&lt;/code&gt; to &lt;code&gt;failed&lt;/code&gt;, and the previously written &lt;code&gt;tokens_used&lt;/code&gt; decides whether that row still counts. The state machine carries the accounting.&lt;/p&gt;

&lt;h2&gt;
  
  
  The reaper created a new problem: a retry budget it can exhaust
&lt;/h2&gt;

&lt;p&gt;Once stuck rows get turned into &lt;code&gt;failed&lt;/code&gt;, something else has to decide whether that commit may be reviewed again. A webhook can be redelivered, and a user can close and reopen a PR, so the slot reservation does two checks before it allows another run for the same &lt;code&gt;(repository, PR, commit)&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// 1-a: already delivered (completed / pending) -&amp;gt; reject immediately&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;delivered&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;count&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;and&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sameCommit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;ne&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;failed&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;delivered&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]?.&lt;/span&gt;&lt;span class="nx"&gt;count&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;allowed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;used&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;record&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;duplicate&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note that &lt;code&gt;pending&lt;/code&gt; counts as "delivered" here. That is what stops a redelivered webhook from running the same review twice while the first is still going. It also means that a stuck pending row blocks the commit entirely until the sweep clears it, which is one more reason the sweep has to exist.&lt;/p&gt;

&lt;p&gt;The second check caps total attempts, because without a cap a close/reopen loop could trigger paid calls forever:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;attemptsSince&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nx"&gt;ATTEMPT_WINDOW_MS&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;attempts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;select&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;count&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;and&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sameCommit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;notQuotaNotice&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="nf"&gt;gte&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;reviews&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;createdAt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;attemptsSince&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]?.&lt;/span&gt;&lt;span class="nx"&gt;count&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nx"&gt;MAX_ATTEMPTS_PER_COMMIT&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;allowed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;plan&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;used&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;record&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;reason&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;attempt-cap&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;gte(reviews.createdAt, attemptsSince)&lt;/code&gt; window is the interesting line, and it exists because of the reaper. A review caught by the sweep is a &lt;code&gt;failed&lt;/code&gt; row, and failed rows count as attempts. My own comment in the code explains the consequence of leaving the window out: the "zero-cost abort" rows the sweeper creates would pile up, and after a few infrastructure accidents of my own making, such as deploys replacing a running instance, that commit could never be reviewed again. The user would have done nothing wrong, and my cleanup logic would have permanently blocked them.&lt;/p&gt;

&lt;p&gt;I only found that by thinking about the reaper as a producer of data, not just a consumer. Every row it writes feeds the other rules in the system. If you add a janitor, trace what its output does to every check that reads the same table.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this still doesn't cover
&lt;/h2&gt;

&lt;p&gt;I'd rather list the gaps than have you find them.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The sweep is lazy.&lt;/strong&gt; It runs when that user's next webhook arrives. If they never send another, their stuck row stays pending. The place that displays the monthly count is read-only, so a stuck row keeps showing as used until the next event cleans it up. I decided that was acceptable for the volumes involved, and I wrote the decision down in a comment, but it is a real gap, and a scheduled sweep would close it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A swept review is simply gone.&lt;/strong&gt; The user's PR doesn't get a review, and nothing retries it. A new push produces a new event with a new commit, which is a fresh attempt. I left it that way because automatic retries on a billable operation are their own can of worms.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The time budget is reasoning, not measurement.&lt;/strong&gt; The 120, the 90 and the 10 come from how I reason about the pieces interacting. I haven't load-tested a run where several of those limits are approached together.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The general lesson
&lt;/h2&gt;

&lt;p&gt;If you only take one thing away: &lt;strong&gt;acknowledging a webhook early doesn't make the work reliable, it makes the work unobserved.&lt;/strong&gt; Before the change, a slow job turned the delivery red, which was annoying but honest. After it, the delivery is always green, and the only truth left is the state you wrote to your own database. So that state has to be designed:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Write a durable record before you acknowledge.&lt;/strong&gt; The row exists before the response goes out, so there is something to inspect if the job vanishes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Give the work a deadline shorter than the platform's, and order the timeouts.&lt;/strong&gt; Make your own timeout fire first, so failure arrives as an exception you can handle rather than a kill you can't.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Put the cleanup outside the process that might die.&lt;/strong&gt; A reaper that runs from somewhere else, later, is the only thing that works when &lt;code&gt;catch&lt;/code&gt; doesn't.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Derive the reaper's threshold from the limit, and decide in advance what a reaped row costs.&lt;/strong&gt; If "failed" means "free," someone will find out how to fail on purpose.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The webhook stayed fast, the delivery log stayed green, and the failure mode that used to be loud became a row with an error message and a timestamp. That is a better place for it to live. But I only got there after a review pass pointed at the one line I thought I'd finished with, which is a good reminder that "the red thing turned green" is not the same as "the problem is solved."&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>nextjs</category>
      <category>debugging</category>
      <category>api</category>
    </item>
    <item>
      <title>My Stripe idempotency key was correct. It just stopped working after 24 hours.</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Wed, 07 Oct 2026 04:24:05 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/my-stripe-idempotency-key-was-correct-it-just-stopped-working-after-24-hours-2h35</link>
      <guid>https://dev.to/phi_blankslate/my-stripe-idempotency-key-was-correct-it-just-stopped-working-after-24-hours-2h35</guid>
      <description>&lt;p&gt;I added a reward feature that pays out through Stripe's customer balance, and I did what every payments guide tells you to do: I passed an idempotency key on the write that moves money. A security review pass then found a double-payment path anyway, and the key was not wrong. The key was correct, unique, and deterministic. It simply had an expiry date that my retry design ignored.&lt;/p&gt;

&lt;p&gt;This post is about the gap between "I used an idempotency key" and "this operation is idempotent." They are not the same claim, and the distance between them is a time window.&lt;/p&gt;

&lt;p&gt;The stack is Next.js, Drizzle, Neon and the Stripe Node SDK, but the lesson has nothing to do with any of those.&lt;/p&gt;

&lt;h2&gt;
  
  
  The feature
&lt;/h2&gt;

&lt;p&gt;When a referred user's paid subscription goes through, the referrer gets a credit on their Stripe customer balance. In Stripe terms that is a negative customer balance transaction, which Stripe automatically applies to the customer's next invoice.&lt;/p&gt;

&lt;p&gt;The reward logic lives in one function, &lt;code&gt;rewardReferralIfEligible&lt;/code&gt;, and it can be called from three places: a webhook for &lt;code&gt;customer.subscription.updated&lt;/code&gt;, a webhook for &lt;code&gt;invoice.paid&lt;/code&gt;, and a manual sweep script I run to catch anything the webhooks missed. Because the same function can be reached from several triggers, it has to be safe to call repeatedly. That requirement is where the story starts.&lt;/p&gt;

&lt;p&gt;There is a &lt;code&gt;referrals&lt;/code&gt; table with a &lt;code&gt;status&lt;/code&gt; column: &lt;code&gt;pending&lt;/code&gt; or &lt;code&gt;rewarded&lt;/code&gt;. The function's job is to move a row from &lt;code&gt;pending&lt;/code&gt; to &lt;code&gt;rewarded&lt;/code&gt; exactly once and, as a side effect of that single transition, create exactly one balance transaction on the referrer's Stripe customer.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer one: make the database the gate
&lt;/h2&gt;

&lt;p&gt;The first thing I wrote was a compare-and-swap. Before touching Stripe, claim the row:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;updated&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;rewarded&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;rewardedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;and&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;returning&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;updated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;WHERE status = 'pending'&lt;/code&gt; clause is the important part. If two webhook deliveries race, both run this UPDATE, but only one of them gets a row back from &lt;code&gt;.returning()&lt;/code&gt;. The loser sees an empty array and exits. Postgres serializes the two updates on the same row, so exactly one caller proceeds. This also means the "already processed" check and the "claim" are one atomic statement instead of a read followed by a write, which is the classic time-of-check-to-time-of-use hole.&lt;/p&gt;

&lt;p&gt;So far so good. The winner now owns the row and goes on to talk to Stripe.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer two: the idempotency key
&lt;/h2&gt;

&lt;p&gt;The winner then creates the balance transaction. This is the call that actually moves money, so this is where I put the key:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;customers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createBalanceTransaction&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nx"&gt;referrerCustomerId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;REFERRAL_REWARD_CENTS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;currency&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;usd&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`Referral reward — referred user converted to Pro Annual &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;marker&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;idempotencyKey&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;`referral-reward-&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The key is derived from &lt;code&gt;referral.id&lt;/code&gt;, which is a UUID assigned once when the referral row is created and never changes. That is the right shape for a key: the same logical operation always produces the same key, and different operations never collide. If the HTTP response is lost and the call is retried, Stripe recognizes the key and returns the original result instead of creating a second transaction.&lt;/p&gt;

&lt;p&gt;I stopped here at first and considered the problem solved. Database CAS stops concurrent callers. The idempotency key stops duplicate writes to Stripe. Two layers.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rollback that made it dangerous
&lt;/h2&gt;

&lt;p&gt;There is a third piece of code, and it is the one I wrote for a good reason. The CAS flips the row to &lt;code&gt;rewarded&lt;/code&gt; &lt;em&gt;before&lt;/em&gt; the Stripe call. That ordering is deliberate: it is what makes concurrent callers safe. But it means that if the Stripe call then fails, the database says "rewarded" while no credit exists. That is a silent loss for the referrer, and the row is now permanently skipped because it is no longer &lt;code&gt;pending&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;So the error path reverts the row (trimmed to the relevant lines):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;rewardedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;rollbackErr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="s2"&gt;`[referral] CRITICAL: rollback to pending also failed for referral &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;. `&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
        &lt;span class="s2"&gt;`Row is stuck as "rewarded" without an actual credit. Manual reconciliation required.`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;originalError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;rollbackError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;rollbackErr&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Rolling back to &lt;code&gt;pending&lt;/code&gt; means the next trigger, whether it is a later webhook or my sweep script, will pick the row up again and retry. That is what I want for a clean failure like a network error before Stripe received anything.&lt;/p&gt;

&lt;p&gt;Now put the two mechanisms next to each other and ask what happens in the ambiguous case.&lt;/p&gt;

&lt;p&gt;An ambiguous failure is when the request reaches Stripe, Stripe creates the balance transaction, and the response never makes it back to me. A timeout, a dropped connection, a function that gets killed mid-flight. From my code's point of view the call threw. From Stripe's point of view the credit exists.&lt;/p&gt;

&lt;p&gt;My catch block cannot tell these two cases apart. It sees an exception and rolls the row back to &lt;code&gt;pending&lt;/code&gt;. Now the database says "not rewarded" and Stripe says "rewarded", and the row is waiting for a retry.&lt;/p&gt;

&lt;p&gt;If the retry happens within 24 hours, the idempotency key saves me: Stripe sees the same key, returns the original transaction, and nothing is duplicated.&lt;/p&gt;

&lt;p&gt;If the retry happens later, it does not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 24-hour window
&lt;/h2&gt;

&lt;p&gt;Stripe's documentation says that keys can be removed from the system automatically once they are at least 24 hours old, and that Stripe generates a new request if a key is reused after the original was pruned. That is the whole mechanism. After the window, the same key means "new request," and a new request creates a new balance transaction.&lt;/p&gt;

&lt;p&gt;I had read that sentence before. I had even written a comment in the code saying the key protects retries "within 24h." What I had not done is ask who controls how long a row can stay in &lt;code&gt;pending&lt;/code&gt; after a rollback.&lt;/p&gt;

&lt;p&gt;The answer was: not me, at least not tightly. A row reverted to &lt;code&gt;pending&lt;/code&gt; waits for the next trigger. The webhook triggers are driven by Stripe's event delivery, which I do not control. The sweep script runs when I run it. If nothing touches that row for a day and a bit, and then the next trigger arrives, the function runs the whole sequence again with an expired key. The sequence passes every check I had written: the row is &lt;code&gt;pending&lt;/code&gt;, the subscription is eligible, the CAS succeeds, and Stripe happily creates a second credit.&lt;/p&gt;

&lt;p&gt;The reviewer's finding, stated plainly: a rollback-and-retry design plus a 24-hour key means the retry safety net has a hole that opens exactly when the system has been quiet the longest. And "quiet for a day" is not an exotic condition. It is what a weekend looks like.&lt;/p&gt;

&lt;p&gt;This is the part worth sitting with. Every individual piece of the code was defensible. The CAS was correct. The key was correct. The rollback was correct. The bug lived in the interaction between the rollback's retry delay, which is unbounded, and the key's lifetime, which is bounded.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: check Stripe's state, not your own memory
&lt;/h2&gt;

&lt;p&gt;An idempotency key is a cache of "I already did this" held by someone else, with a TTL. When the question is "did this money already move," the authoritative answer is the ledger, not the cache. So before creating the balance transaction, I now look at what is actually on the referrer's customer balance:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;marker&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;`(referral &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;)`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;allTxs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;customers&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;listBalanceTransactions&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrerCustomerId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;autoPagingToArray&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;limit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;allTxs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;`referrer &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;referrerCustomerId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; has 1000+ balance transactions; manual reconciliation required`&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;existing&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;allTxs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;description&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="dl"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;marker&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;existing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;warn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;`[referral] credit already present on Stripe for referral &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;; marked rewarded without a new credit`&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;recordBalanceTransactionId&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;existing&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The marker is a string embedded in the &lt;code&gt;description&lt;/code&gt; of every credit I create, containing the referral's own ID. If a credit with that marker already exists on the customer, the function does not create another one. It records the existing transaction's ID against the row and finishes. The state of the world is read from Stripe at the moment of the decision, so it does not matter how long the row sat in &lt;code&gt;pending&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;I kept the idempotency key. It is still the cheapest protection against the common case, a retry seconds after a flaky response, and it costs nothing. But it is now the first line of defense, not the only one. The ledger lookup is what is left when the key has expired.&lt;/p&gt;

&lt;p&gt;Two things I want to be honest about, because this fix is not free of tradeoffs:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Matching on a description string is a convention, not a constraint.&lt;/strong&gt; It works because I control every writer of these credits. If someone adds a credit by hand in the dashboard with a different description, it is invisible to the check, and if someone edits a description, the check can miss it. I treat it as a second line of defense, not a proof.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The scan is bounded.&lt;/strong&gt; Listing every balance transaction for a customer is fine for a customer with a handful of them and not fine for one with thousands. I cap the scan at 1,000 entries, and if the cap is hit I throw instead of guessing, which sends the row to manual reconciliation. Failing closed seemed better than silently assuming "not found" on a list I did not finish reading.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  A stronger anchor: record the transaction ID
&lt;/h2&gt;

&lt;p&gt;The description marker is a search. A search can be wrong in ways a direct lookup cannot, so I added a column to the &lt;code&gt;referrals&lt;/code&gt; table that stores the balance transaction ID returned by Stripe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;recordBalanceTransactionId&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referralId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;txId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="k"&gt;void&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;stripeBalanceTransactionId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;txId&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;referralId&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
      &lt;span class="s2"&gt;`[referral] CRITICAL: credit &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;txId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; granted but failed to record on referral &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;referralId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;. `&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt;
        &lt;span class="s2"&gt;`Row will show as rewardedWithoutCredit in KPI; reconcile manually.`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nx"&gt;err&lt;/span&gt;
    &lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and the top of the function checks it before doing anything else:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;stripeBalanceTransactionId&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;rewarded&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;rewardedAt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;rewardedAt&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Date&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;and&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;referral&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;referrals&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;pending&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the transaction ID is already recorded, the credit exists by definition, and the function just repairs the status without calling Stripe at all.&lt;/p&gt;

&lt;p&gt;Notice the comment I left in &lt;code&gt;recordBalanceTransactionId&lt;/code&gt;: it deliberately does not rethrow. The credit has already been granted at that point. If recording the ID failed and I threw, the outer catch would roll the row back to &lt;code&gt;pending&lt;/code&gt;, and I would have built a brand-new route to the exact double-credit I had just closed. The error handling for "the money moved but my bookkeeping failed" has to be different from the error handling for "the money did not move." Mixing them is how a safety mechanism turns into a hazard.&lt;/p&gt;

&lt;h2&gt;
  
  
  The related trap I found one pass later
&lt;/h2&gt;

&lt;p&gt;While the same function was under review, a second finding turned up that has the same flavor: an assumption about what a Stripe field means.&lt;/p&gt;

&lt;p&gt;The reward was originally gated on the subscription's &lt;code&gt;status&lt;/code&gt; being &lt;code&gt;active&lt;/code&gt;. That sounds like "the customer paid." It is not. When a trial ends, Stripe moves the subscription to &lt;code&gt;active&lt;/code&gt; before the payment for the first invoice has been collected. For a short window the subscription is active and nothing has been paid, which means a status check alone can hand out a reward for money that never arrived.&lt;/p&gt;

&lt;p&gt;The gate now looks at the subscription's latest invoice:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;sub&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;stripe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;subscriptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;retrieve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;stripeSubscriptionId&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="na"&gt;expand&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;latest_invoice&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;active&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;// ...&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;latestInvoice&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;latestInvoice&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;paid&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;latestInvoice&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;amount_paid&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An invoice that is &lt;code&gt;paid&lt;/code&gt; with &lt;code&gt;amount_paid &amp;gt; 0&lt;/code&gt; is evidence that money moved. A subscription status is a state machine label. I also added &lt;code&gt;invoice.paid&lt;/code&gt; as a trigger for the function so that the payment itself, not a status change, wakes it up. Because the function re-derives eligibility from Stripe on every call and returns quietly when the answer is "not yet," it is safe to trigger from any event, and a trigger that arrives too early costs nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would tell myself before writing this
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;An idempotency key is a time-limited promise from someone else.&lt;/strong&gt; Before relying on one, write down its TTL next to the longest delay your own system can introduce between an attempt and its retry. If the second number can exceed the first, the key is not your idempotency story. Mine could, because a rolled-back row waits for an external trigger I do not control.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rollback plus retry is where idempotency actually gets tested.&lt;/strong&gt; Straight-line code with a key looks airtight. The risk concentrates wherever you convert an &lt;em&gt;ambiguous&lt;/em&gt; failure into a "try again later." An ambiguous failure means you do not know whether the side effect happened, and "later" is unbounded unless you bound it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ask the ledger, not the cache.&lt;/strong&gt; When the side effect lives in someone else's system, the safest idempotency check reads that system's current state at the moment of the decision. It costs an extra API call. It also stays correct no matter how long the row sat, how many times it was retried, or whether a key was pruned in between.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Order your layers by what they protect against.&lt;/strong&gt; In this function the database CAS stops concurrent callers, the idempotency key stops fast duplicate retries, the ledger lookup stops slow ones, and the stored transaction ID makes the repair path cheap. Each one covers a failure the others do not. Removing any single layer reintroduces a specific bug, so it helps to be able to say which one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Treat "the money moved but my bookkeeping failed" as its own case.&lt;/strong&gt; It must not share an error path with "the money did not move." Rolling back after a successful side effect is how you manufacture the duplicate you were trying to prevent.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Status fields are not receipts.&lt;/strong&gt; If a decision depends on "was it paid," find the object that records payment, an invoice or a charge, and check that. A status can be reached by paths you did not anticipate.&lt;/p&gt;

&lt;h2&gt;
  
  
  The uncomfortable part
&lt;/h2&gt;

&lt;p&gt;I want to be clear about how this was found: not by a test and not by production. A review pass that was specifically told to attack the money path read the rollback code and the key's lifetime side by side and asked how long a row could stay pending. I had the 24-hour fact in a code comment, one screen away from the bug, and still did not connect the two.&lt;/p&gt;

&lt;p&gt;My tests would not have caught it either, because reproducing it needs an ambiguous failure followed by a gap of more than a day, and nobody writes a test with a 25-hour sleep. If a bug only exists across a time window longer than your test suite's patience, the fix has to come from reasoning about the window, not from waiting for a failure to show it to you.&lt;/p&gt;

&lt;p&gt;The fix is a handful of extra lines. The expensive part was learning to read &lt;code&gt;idempotencyKey&lt;/code&gt; as a claim with an expiry date instead of as a property of the operation.&lt;/p&gt;

</description>
      <category>stripe</category>
      <category>typescript</category>
      <category>webdev</category>
      <category>api</category>
    </item>
    <item>
      <title>drizzle-kit generated a DROP TABLE for a table that was still in production. drizzle-kit wasn't the problem.</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Fri, 02 Oct 2026 12:27:09 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/drizzle-kit-generated-a-drop-table-for-a-table-that-was-still-in-production-drizzle-kit-wasnt-the-4o</link>
      <guid>https://dev.to/phi_blankslate/drizzle-kit-generated-a-drop-table-for-a-table-that-was-still-in-production-drizzle-kit-wasnt-the-4o</guid>
      <description>&lt;p&gt;I asked drizzle-kit for a migration that added two tables. The SQL it generated also contained a &lt;code&gt;DROP TABLE ... CASCADE&lt;/code&gt; for a table that was still sitting in my production database, and an &lt;code&gt;ADD COLUMN&lt;/code&gt; for a column that production already had.&lt;/p&gt;

&lt;p&gt;Neither change had anything to do with the feature I was building. If I had applied the file as it came out, the first statement would have deleted a live table along with anything that depended on it. The second would have failed with "column already exists" and taken the rest of the migration with it, or left it half-applied, depending on how I ran it.&lt;/p&gt;

&lt;p&gt;drizzle-kit did nothing wrong. It did exactly what it is designed to do. The problem was that I had three different ideas of "the current schema" in my project and assumed they were the same thing. This post explains how they drifted apart, why the diff came out the way it did, and the checks I now run before any generated migration goes near production.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The app is Next.js with Drizzle ORM on Postgres (Neon). The schema lives in one TypeScript file, &lt;code&gt;lib/db/schema.ts&lt;/code&gt;. Migrations are generated with &lt;code&gt;drizzle-kit generate&lt;/code&gt; into &lt;code&gt;drizzle/migrations/&lt;/code&gt;, and drizzle-kit keeps a JSON snapshot of the schema next to each migration in &lt;code&gt;drizzle/migrations/meta/&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;I don't let tooling apply migrations to production. I don't use &lt;code&gt;drizzle-kit push&lt;/code&gt;, and the header of my migration files literally says not to use &lt;code&gt;drizzle-kit migrate&lt;/code&gt; either. Production changes are applied by hand, inside a transaction, after I've read the SQL. That rule is the only reason this story ends with a cleanup instead of a restore.&lt;/p&gt;

&lt;p&gt;There is one more habit in the background that matters. For small additive changes I sometimes skipped the generator and wrote a one-off script that runs a single idempotent DDL statement. That habit is half of the bug.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the drift happened
&lt;/h2&gt;

&lt;p&gt;The baseline migration (&lt;code&gt;0000&lt;/code&gt;) was generated at the end of July. At that point the schema had a &lt;code&gt;waitlist_entries&lt;/code&gt; table for a pre-launch email list, and the &lt;code&gt;reviews&lt;/code&gt; table did not have a &lt;code&gt;suppressed_count&lt;/code&gt; column yet.&lt;/p&gt;

&lt;p&gt;The next day I made two schema changes in one commit, and neither went through &lt;code&gt;drizzle-kit generate&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Change 1: I deleted the waitlist feature.&lt;/strong&gt; We had decided to launch directly instead of collecting a waitlist, so I removed the route, the form, the email-sending module, and the table definition from &lt;code&gt;schema.ts&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight diff"&gt;&lt;code&gt; // ウェイトリスト登録（プレローンチ用リード収集）
&lt;span class="gd"&gt;-export const waitlistEntries = pgTable(
-  "waitlist_entries",
-  {
-    id: uuid("id").defaultRandom().primaryKey(),
-    email: text("email").notNull().unique(),
-    source: text("source"), // UTM等の登録元。任意
-    createdAt: timestamp("created_at").defaultNow().notNull(),
-  },
-  (table) =&amp;gt; ({
-    emailIdx: uniqueIndex("waitlist_entries_email_idx").on(table.email),
-  })
-);
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(The leftover Japanese comment above it says "waitlist signups (pre-launch lead collection)". It outlived the code it described, which is a small preview of the whole problem.)&lt;/p&gt;

&lt;p&gt;I deleted the &lt;em&gt;definition&lt;/em&gt;. I did not drop the &lt;em&gt;table&lt;/em&gt;. In my head the feature was gone. In production the table was still there, empty and unreferenced.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Change 2: I added a column with a one-off script.&lt;/strong&gt; In the same commit, &lt;code&gt;reviews&lt;/code&gt; gained a column:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight diff"&gt;&lt;code&gt;     commentsCount: integer("comments_count").default(0),
&lt;span class="gi"&gt;+    suppressedCount: integer("suppressed_count").default(0),
&lt;/span&gt;     tokensUsed: integer("tokens_used").default(0),
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Instead of generating a migration, I added it to production with a small script that runs one statement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;sql&lt;/span&gt;&lt;span class="s2"&gt;`
  ALTER TABLE reviews
    ADD COLUMN IF NOT EXISTS suppressed_count integer DEFAULT 0
`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's idempotent, it checks &lt;code&gt;information_schema&lt;/code&gt; afterwards, and it worked. From production's point of view, everything was correct.&lt;/p&gt;

&lt;p&gt;So now:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;schema.ts&lt;/code&gt; had no waitlist table and did have &lt;code&gt;suppressed_count&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Production still had the waitlist table and also had &lt;code&gt;suppressed_count&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;The latest drizzle snapshot (&lt;code&gt;0000_snapshot.json&lt;/code&gt;) still had the waitlist table and did &lt;strong&gt;not&lt;/strong&gt; have &lt;code&gt;suppressed_count&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Nothing complained, because nothing compares those three. The app only reads &lt;code&gt;schema.ts&lt;/code&gt;. Production only knows what has been executed against it. The snapshot only changes when you run &lt;code&gt;generate&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  More than a month later: &lt;code&gt;drizzle-kit generate&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;In early September I built a referral feature that needed two new tables, &lt;code&gt;referral_codes&lt;/code&gt; and &lt;code&gt;referrals&lt;/code&gt;. I added them to &lt;code&gt;schema.ts&lt;/code&gt; and ran &lt;code&gt;npm run db:generate&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The generated file contained the two &lt;code&gt;CREATE TABLE&lt;/code&gt; statements I expected, and two statements I didn't:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;DROP&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="nv"&gt;"waitlist_entries"&lt;/span&gt; &lt;span class="k"&gt;CASCADE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="nv"&gt;"reviews"&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="nv"&gt;"suppressed_count"&lt;/span&gt; &lt;span class="p"&gt;...;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(I edited the generated file before committing it, so these are the two lines as I recorded them in my notes at the time, not a byte-for-byte copy of the original output.)&lt;/p&gt;

&lt;p&gt;My first reaction was that drizzle-kit had somehow connected to the wrong database. It hadn't connected to any database.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the diff looks like that
&lt;/h2&gt;

&lt;p&gt;This was the part I had wrong in my head for months: &lt;strong&gt;&lt;code&gt;drizzle-kit generate&lt;/code&gt; does not diff your schema against your database. It diffs your schema against its own last snapshot.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The flow is:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Read &lt;code&gt;schema.ts&lt;/code&gt; and build a schema object.&lt;/li&gt;
&lt;li&gt;Read the most recent snapshot JSON in &lt;code&gt;meta/&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Emit SQL for every difference between the two.&lt;/li&gt;
&lt;li&gt;Write a new snapshot equal to the current &lt;code&gt;schema.ts&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;My &lt;code&gt;drizzle.config.ts&lt;/code&gt; has a &lt;code&gt;dbCredentials&lt;/code&gt; block with the production-shaped URL, which made it easy to believe that &lt;code&gt;generate&lt;/code&gt; looks at the database. It doesn't. Those credentials are for commands that actually talk to a database (&lt;code&gt;push&lt;/code&gt;, &lt;code&gt;migrate&lt;/code&gt;, &lt;code&gt;pull&lt;/code&gt;, &lt;code&gt;studio&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;Run the drift through that flow:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Last snapshot (&lt;code&gt;0000&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;
&lt;code&gt;schema.ts&lt;/code&gt; now&lt;/th&gt;
&lt;th&gt;Generated SQL&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;waitlist_entries&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;exists&lt;/td&gt;
&lt;td&gt;gone&lt;/td&gt;
&lt;td&gt;&lt;code&gt;DROP TABLE ... CASCADE&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;reviews.suppressed_count&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;missing&lt;/td&gt;
&lt;td&gt;exists&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ADD COLUMN&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;referral_codes&lt;/code&gt;, &lt;code&gt;referrals&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;missing&lt;/td&gt;
&lt;td&gt;exist&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;CREATE TABLE&lt;/code&gt; x2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every line is correct relative to the snapshot. The snapshot was the stale party, and it was stale because I had changed the schema twice without telling the generator.&lt;/p&gt;

&lt;p&gt;I can confirm this from the repository now. &lt;code&gt;grep&lt;/code&gt; over the snapshot files shows &lt;code&gt;waitlist_entries&lt;/code&gt; present in &lt;code&gt;0000_snapshot.json&lt;/code&gt; and absent from &lt;code&gt;0001&lt;/code&gt; onward, and &lt;code&gt;suppressed_count&lt;/code&gt; absent from &lt;code&gt;0000&lt;/code&gt; and present from &lt;code&gt;0001&lt;/code&gt; onward. The snapshot quietly absorbed both changes at the moment I ran &lt;code&gt;generate&lt;/code&gt; for an unrelated feature.&lt;/p&gt;

&lt;h2&gt;
  
  
  What each statement would have done
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The DROP.&lt;/strong&gt; In my case the damage would have been close to zero. Before deciding anything, I checked production: the table existed, and &lt;code&gt;SELECT COUNT(*)&lt;/code&gt; on it returned an empty result set. But "it happened to be empty" is luck, not a safety property. Put any other "dead" table in that position (an audit log you stopped writing to, a table a cron job still reads, a table a teammate is about to wire up) and the same &lt;code&gt;generate&lt;/code&gt; run produces the same line. &lt;code&gt;CASCADE&lt;/code&gt; widens it further: it also drops whatever depends on the table, such as foreign keys from other tables and views.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The ADD COLUMN.&lt;/strong&gt; This one would have failed loudly, since production already had the column, and Postgres rejects a plain &lt;code&gt;ADD COLUMN&lt;/code&gt; for an existing column. Loud is better than silent, but it still matters where it fails. If the file is applied as a single transaction, the whole migration rolls back and you retry. If it's applied statement by statement (some SQL consoles autocommit each statement), you end up with whatever ran before the failure applied and whatever came after it not. That's a schema nobody designed.&lt;/p&gt;

&lt;p&gt;My one-off script had used &lt;code&gt;ADD COLUMN IF NOT EXISTS&lt;/code&gt;, which is idempotent. The generator emits a plain &lt;code&gt;ADD COLUMN&lt;/code&gt;, because as far as the snapshot knows the column has never existed. The idempotency lived in the script, not in the migration history.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I actually did
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Checked production before touching anything.&lt;/strong&gt; I queried &lt;code&gt;information_schema.tables&lt;/code&gt; and &lt;code&gt;information_schema.columns&lt;/code&gt; to confirm what really existed. The table was there, and the column was there.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Removed both unrelated statements from the generated file.&lt;/strong&gt; The migration was narrowed to the referral tables only. A migration should do what its name says and nothing else.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cleaned up the dead table as a separate, deliberate step.&lt;/strong&gt; After confirming it was empty, and with explicit sign-off, I dropped &lt;code&gt;waitlist_entries&lt;/code&gt; by hand. It was its own decision instead of a side effect of a referral feature.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Applied the migration by hand, in a transaction.&lt;/strong&gt; The committed file is now wrapped in &lt;code&gt;BEGIN; ... COMMIT;&lt;/code&gt; with a header comment explaining why. A later edit to the same file made the transaction even more important: it replaces an index with a unique version (&lt;code&gt;DROP INDEX IF EXISTS&lt;/code&gt; followed by &lt;code&gt;CREATE UNIQUE INDEX&lt;/code&gt;). In an autocommit console, if the create fails after the drop succeeds, you're left with no index at all, which breaks every &lt;code&gt;ON CONFLICT&lt;/code&gt; that relied on it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Because the snapshot had already been rewritten to match &lt;code&gt;schema.ts&lt;/code&gt;, the next &lt;code&gt;generate&lt;/code&gt; (for later changes) produced clean diffs. The drift was absorbed into history. Repository and production agreed again, but only because a human reconciled them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the side door existed in the first place
&lt;/h2&gt;

&lt;p&gt;It's worth being honest about why I was making schema changes outside the generator at all, because the reason wasn't laziness. It was friction, and friction is what pushes everyone toward side doors.&lt;/p&gt;

&lt;p&gt;The header comment of my one-off column script explains it: I didn't use &lt;code&gt;drizzle-kit push&lt;/code&gt; because push can stop and ask interactive questions (for example, whether it's allowed to truncate a table), and a command that waits for a keypress in a non-interactive context just hangs. So for a single additive column against production, a tiny script with &lt;code&gt;ADD COLUMN IF NOT EXISTS&lt;/code&gt;, a dry-run mode by default, an explicit &lt;code&gt;--apply&lt;/code&gt; flag, and an &lt;code&gt;information_schema&lt;/code&gt; check afterwards felt &lt;em&gt;safer&lt;/em&gt; than the official tool. For that one change, it was.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;generate&lt;/code&gt; has the same trait. When I ran the referral migration, I first tried it from an automated, non-TTY session, and it stopped at an interactive prompt it couldn't get an answer to. It had to be run from a real terminal. That's a reasonable design, since some schema changes are genuinely ambiguous (is this a rename or a drop-and-create?) and a tool shouldn't guess. But it means the "proper" path costs more than the side door, and every time the proper path costs more, some changes will skip it.&lt;/p&gt;

&lt;p&gt;The fix isn't to ban side doors. Sometimes a hand-written, idempotent statement really is the right way to change production. The fix is to make sure every side door ends with a step that updates the front door's records: after the script runs, run &lt;code&gt;generate&lt;/code&gt; so the snapshot catches up, and commit the result as "already applied". The script changes the database. The generate run changes the history. You need both, or the history starts lying.&lt;/p&gt;

&lt;h2&gt;
  
  
  The check I already had, and why it wasn't enough
&lt;/h2&gt;

&lt;p&gt;I already had a read-only preflight script that runs before any DDL. It opens a &lt;code&gt;READ ONLY&lt;/code&gt; transaction, lists the tables in &lt;code&gt;public&lt;/code&gt;, and compares them with what &lt;code&gt;schema.ts&lt;/code&gt; defines. Here's the core of it (comments translated):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Query inside a READ ONLY transaction.&lt;/span&gt;
&lt;span class="c1"&gt;// Even if code below ever tries to write, the database refuses it.&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;currentDatabase&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;tables&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;begin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="s2"&gt;`SET TRANSACTION READ ONLY`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[{&lt;/span&gt; &lt;span class="nx"&gt;current_database&lt;/span&gt; &lt;span class="p"&gt;}]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;current_database&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}[]&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="s2"&gt;`
    SELECT current_database()
  `&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;tx&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="p"&gt;}[]&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="s2"&gt;`
    SELECT table_name
    FROM information_schema.tables
    WHERE table_schema = 'public' AND table_type = 'BASE TABLE'
    ORDER BY table_name
  `&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;currentDatabase&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;current_database&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;tables&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and later:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;warnings&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;unexpected&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;tables&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;ALL_EXPECTED&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;unexpected&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;warnings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`table exists that schema.ts does not define: &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;unexpected&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;, &lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That warning is exactly the signal for the waitlist half of this bug: a table in production that the code doesn't define. It would have fired. But a warning in a JSON report is something you have to go and read, and the preflight answers a different question ("am I connected to the right database?"). It checks tables, not columns, so it would never have caught the &lt;code&gt;suppressed_count&lt;/code&gt; half. And it compares production with &lt;code&gt;schema.ts&lt;/code&gt;, not with the snapshot. The snapshot is the third party that actually drives &lt;code&gt;generate&lt;/code&gt;, and nothing was looking at it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I do now
&lt;/h2&gt;

&lt;p&gt;None of this needs special tooling. It's mostly about treating a generated migration as a &lt;em&gt;proposal&lt;/em&gt; and treating the snapshot as state that can go stale.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Read every generated statement and trace it back to your change.&lt;/strong&gt; For each line, ask "which edit in this branch caused this?" If the answer is "none", stop. An unexplained &lt;code&gt;DROP&lt;/code&gt; or &lt;code&gt;ALTER&lt;/code&gt; means the snapshot and reality have diverged somewhere, and the fix is to find out where, not to delete the line and move on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Treat &lt;code&gt;DROP&lt;/code&gt; and &lt;code&gt;CASCADE&lt;/code&gt; in a generated file as a hard stop.&lt;/strong&gt; Any destructive statement gets checked against the live database (does it exist, does it have rows, what depends on it) before it's allowed to stay. If it stays, it should be its own reviewed change, not a side effect of something else.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Don't change the schema outside the generator, or if you do, close the loop.&lt;/strong&gt; One-off scripts are fine and sometimes the right call for production. But if a column goes in by script, run &lt;code&gt;generate&lt;/code&gt; immediately afterwards so the snapshot learns about it, and commit that migration marked as already applied. Otherwise the generator will "discover" the change later, inside somebody else's feature branch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Removing a table from &lt;code&gt;schema.ts&lt;/code&gt; is a migration, not a cleanup.&lt;/strong&gt; Deleting the definition is the start of removing a table, not the end. Either drop it in the same change, through a reviewed migration, or leave the definition in place with a comment until you're ready to drop it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Wrap hand-applied migrations in a transaction.&lt;/strong&gt; Partial application is the failure mode that turns an ordinary error into an unplanned schema.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Check production against &lt;code&gt;schema.ts&lt;/code&gt; periodically, not just before migrations.&lt;/strong&gt; Tables &lt;em&gt;and&lt;/em&gt; columns. An &lt;code&gt;information_schema&lt;/code&gt; query is enough:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;column_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;data_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;column_default&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;information_schema&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;columns&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;table_schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'public'&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ordinal_position&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Compare that list with your schema file. Anything that exists on only one side is drift that your next &lt;code&gt;generate&lt;/code&gt; will try to "fix" for you.&lt;/p&gt;

&lt;h2&gt;
  
  
  The general lesson
&lt;/h2&gt;

&lt;p&gt;Most migration tools that generate SQL from a declarative schema work this way. They have to compare against &lt;em&gt;something&lt;/em&gt;, and if that something is a file in your repo rather than the live database, it can only be as accurate as the history you fed it. Every schema change made by hand, by script, from a SQL console, or by deleting code without dropping the table it described creates a gap between the snapshot and reality. The generator closes that gap the only way it can: by writing SQL that makes the database match the file, whether you meant to ask for that or not.&lt;/p&gt;

&lt;p&gt;The dangerous part isn't the generator. It's the assumption that "the schema" is one thing. In my project it was three: what the code declares, what the tool's snapshot records, and what the database actually contains. They agree only as long as every change goes through the same door. The day one change uses a side door, the next person to use the front door gets a migration that "fixes" it.&lt;/p&gt;

&lt;p&gt;So the rule I follow now is short: a generated migration is a diff against history, not against production. Read it like a diff from a teammate who has been away for a month, because in effect that's what it is.&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>database</category>
      <category>typescript</category>
      <category>webdev</category>
    </item>
    <item>
      <title>A stray .env file broke exactly 13 pages of my Next.js build — and dev mode never saw it</title>
      <dc:creator>phi_blankslate</dc:creator>
      <pubDate>Mon, 28 Sep 2026 08:37:19 +0000</pubDate>
      <link>https://dev.to/phi_blankslate/a-stray-env-file-broke-exactly-13-pages-of-my-nextjs-build-and-dev-mode-never-saw-it-i44</link>
      <guid>https://dev.to/phi_blankslate/a-stray-env-file-broke-exactly-13-pages-of-my-nextjs-build-and-dev-mode-never-saw-it-i44</guid>
      <description>&lt;p&gt;&lt;code&gt;npx next build&lt;/code&gt; was failing on precisely 13 pages, every time, with the same cryptic error:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;TypeError: Invalid URL, input: ''
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;npm run dev&lt;/code&gt; worked fine. &lt;code&gt;git stash&lt;/code&gt; back to a commit from weeks earlier — same 13 pages, same error. Whatever this was, it wasn't something I'd just written.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;git stash&lt;/code&gt; step mattered more than it might look. My first assumption, like most people's, was "I broke something recently." Stashing back past several days of commits and reproducing the exact same failure on the exact same 13 pages ruled that out immediately — whatever this was, it wasn't in the diff. It was in the environment, which meant it was invisible to &lt;code&gt;git log&lt;/code&gt;, invisible to code review, and invisible to anyone just reading the codebase. That's the property that made this bug worth writing up: it wasn't a logic error anyone could have caught by reading code, because the code was never wrong.&lt;/p&gt;

&lt;p&gt;This is the story of chasing that error down to a single leftover file from a debugging session a month prior, and why the fix required understanding something about how Next.js loads &lt;code&gt;.env*&lt;/code&gt; files that I'd never had a reason to learn before.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The app is a Next.js 14 (App Router) SaaS with NextAuth for GitHub OAuth login. &lt;code&gt;app/layout.tsx&lt;/code&gt; is the root layout every page goes through, and it calls &lt;code&gt;getServerSession()&lt;/code&gt; so the UI can tell if you're logged in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;RootLayout&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;children&lt;/span&gt; &lt;span class="p"&gt;}:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;children&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;React&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ReactNode&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;session&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;getServerSession&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;authOptions&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;error&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;digest&lt;/span&gt;&lt;span class="p"&gt;?:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt; &lt;span class="p"&gt;})?.&lt;/span&gt;&lt;span class="nx"&gt;digest&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;DYNAMIC_SERVER_USAGE&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;[layout] getServerSession failed (likely missing env vars):&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;error&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;html&lt;/span&gt; &lt;span class="na"&gt;lang&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"en"&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nt"&gt;body&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
        &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Providers&lt;/span&gt; &lt;span class="na"&gt;session&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;session&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;children&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nc"&gt;Providers&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;body&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nt"&gt;html&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note the &lt;code&gt;try/catch&lt;/code&gt; there. I'd added it specifically so that missing env vars during a static build wouldn't crash the whole page — log it, fall back to &lt;code&gt;session = null&lt;/code&gt;, move on. It felt like it should be bulletproof. It was not the code that was crashing.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Providers&lt;/code&gt; is a small client component:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;use client&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;SessionProvider&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;next-auth/react&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;Providers&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="nx"&gt;children&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;session&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;SessionProvider&lt;/span&gt; &lt;span class="na"&gt;session&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;session&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;children&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="nc"&gt;SessionProvider&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Thirteen lines, no logic. And yet this is where the trail eventually led.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two wrong guesses, then I stopped guessing
&lt;/h2&gt;

&lt;p&gt;My first instinct was: something's missing from &lt;code&gt;.env.local&lt;/code&gt;. I checked &lt;code&gt;NEXTAUTH_URL&lt;/code&gt; — it was there, with a value. Not that.&lt;/p&gt;

&lt;p&gt;Second guess: maybe &lt;code&gt;NEXTAUTH_URL_INTERNAL&lt;/code&gt; (a lesser-known NextAuth variable used behind proxies) was set to something malformed. I grepped &lt;code&gt;.env.local&lt;/code&gt; for it. The line didn't exist at all. Not that either.&lt;/p&gt;

&lt;p&gt;Two guesses, two misses. At that point I stopped inventing theories about &lt;code&gt;.env.local&lt;/code&gt; and went looking for hard evidence instead — specifically, whatever code was actually constructing the URL that was failing. The build output is right there in &lt;code&gt;.next/server/&lt;/code&gt;, so I grepped the compiled chunk for the literal pattern the stack trace implied:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="s1"&gt;'new URL([^)]*'&lt;/span&gt; .next/server/chunks/662.js
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nineteen matches. All nineteen referenced exactly three identifiers: &lt;code&gt;NEXTAUTH_URL&lt;/code&gt;, &lt;code&gt;NEXTAUTH_URL_INTERNAL&lt;/code&gt;, and &lt;code&gt;VERCEL_URL&lt;/code&gt;. That's not application code — that's &lt;code&gt;next-auth/react&lt;/code&gt; itself. My own code doesn't construct a &lt;code&gt;URL&lt;/code&gt; object anywhere near auth.&lt;/p&gt;

&lt;p&gt;That grep was the actual turning point, and it's worth naming why it worked when guessing didn't. A &lt;code&gt;.env&lt;/code&gt; file is a hypothesis space — you can stare at it and imagine a dozen ways a value could be wrong. A compiled bundle is not a hypothesis space; it's a record of what the code actually does. Every &lt;code&gt;new URL(...)&lt;/code&gt; call in that chunk is a call some real module makes, using real variable names, and there's no interpretation involved in reading it. I didn't need to know anything about NextAuth internals going in — the grep told me exactly which three environment variables mattered and, implicitly, which package to go read next. Compiled output is one of the few artifacts in a JS project that can't be wrong about what it contains.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a client component can crash a build with an uncatchable error
&lt;/h2&gt;

&lt;p&gt;Here's the part that made the &lt;code&gt;try/catch&lt;/code&gt; in &lt;code&gt;layout.tsx&lt;/code&gt; irrelevant: &lt;code&gt;next-auth/react&lt;/code&gt;'s &lt;code&gt;SessionProvider&lt;/code&gt; module runs &lt;code&gt;parseUrl(process.env.NEXTAUTH_URL ?? process.env.VERCEL_URL)&lt;/code&gt; at &lt;strong&gt;import time&lt;/strong&gt; — at the top level of the module, not inside a function that gets called (and could be wrapped) later. By the time &lt;code&gt;Providers&lt;/code&gt; even renders, the module has already executed and either succeeded or thrown.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;parseUrl&lt;/code&gt; itself (from &lt;code&gt;next-auth@4.24.15&lt;/code&gt;'s &lt;code&gt;utils/parse-url.js&lt;/code&gt;) is short:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;parseUrl&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;_url2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;defaultUrl&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;http://localhost:3000/api/auth&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startsWith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;http&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;`https://&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;_url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;_url2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;_url2&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="k"&gt;void&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;_url2&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;defaultUrl&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="c1"&gt;// ...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The bug hinges on one line: &lt;code&gt;(_url2 = url) !== null &amp;amp;&amp;amp; _url2 !== void 0 ? _url2 : defaultUrl&lt;/code&gt;. That's just &lt;code&gt;url ?? defaultUrl&lt;/code&gt; written out — nullish coalescing. If &lt;code&gt;url&lt;/code&gt; is &lt;code&gt;undefined&lt;/code&gt;, this correctly falls back to &lt;code&gt;defaultUrl&lt;/code&gt;. But if &lt;code&gt;url&lt;/code&gt; is an &lt;strong&gt;empty string&lt;/strong&gt;, &lt;code&gt;""&lt;/code&gt; is not &lt;code&gt;null&lt;/code&gt; and not &lt;code&gt;undefined&lt;/code&gt;, so the nullish check passes it straight through. And a few lines up, &lt;code&gt;if (url &amp;amp;&amp;amp; !url.startsWith("http"))&lt;/code&gt; also short-circuits on falsy, so an empty string skips the &lt;code&gt;https://&lt;/code&gt; prefix step too. The empty string sails through both guards unmodified and lands in &lt;code&gt;new URL("")&lt;/code&gt;, which throws immediately: &lt;code&gt;TypeError: Invalid URL, input: ''&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;So somewhere, &lt;code&gt;NEXTAUTH_URL&lt;/code&gt; was resolving to &lt;code&gt;""&lt;/code&gt; — not undefined, not missing, but an actual empty string. And because this throw happens during module evaluation inside a Client Component that every route imports through the root layout, it aborts static generation for every route where that import gets evaluated — with no &lt;code&gt;try/catch&lt;/code&gt; in my own code anywhere near it, because the exception isn't inside a function call I made; it's inside a &lt;code&gt;&amp;lt;script&amp;gt;&lt;/code&gt;-equivalent module body that runs as a side effect of importing &lt;code&gt;next-auth/react&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This is a general property of &lt;code&gt;import&lt;/code&gt;, not something specific to this library: importing a module runs its top-level code immediately, synchronously, before you get a reference to anything it exports. A &lt;code&gt;try/catch&lt;/code&gt; you write can only guard the code between its braces — it has no reach into work a dependency chose to do the moment it was loaded. Singleton clients, feature-flag SDKs, analytics initializers, anything that reads &lt;code&gt;process.env&lt;/code&gt; at the top of a file rather than inside an exported function, can all fail this same way: the crash happens on import, in a stack frame that belongs to someone else's package, at a point in the render tree your own error boundaries were never positioned to reach.&lt;/p&gt;

&lt;h2&gt;
  
  
  The number that confirmed it
&lt;/h2&gt;

&lt;p&gt;The stack trace hit exactly 13 pages: &lt;code&gt;/&lt;/code&gt;, &lt;code&gt;/blog&lt;/code&gt;, four blog post pages, &lt;code&gt;/compare&lt;/code&gt;, &lt;code&gt;/faq&lt;/code&gt;, &lt;code&gt;/how-it-works&lt;/code&gt;, and four &lt;code&gt;/legal/*&lt;/code&gt; pages. Three routes in the app — &lt;code&gt;/signup&lt;/code&gt;, &lt;code&gt;/dashboard&lt;/code&gt;, &lt;code&gt;/dashboard/team&lt;/code&gt; — were &lt;strong&gt;not&lt;/strong&gt; affected, and all three had &lt;code&gt;export const dynamic = "force-dynamic"&lt;/code&gt;, meaning Next.js skips prerendering them at build time entirely. They never evaluate the layout's module tree during &lt;code&gt;next build&lt;/code&gt; the same way, so they never hit the crash.&lt;/p&gt;

&lt;p&gt;Thirteen static pages, thirteen failures, zero false positives among the dynamic ones. That's the kind of exact match that turns a hypothesis into a near-certainty before you've even fixed anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the empty string was coming from
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;NEXTAUTH_URL&lt;/code&gt; in &lt;code&gt;.env.local&lt;/code&gt; had a real value. So why was the build reading &lt;code&gt;""&lt;/code&gt;?&lt;/p&gt;

&lt;p&gt;Next.js loads env files in a specific priority order, and it's different depending on which command you run. For &lt;code&gt;next build&lt;/code&gt; (which runs with &lt;code&gt;NODE_ENV=production&lt;/code&gt;), the order is roughly:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;code&gt;.env.production.local&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;.env.local&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;.env.production&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;.env&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;code&gt;next dev&lt;/code&gt;, meanwhile, prefers &lt;code&gt;.env.development.local&lt;/code&gt; — a completely different file. That's exactly why &lt;code&gt;npm run dev&lt;/code&gt; never showed the problem: it was never reading the file that had the bad value.&lt;/p&gt;

&lt;p&gt;And sitting in my working directory, dated about a month earlier, was a leftover &lt;code&gt;.env.production.local&lt;/code&gt;. I'd created it running &lt;code&gt;vercel env pull .env.production.local --environment=production&lt;/code&gt; while debugging something else entirely. What I hadn't caught at the time: Vercel's CLI doesn't return the actual values for env vars marked "sensitive" via &lt;code&gt;env pull&lt;/code&gt; — it writes them back as &lt;strong&gt;empty strings&lt;/strong&gt;. I'd pulled a file full of blanks, moved on to the actual thing I was debugging, and never deleted it. Next.js then quietly gave that file top priority over my real &lt;code&gt;.env.local&lt;/code&gt; on every production build, from that day forward, for everyone who ran &lt;code&gt;next build&lt;/code&gt; locally.&lt;/p&gt;

&lt;p&gt;And here's the part that let it hide for a month: &lt;code&gt;.env.production.local&lt;/code&gt; matches the &lt;code&gt;.env*.local&lt;/code&gt; pattern that ships in every default Next.js &lt;code&gt;.gitignore&lt;/code&gt;, right alongside &lt;code&gt;.env.local&lt;/code&gt; itself. It was never staged, never committed, never visible in a diff, never something a teammate or a past version of me could have flagged in review. It just sat on disk, correctly ignored by git for exactly the reason &lt;code&gt;.env.local&lt;/code&gt; files are supposed to be ignored — because they're meant to hold real secrets — while accidentally also being an override file that Next.js would treat as authoritative the next time anyone ran a production build from that machine. The gitignore rule that protects secrets and the priority rule that resolves env files are two completely independent systems that happen to both key off the same filename convention, and nothing forces them to agree with your intentions at the same time.&lt;/p&gt;

&lt;p&gt;The fix, once identified, took one command:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;rm&lt;/span&gt; .env.production.local
npx next build
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first time I ran that rebuild, it still looked broken — until I noticed the &lt;code&gt;.next/BUILD_ID&lt;/code&gt; timestamp didn't match anything I'd just done. I had two checkouts of the repo open (a worktree for an unrelated branch, and the main one), and the rebuild had run in the wrong directory against a &lt;code&gt;.next&lt;/code&gt; cache that had nothing to do with the fix. Once I chained the &lt;code&gt;cd&lt;/code&gt; and the build into a single command instead of relying on whichever shell happened to have focus, the real result came back clean: all 24 pages generated, zero errors, &lt;code&gt;Environments: .env.local&lt;/code&gt; printed at the top of the build log where &lt;code&gt;.env.production.local&lt;/code&gt; used to be listed first.&lt;/p&gt;

&lt;p&gt;It's a small thing, but it's the same category of mistake as the bug itself — trusting which directory or which file you &lt;em&gt;think&lt;/em&gt; you're operating in, instead of checking what the tool actually reports it used. The build log telling you which env files it loaded, in priority order, was there the whole time; I just hadn't been reading that line until this happened.&lt;/p&gt;

&lt;h2&gt;
  
  
  The lesson that generalizes
&lt;/h2&gt;

&lt;p&gt;A few things about this are specific to Next.js and NextAuth, but the shape of the bug isn't:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;next build&lt;/code&gt; and &lt;code&gt;next dev&lt;/code&gt; don't read the same env files.&lt;/strong&gt; This is documented, but it's the kind of documented-but-easy-to-forget fact that only bites you when a stale file happens to exist. If a bug only reproduces in one command and not the other, checking &lt;em&gt;which&lt;/em&gt; &lt;code&gt;.env*&lt;/code&gt; files each one actually loads should be an early step, not a last resort.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A tool that writes credentials back to disk can write blanks instead of failing loudly.&lt;/strong&gt; &lt;code&gt;vercel env pull&lt;/code&gt; succeeding with exit code 0 told me nothing about whether the values inside were real. A pull that returns empty strings for sensitive vars is functionally a silent failure wearing a success message. If you ever run a credential-pulling command for a one-off debugging session, treat the output file as radioactive — name it something that will never be picked up by convention (not &lt;code&gt;.env.production.local&lt;/code&gt;), and delete it the moment you're done.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Two wrong guesses is a signal to stop guessing.&lt;/strong&gt; Both of my first theories were reasonable and both were about the same file (&lt;code&gt;.env.local&lt;/code&gt;) that turned out to be completely fine. The thing that actually worked was going one level closer to the truth: instead of theorizing about what &lt;em&gt;might&lt;/em&gt; be in an env file, I grepped the compiled output for the literal code path that was throwing. Compiled output doesn't lie about what identifiers it references. If you find yourself on a second incorrect theory, that's usually the point to trade speculation for whatever raw artifact — a compiled bundle, a request log, a database row — can tell you definitively what actually ran.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A &lt;code&gt;try/catch&lt;/code&gt; only catches what runs inside it.&lt;/strong&gt; My layout's &lt;code&gt;try/catch&lt;/code&gt; around &lt;code&gt;getServerSession()&lt;/code&gt; was correct and necessary, but it gave me false confidence that env-var failures near auth were handled. A separate, unrelated code path — a third-party client component's module-level side effect — could still crash the same render with no relation to the code I'd defended. When a library does real work at import time rather than inside a function you call, no amount of wrapping your own call sites protects you from it.&lt;/p&gt;

&lt;p&gt;None of this was a Next.js bug or a NextAuth bug. Both behaved exactly as documented. It was one leftover debugging artifact, two library behaviors that are individually reasonable, and a build tool that reads a different file than the one I was staring at. That combination is what took an afternoon to trace back to a single &lt;code&gt;rm&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  A short checklist, if this ever happens to you
&lt;/h2&gt;

&lt;p&gt;If &lt;code&gt;next build&lt;/code&gt; fails somewhere &lt;code&gt;next dev&lt;/code&gt; doesn't, and the error points toward configuration rather than logic, this is roughly the order I'd check things in now:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;ls -la .env*&lt;/code&gt; in the project root. Anything named &lt;code&gt;.env.production.local&lt;/code&gt;, &lt;code&gt;.env.production&lt;/code&gt;, or similar that you don't remember writing on purpose is suspect by default — delete it and rebuild before doing anything else.&lt;/li&gt;
&lt;li&gt;If the build output already exists, grep the compiled chunks for the specific string in the error (a URL, a variable name, whatever's unique) before forming a third theory about source files you've already checked twice.&lt;/li&gt;
&lt;li&gt;Treat any command that pulls secrets from a remote source (&lt;code&gt;vercel env pull&lt;/code&gt;, &lt;code&gt;aws secretsmanager get-secret-value&lt;/code&gt;, &lt;code&gt;doppler secrets download&lt;/code&gt;, and so on) as something that can succeed with exit code 0 and still hand you garbage. Sensitive/encrypted values are the most likely category to come back blank, redacted, or truncated depending on the tool's permission model — check a value, don't just check the exit code.&lt;/li&gt;
&lt;li&gt;If a third-party package throws during static generation, don't assume the crash originates in a function you called. Client components and providers can run real code — including &lt;code&gt;new URL(...)&lt;/code&gt;, &lt;code&gt;fetch&lt;/code&gt;, or anything else — the moment they're imported, before your own &lt;code&gt;try/catch&lt;/code&gt; blocks ever get a chance to run.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Every one of those steps would have gotten me to the answer faster than the two guesses I actually made.&lt;/p&gt;

</description>
      <category>nextjs</category>
      <category>typescript</category>
      <category>webdev</category>
      <category>debugging</category>
    </item>
  </channel>
</rss>
