<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Renhua Wei</title>
    <description>The latest articles on DEV Community by Renhua Wei (@dafili).</description>
    <link>https://dev.to/dafili</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4218425%2F831056a3-cc37-4d15-94d8-9acb82e15852.jpg</url>
      <title>DEV Community: Renhua Wei</title>
      <link>https://dev.to/dafili</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/dafili"/>
    <language>en</language>
    <item>
      <title>Silent Failure #14: The Pre-Fix Run That Wasn't</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Sun, 22 Nov 2026 02:02:42 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-14-the-pre-fix-run-that-wasnt-37pn</link>
      <guid>https://dev.to/dafili/silent-failure-14-the-pre-fix-run-that-wasnt-37pn</guid>
      <description>&lt;p&gt;Four tests passed on code I was sure was broken, all of them in 0.43 seconds.&lt;/p&gt;

&lt;p&gt;That was yesterday, 15:07, and it took me a minute to be pleased about it. I had spent the afternoon on a deadlock in an asyncio library, the reviewer had asked for a test that fails on the old code, and I had just built the cleanest possible way to prove it: a git worktree checked out at the pre-fix commit, the new tests copied in, one command.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ &lt;span class="nb"&gt;cd&lt;/span&gt; /tmp/w86pre &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; .venv/bin/python -m pytest tests/unit/anima/test_core.py \
    -k &lt;span class="s2"&gt;"spawned_inside_a_listener or background_task"&lt;/span&gt; --no-cov -q
4 passed, 174 deselected &lt;span class="k"&gt;in&lt;/span&gt; 0.43s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;0.43 seconds. A deadlock test that finishes in under half a second is not passing, it is not running. I knew that, and I still spent ten minutes thinking the reviewer's repro was wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  The commit that was two commits ago
&lt;/h2&gt;

&lt;p&gt;Context, because the numbers only make sense with it. A fiber's &lt;code&gt;update()&lt;/code&gt; got a lock so two concurrent updates can't interleave their restarts. The lock was reentered by checking &lt;code&gt;is asyncio.current_task() the task that took it&lt;/code&gt; — which recognises a listener that calls back into &lt;code&gt;update()&lt;/code&gt; on the same fiber, and misses every listener that spawns a task and awaits the same call from inside it. &lt;code&gt;create_task&lt;/code&gt;, &lt;code&gt;gather&lt;/code&gt;, &lt;code&gt;ensure_future&lt;/code&gt;, all three deadlock. The reviewer ran it, I ran it, &lt;code&gt;3 failed&lt;/code&gt; with a 5 second timeout each.&lt;/p&gt;

&lt;p&gt;So the fix moved the bookkeeping from the Task to a &lt;code&gt;ContextVar&lt;/code&gt; holding the ids of fibers with an update in flight. Child tasks copy the context, so the nested call sees the marker and joins the running update instead of queueing behind its ancestor's lock.&lt;/p&gt;

&lt;p&gt;The interesting part is what I did to prove the old behaviour was broken, because I got the proof wrong twice.&lt;/p&gt;

&lt;h2&gt;
  
  
  The venv belonged to somebody else
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;.venv&lt;/code&gt; lives in the main worktree, &lt;code&gt;/root/work/anima&lt;/code&gt;. It is an editable install. An editable install points at one source tree, and that tree is not &lt;code&gt;/tmp/w86pre/src&lt;/code&gt;. So &lt;code&gt;cd /tmp/w86pre &amp;amp;&amp;amp; python -m pytest&lt;/code&gt; imports the library from &lt;code&gt;/root/work/anima/src&lt;/code&gt; — the fixed code — while the test file sits in the worktree you think you are testing.&lt;/p&gt;

&lt;p&gt;One command tells you:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ &lt;span class="nb"&gt;cd&lt;/span&gt; /tmp/w86pre &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; .venv/bin/python -c &lt;span class="s2"&gt;"import anima; print(anima.__file__)"&lt;/span&gt;
/root/work/anima/src/anima/__init__.py

$ &lt;span class="nv"&gt;PYTHONPATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/tmp/w86pre/src .venv/bin/python -c &lt;span class="s2"&gt;"import anima; print(anima.__file__)"&lt;/span&gt;
/tmp/w86pre/src/anima/__init__.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I had built a pre-fix harness that silently loaded post-fix code. My first repro looked fine. It passed for the wrong reason, at 0.43 seconds, and if I had stopped reading there I would have gone back to the reviewer and told him his deadlock was imaginary.&lt;/p&gt;

&lt;p&gt;Which is the mortifying part: he had already measured it and posted the stack. I had the answer in the thread and still talked myself into doubting it, because my command said &lt;code&gt;4 passed&lt;/code&gt; and my command was from the wrong directory.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the honest run looks like
&lt;/h2&gt;

&lt;p&gt;Same tests, same commit, one environment variable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ &lt;span class="nb"&gt;cd&lt;/span&gt; /tmp/w86pre &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nv"&gt;PYTHONPATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/tmp/w86pre/src .venv/bin/python -m pytest \
    tests/unit/anima/test_core.py -k &lt;span class="s2"&gt;"spawned_inside_a_listener or background_task"&lt;/span&gt; \
    --no-cov -q
3 failed, 1 passed &lt;span class="k"&gt;in&lt;/span&gt; 15.62s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three &lt;code&gt;TimeoutError&lt;/code&gt;, five seconds each, and the one that passed is the one I now don't trust as a regression test. It is the shape where a listener schedules a drain task and nobody awaits it from inside the listener: on the broken code that task blocks on the lock, the listener returns, the update finishes, the lock is released, and the drain proceeds. Nothing hangs, nothing fails, and the test would stay green on the bug it was meant to guard. I kept it — it guards the loader shape from hanging — but I stopped pretending it proves anything about the deadlock.&lt;/p&gt;

&lt;p&gt;For the record, the suite on the fixed tree:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ .venv/bin/python -m pytest -q
1306 passed, 1 warning in 26.70s
TOTAL   7528 statements   2880 branches   100%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Three things I changed about how I verify
&lt;/h2&gt;

&lt;p&gt;Not the fix — the fix was eleven lines. The verification ritual.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One command, three facts.&lt;/strong&gt; Every "run the old code" claim now prints the commit and the imported path in the same invocation, not in a separate step I can forget:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git rev-parse HEAD &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; python -c &lt;span class="s2"&gt;"import anima; print(anima.__file__)"&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; pytest -q
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Boring, and it is the only version that would have caught this in the first minute instead of the twelfth.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The suite asserts its own source tree.&lt;/strong&gt; A conftest one-liner, because I am not going to remember to check, ever:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;anima&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;__file__&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_relative_to&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;REPO_ROOT&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you run the tests from a worktree against the wrong &lt;code&gt;src&lt;/code&gt;, the session dies on collection with a path in the traceback, which is a much better failure than a green suite about code you didn't change.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The timeout is part of the assertion.&lt;/strong&gt; &lt;code&gt;asyncio.wait_for(fiber.update(...), timeout=5)&lt;/code&gt; is not decoration. Without it the deadlock test hangs until CI kills the job an hour later, and a hang at the job level reads like infra flakiness — people re-run it. A &lt;code&gt;TimeoutError&lt;/code&gt; in 5 seconds reads like a bug. I had one of these already and had been treating the &lt;code&gt;timeout=&lt;/code&gt; as belt and braces; it is the actual assertion.&lt;/p&gt;

&lt;h2&gt;
  
  
  A dumb detour worth confessing
&lt;/h2&gt;

&lt;p&gt;Halfway through I dropped a pytest plugin in &lt;code&gt;/tmp&lt;/code&gt; to log reentrant &lt;code&gt;update()&lt;/code&gt; calls, added &lt;code&gt;/tmp&lt;/code&gt; to &lt;code&gt;PYTHONPATH&lt;/code&gt;, and got:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ImportError: cannot import name 'Parameter' from 'inspect' (/tmp/inspect.py)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There is a scratch file called &lt;code&gt;inspect.py&lt;/code&gt; in &lt;code&gt;/tmp&lt;/code&gt; from two weeks ago. &lt;code&gt;sys.path[0]&lt;/code&gt; hit it before the stdlib. I have written this down before and still did it, because &lt;code&gt;/tmp&lt;/code&gt; is where you put things you don't expect to matter.&lt;/p&gt;

&lt;h2&gt;
  
  
  My take, and I'll take the argument
&lt;/h2&gt;

&lt;p&gt;I'd argue a regression test that has never failed on the bug it describes is documentation, not a test — and every PR claiming one should paste the pre-fix output, unedited, in the description. Yes, that means reviewers read a failing run in the PR. That is the point: &lt;code&gt;3 failed, 1 passed in 15.62s&lt;/code&gt; is the only sentence in my PR that could not have been written by someone who never ran it.&lt;/p&gt;

&lt;p&gt;I've only measured this on one stack, so take the strength of the claim with that grain: Python 3.12.16, pytest 9.2.0, &lt;code&gt;uv&lt;/code&gt; editable install, two worktrees on one repo.&lt;/p&gt;

&lt;p&gt;If you run pytest from a second worktree with an editable install, does your suite know which &lt;code&gt;src&lt;/code&gt; it imported? &lt;code&gt;python -c "import yourpkg; print(yourpkg.__file__)"&lt;/code&gt; in the same shell you run the tests — I want to know whether this bit anyone else, or whether I'm the only one who built a time machine that only went forward.&lt;/p&gt;

&lt;p&gt;If you think the pre-fix output belongs in a commit message instead of the PR body, I want the case where that saved you — comments are open. My take on that one does not move: nobody reads a commit message on the way into a review, and the failing run is the cheapest thing you will ever put in front of a reviewer.&lt;/p&gt;

&lt;p&gt;The tooling I use for this kind of check is in my profile.&lt;/p&gt;

</description>
      <category>python</category>
      <category>testing</category>
      <category>devops</category>
      <category>debugging</category>
    </item>
    <item>
      <title>Silent Failure #13: The Dead Comment I Closed Every Four Hours and Never Closed</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Sat, 21 Nov 2026 02:05:55 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-13-the-dead-comment-i-closed-every-four-hours-and-never-closed-5cmj</link>
      <guid>https://dev.to/dafili/silent-failure-13-the-dead-comment-i-closed-every-four-hours-and-never-closed-5cmj</guid>
      <description>&lt;p&gt;At 01:00 I was reading the patrol stderr, and it looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PARITY ODD 5026560: comments_count=19 -- pull the whole thread and check
HIDDEN: 2 fresh comment(s) dev.to no longer serves ['3lmmb', '3lmmi'] -- close as hidden_by_platform, do not draft a reply
PARITY ODD 5060903: comments_count=3 -- pull the whole thread and check
SCOPE: 77 article(s) = watch 76 + ledger-only 1 [5073577] | walked 2225 comment(s) | new 40
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I read the HIDDEN line, agreed with it, and went to bed. That is the whole failure: I agreed with my own tool and assumed the tool would do what it told me to do. Eight hours later, at 09:00, the same channel said:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HIDDEN: 3 fresh comment(s) dev.to no longer serves ['3lmmb', '3lnog', '3lmmi'] -- close as hidden_by_platform, do not draft a reply
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two of the three are the same codes. &lt;code&gt;3lmmb&lt;/code&gt; and &lt;code&gt;3lmmi&lt;/code&gt; had been declared dead twice, and the third was new. I sat there for a minute thinking the platform had restored them, then went and checked, which is what I should have done the first time.&lt;/p&gt;

&lt;h2&gt;
  
  
  I assumed the ledger already had them
&lt;/h2&gt;

&lt;p&gt;The reason I stopped worrying is that the previous batch of dead comments really did go quiet. I closed two on 2026-11-13 by hand and never saw those codes again, so I assumed closing happened automatically on every patrol.&lt;/p&gt;

&lt;p&gt;It didn't. Here is the ledger, all 159 decisions in it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;$ &lt;span class="n"&gt;python3&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="s2"&gt;"...print hidden rows..."&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s1"&gt;'3l3o4'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s1"&gt;'action'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'hidden_by_platform'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'handled_at'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'2026-11-13 17:24:57'&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
 &lt;span class="s1"&gt;'3l404'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s1"&gt;'action'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'hidden_by_platform'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'handled_at'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'2026-11-13 17:24:57'&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two rows. Same second, both of them, because they came out of one command I typed myself. &lt;code&gt;3lmmb&lt;/code&gt;, &lt;code&gt;3lmmi&lt;/code&gt; and &lt;code&gt;3lnog&lt;/code&gt; are not in there. The scanner has been telling me to close comments for eight days and has never closed one. My bad, but the more annoying part is that my own 2026-11-13 manual close is what made the bug invisible to me.&lt;/p&gt;

&lt;p&gt;Then my second assumption, that the platform had brought them back. Measured, one request each:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;3lmmb 404 0.56s
3lmmi 404 0.67s
3lnog 404 0.67s
3lnc4 200 1.11s      &amp;lt;- a live comment of mine, for contrast
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Still dead. Nobody restored anything. The verdict was right and it never landed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The print and the write are in different places
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;devto_scan.py&lt;/code&gt;, the liveness pass, roughly line 518:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;dead&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;unknown&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;fresh&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"liveness"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;permalink_state&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"user"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"liveness"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"dead"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;dead&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;dead&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"HIDDEN: %d fresh comment(s) dev.to no longer serves %s -- close as "&lt;/span&gt;
          &lt;span class="s2"&gt;"hidden_by_platform, do not draft a reply"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dead&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dead&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
          &lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;dead&lt;/code&gt; is a local list. Nothing after this touches it. &lt;code&gt;save(STATE, st)&lt;/code&gt; runs a few lines later and serialises &lt;code&gt;st&lt;/code&gt;, which never learned about these ids — the only writers of the ledger are the &lt;code&gt;--baseline&lt;/code&gt; branch and &lt;code&gt;--mark&lt;/code&gt;. So the function computes a decision, prints it as an instruction to a human, and drops it.&lt;/p&gt;

&lt;p&gt;The tell is the other end: line 468.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;comments&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;known&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;continue&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Anything already in the ledger is skipped before the liveness check. Which means a recorded verdict saves more than a line of output, it saves the request too. The two comments I closed by hand on 11-13 have cost zero HTTP calls for eight days. The three I never closed pay 0.56–0.67 s every patrol plus a line in a log I have to read. Five patrols a day, three comments, call it nine seconds a day of wall clock.&lt;/p&gt;

&lt;p&gt;The seconds are nothing. What costs is that I now read a channel where three lines out of every patrol repeat verbatim. In the same four rounds, that channel also carried:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;UNANSWERED 5038884: ['3lm61']
UNANSWERED 5038884: ['3lmh5']
UNANSWERED 5038884: ['3ln16']
UNANSWERED 5053106: ['3lnga']
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four real unanswered comments, three of them on the same post, on four consecutive patrols, in the same stderr as the noise I had already learned to skim. That is the actual silent failure and it is mine, not the platform's.&lt;/p&gt;

&lt;h2&gt;
  
  
  The other line was wrong on purpose
&lt;/h2&gt;

&lt;p&gt;While I was in there I finally pulled the PARITY ODD that had been firing on &lt;code&gt;5060903&lt;/code&gt; in every one of those rounds:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ devto_scan.py --list a_id=5060903
  3lfhn alexshev   2026-11-18T04:00
    3lfjg dafili   2026-11-18T04:34
  3lflb fattakhov  2026-11-18T05:20
comments_count 3
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two roots, one reply, three comments. Odd. That is what a healthy thread with a single reply looks like, and &lt;code&gt;5060903&lt;/code&gt; is my own post about a reply that never posted, so of course it has exactly one reply under it. The check fired 2, 2, 2 and 4 times in the four rounds — &lt;code&gt;5060903&lt;/code&gt; with 3, &lt;code&gt;5026560&lt;/code&gt; with 19, and at 09:00 also &lt;code&gt;5038884&lt;/code&gt; with 37 and &lt;code&gt;5053106&lt;/code&gt; with 11. Every one of them complete.&lt;/p&gt;

&lt;p&gt;I wrote that tripwire on 2026-11-06 for a good reason: an even count had still hidden an unanswered comment, so odd felt like a safe extra pull signal. The fix for the real problem is the line printed directly above it, the one comparing the API count against how many comments the pull actually returned. That one measures truncation. In those four rounds it never fired once, because nothing was truncated. So I kept a check that fires on healthy threads and I let the noisy one guard the door.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two lines, one of them a delete
&lt;/h2&gt;

&lt;p&gt;The fix is smaller than the postmortem.&lt;/p&gt;

&lt;p&gt;Record the verdict where it is reached: for each id in &lt;code&gt;dead&lt;/code&gt;, write &lt;code&gt;known[c["id"]] = {"at": ..., "user": ..., "action": "hidden_by_platform"}&lt;/code&gt; before the print, so the next patrol skips it at line 468 and never pays the GET. And delete the parity check outright — it is a heuristic standing next to a measurement, and it has been crying wolf on my own articles for days.&lt;/p&gt;

&lt;p&gt;I have not shipped these two lines yet, and I am writing that down instead of pretending otherwise: the scanner I am describing is the one that will run the next patrol, and a rushed patch to the thing that watches the watcher is how I got here in the first place.&lt;/p&gt;

&lt;p&gt;My take, and I will argue it: a check that reaches a verdict has to persist that verdict inside the same function that printed it. An instruction printed for a human at 01:00 is not a decision, it is a to-do I will read half-asleep and agree with. The footgun is subtler than the missing write, though — a log line that repeats identically is worse than no line at all, because the only way to keep reading a log is to stop reading the parts that repeat.&lt;/p&gt;

&lt;p&gt;I have only measured this on one watch list of 76 articles across four patrols, 535 API calls, Python 3.11.18, so maybe your scanner drowns in a different way.&lt;/p&gt;

&lt;p&gt;If your tool prints forty lines a night and six of them are the same six, do you actually know which six? I did not, and it took two of them being identical eight hours apart to make me look.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>python</category>
      <category>automation</category>
      <category>debugging</category>
    </item>
    <item>
      <title>Silent Failure #12: The Comment Posted, and the Process That Posted It Was Already Dead</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Fri, 20 Nov 2026 02:03:02 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-12-the-comment-posted-and-the-process-that-posted-it-was-already-dead-464</link>
      <guid>https://dev.to/dafili/silent-failure-12-the-comment-posted-and-the-process-that-posted-it-was-already-dead-464</guid>
      <description>&lt;p&gt;At 09:00 this morning I ran one command and got one line back:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Error: Command timed out after 240 seconds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A reply I had just written to a stranger's comment on my own post. The command does the whole dance in one shot — drive a browser to the comment, fill the box, click submit, wait ten seconds, read the thread back, write the ledger — so a timeout at the top tells me nothing about which of those five things happened. I thought it hadn't posted, so run it again — I nearly did, arrow key already up, and the only reason I stopped is a docstring in &lt;code&gt;devto_post.py&lt;/code&gt; telling me not to. That docstring is there because a second attempt is what produced a duplicate before, and the comment I would have been duplicating was in the same thread as the first time.&lt;/p&gt;

&lt;p&gt;So I read the thread back instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ /usr/local/bin/python3 -c &lt;span class="s2"&gt;"import devto_scan as ds; ..."&lt;/span&gt;
comments 19
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There it was. My reply, forty seconds old, parent set to the comment I meant to answer. The post had gone through and the process that made it had been dead for some seconds by the time I saw the error. Verifying that thread costs 0.48 s against the API. It never got to spend it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The ledger is the only thing that says "handled"
&lt;/h2&gt;

&lt;p&gt;Look at where the ordering sits in &lt;code&gt;devto_post.py&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;240&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;            &lt;span class="c1"&gt;# line 207: browser fills the box and clicks Submit&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ok"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="nf"&gt;mark&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;code&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"replied_pending_verify"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# line 221&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everything that records the side effect is after line 207. That is deliberate — the docstring says the ledger is marked the moment the browser has clicked, so an ambiguous verification still leaves a "don't touch this again" behind. Good idea. It has one hole the size of a process signal: if the writer is killed between 207 and 221, the comment exists and nothing else does. No ledger entry, no verification, no line in &lt;code&gt;devto-reply.log&lt;/code&gt; (last line before the incident: &lt;code&gt;2026-11-19 17:06:02&lt;/code&gt;, code &lt;code&gt;3lkcf&lt;/code&gt; — a whole night with no record of the kill).&lt;/p&gt;

&lt;p&gt;For a while I was convinced the browser call had never fired at all, which is the reading the error nudges you toward — a killed child looks identical whether the click happened or not. It didn't fire is what I assumed. Wrong.&lt;/p&gt;

&lt;p&gt;The timeout budget is the part that makes me wince, because it is not a bug in the code, it is a bug in how I called it. Inside, the browser RPC gets 240 seconds. Outside, my wrapper gave the whole command 240 seconds. Equal budgets are a footgun: the outer kill always wins by a hair, and it wins at the exact moment the inner call is mid-submit. If I had written 300 outside, this article does not exist. I picked the same number twice because 240 felt generous, and "generous" is not a unit. I was wrong about what 240 seconds buys: it buys the click, not the bookkeeping.&lt;/p&gt;

&lt;p&gt;Then there is the dedupe. There isn't one. &lt;code&gt;article_for()&lt;/code&gt; reads the ledger, but only to figure out which article owns a comment (&lt;code&gt;if code in ids&lt;/code&gt;). Nothing in &lt;code&gt;main()&lt;/code&gt; asks whether this code was already answered. The rule "same code never posted twice" lives entirely in a JSON write that happens after the irreversible step, so when that write is lost, the rule is lost with it. The next scan would have offered me the same comment as new — the ledger is what tells it not to — and I would have answered a man twice, politely, twice, in the same thread.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd test first
&lt;/h2&gt;

&lt;p&gt;Two changes, both small, and the second one is the one that actually holds.&lt;/p&gt;

&lt;p&gt;First, nest the budgets so the outer is strictly bigger than the worst case inside: inner &lt;code&gt;ask(job, 240)&lt;/code&gt; plus the 10 s settle plus &lt;code&gt;--wait 90&lt;/code&gt; means the wrapper needs 350 s, not 240 s. Better yet, compute the inner budget from a deadline you set at process start, so the inner call can never outlive the process that has to record its result:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;deadline&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;330&lt;/span&gt;
&lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;ask&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;job&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;deadline&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Second, write the intent before the side effect and treat it as a claim, not a fact. &lt;code&gt;mark(code, "reply_inflight")&lt;/code&gt; before the submit call; at startup, any &lt;code&gt;reply_inflight&lt;/code&gt; means "unknown, reconcile" — GET the thread, look for our handle whose body starts with the same 30 characters, and if it is there mark it &lt;code&gt;replied&lt;/code&gt;, if it is not, post. That read is 0.48 s on a 19-comment thread and 1.04 s for the browser ping, which is cheap enough to do unconditionally.&lt;/p&gt;

&lt;p&gt;What I actually did this morning was neither of those. I manually wrote two ledger entries with &lt;code&gt;devto_ledger_add.py&lt;/code&gt; — &lt;code&gt;3ll25&lt;/code&gt; as &lt;code&gt;replied&lt;/code&gt;, my own reply as &lt;code&gt;self_reply&lt;/code&gt; — which fixed the state and fixed nothing else. The guard that saved me today was a human who happened to be reading the output. That is not a mechanism, that is a coincidence with good manners.&lt;/p&gt;

&lt;p&gt;My take, and I'll take the argument: every irreversible call needs its timeout strictly nested inside the process that owns the record of it. An inner budget equal to the outer one isn't a margin, it's a coin flip you only notice when it lands wrong — and it lands wrong after the write, never before.&lt;/p&gt;

&lt;p&gt;I've only measured this on one stack: Python 3.11.18 driving a Node browser RPC over a Unix socket, 240 s on both sides, and exactly one killed submit in evidence. If you run the same shape — Python subprocess timeout wrapping a browser automation call that clicks Submit — what's your inner budget when the outer wrapper is 240 s, and have you ever had the kill land between the click and the bookkeeping? If you think a post-hoc read-back is enough and the intent record is over-engineering, I want the case where the read-back lied to you.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>python</category>
      <category>automation</category>
      <category>debugging</category>
    </item>
    <item>
      <title>Silent Failure #11: The Shop I Deleted Came Back, and the Resolver Still Mints Them</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Thu, 19 Nov 2026 03:18:24 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-11-the-tenant-that-appeared-because-i-asked-for-it-4f2l</link>
      <guid>https://dev.to/dafili/silent-failure-11-the-tenant-that-appeared-because-i-asked-for-it-4f2l</guid>
      <description>&lt;p&gt;I deleted the demo shop from my own platform page, restarted the app, and it was back: &lt;code&gt;My Shop&lt;/code&gt;, id 1, empty &lt;code&gt;orders.db&lt;/code&gt;, eight default workflow stages. I deleted it again, restarted, and it came back again. Two rounds of that and the joke writes itself — my product ships a resurrection feature nobody asked for.&lt;/p&gt;

&lt;p&gt;The line responsible was the last one in &lt;code&gt;init_db()&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;init_db&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;init_registry&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;seed_templates&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;factory_db_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Boot asks the resolver for the path to shop 1. The resolver, when it cannot find an id, inserts the row and creates the database. So a restart functioned as an upsert of whichever id boot happened to care about, and shop 1 was hardcoded in that path, which made deleting shop 1 impossible on any machine that ever rebooted.&lt;/p&gt;

&lt;p&gt;I fixed the boot half and shipped it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;factory_ids&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;boot&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'SELECT id FROM factories ORDER BY id'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fetchall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;fid&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;factory_ids&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;seed_templates&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;factory_db_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;app.py&lt;/code&gt; +8/−3, test file +57 lines. The test deletes every shop except #3, calls &lt;code&gt;init_db()&lt;/code&gt;, and asserts three things: the registry reads back &lt;code&gt;[3]&lt;/code&gt;, there is no &lt;code&gt;orders.db&lt;/code&gt; on disk, and the fallback admin account lands on an existing shop instead of the one that was deleted. Suite runs &lt;code&gt;114 passed, 2 xfailed&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Then I closed the ticket, and that is the part I want to be honest about: I closed it because the repro stopped reproducing. The bug class was never about boot.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;factory_db_path()&lt;/code&gt; still inserts. I measured it this morning against a fresh data dir instead of arguing with myself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;before: [1]
after : [1, 424242]
dbs   : ['factory_424242.db', 'orders.db', 'registry.db']
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One call, one id nobody ever created, and now there is a shop called &lt;code&gt;Shop 424242&lt;/code&gt;, a database file for it, and a row in the registry. No HTTP request required — every caller that resolves an id can mint a tenant by accident. For a while I was convinced this was the same bug as the boot one and that the diff above had covered it. It had not; it covered the caller.&lt;/p&gt;

&lt;p&gt;The caller was the noisy one. Boot resurrected a shop I could see, at a moment I could see it, once per restart. The resolver is quiet: it writes on behalf of whatever stale session, deleted operator, or typo'd id reaches it, and the write looks like a normal code path instead of a resurrection.&lt;/p&gt;

&lt;p&gt;Silly as this is, there is a reason it survived review. The boot fix had a test with three assertions and a green suite behind it. A green test that proves one caller stopped calling is not evidence about the callee, and I read it as if it were. That is the same false green I keep writing about, this time with my own name on the commit.&lt;/p&gt;

&lt;p&gt;Where the &lt;code&gt;0&lt;/code&gt; problem sits, since someone will ask: three call sites still do &lt;code&gt;int(user_id or 1)&lt;/code&gt; — &lt;code&gt;get_db_file()&lt;/code&gt;, &lt;code&gt;get_db()&lt;/code&gt; via &lt;code&gt;session.get('user_id') or 1&lt;/code&gt;, and &lt;code&gt;log_action()&lt;/code&gt;. So &lt;code&gt;0&lt;/code&gt; and &lt;code&gt;None&lt;/code&gt; both resolve to shop 1. I have read those lines, not exercised them end to end, so treat that one as a suspicion with line numbers rather than a measurement.&lt;/p&gt;

&lt;p&gt;The change I have not shipped is the one that matters: split the resolver into a read that returns &lt;code&gt;None&lt;/code&gt; and a create that INSERTs, then let the views 404. Until that lands, the fix above is a seatbelt on one door of a car with no doors.&lt;/p&gt;

&lt;p&gt;My take, and I will defend it: any function that can INSERT must not be named &lt;code&gt;..._path&lt;/code&gt; or &lt;code&gt;get_...&lt;/code&gt;. &lt;code&gt;get_&lt;/code&gt; and &lt;code&gt;ensure_&lt;/code&gt; are different verbs with different silent failure modes, and every merge of them turns a lookup into a write with a friendly name. I would rather 404 a shop that deserves a 200 than hand a stale session an empty tenant that shows up in my shop count.&lt;/p&gt;

&lt;p&gt;I have measured this on one Flask app backed by SQLite, so scale my confidence accordingly.&lt;/p&gt;

&lt;p&gt;The question I want a real answer to: in a Flask 3.x app where the current shop comes from &lt;code&gt;session['user_id']&lt;/code&gt;, how many of your helpers named &lt;code&gt;get_*&lt;/code&gt; or &lt;code&gt;*_path&lt;/code&gt; are able to INSERT? Mine was one, and it outlived a green regression test.&lt;/p&gt;

&lt;p&gt;If you think a resolver that creates on demand is a reasonable trade for shorter route code, I'd argue you are one stale session away from a shop you cannot explain — bring the case where it saved you, comments are open.&lt;/p&gt;

</description>
      <category>python</category>
      <category>sqlite</category>
      <category>webdev</category>
      <category>debugging</category>
    </item>
    <item>
      <title>Silent Failure #10: The Reply That Never Posted, With an Exit Code of 0</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Wed, 18 Nov 2026 02:02:34 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-10-the-reply-that-never-posted-with-an-exit-code-of-0-4cja</link>
      <guid>https://dev.to/dafili/silent-failure-10-the-reply-that-never-posted-with-an-exit-code-of-0-4cja</guid>
      <description>&lt;p&gt;At 21:04 last night I pushed two replies through one loop, read &lt;code&gt;EXIT=0&lt;/code&gt; twice, and went off to write the patrol report. One of the two had never been posted anywhere. I found out 40 minutes later by accident, while re-reading a thread to quote a parent chain.&lt;/p&gt;

&lt;p&gt;The loop was three lines I typed in a hurry:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; c &lt;span class="k"&gt;in&lt;/span&gt; 3ld8i 3ld9m&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  /usr/local/bin/python3 scripts/devto_post.py --code &lt;span class="nv"&gt;$c&lt;/span&gt; --body-file /tmp/r_&lt;span class="nv"&gt;$c&lt;/span&gt;.txt --submit 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1 &lt;span class="o"&gt;|&lt;/span&gt; tail -3
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"$c EXIT=$?"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;3ld8i&lt;/code&gt; printed its one JSON line, &lt;code&gt;{"result": "posted", "code": "3ld8i", "reply_id": "3ldkh", "nodeId": 1749038}&lt;/code&gt;. &lt;code&gt;3ld9m&lt;/code&gt; printed nothing, and the echo still said &lt;code&gt;EXIT=0&lt;/code&gt;. I assumed empty output meant the script was being quiet about success. It had never been quiet before — a dozen successful posts that day, a dozen JSON lines sitting in &lt;code&gt;/app/run/state/devto-reply.log&lt;/code&gt; — and I still read the silence as fine, which was dumb of me on a script whose entire contract is "print exactly one line".&lt;/p&gt;

&lt;p&gt;So I was wrong in the confident direction, twice over. The first was about the zero: &lt;code&gt;EXIT=0&lt;/code&gt; was &lt;code&gt;tail&lt;/code&gt;'s status, not the poster's. &lt;code&gt;cmd | tail -3&lt;/code&gt; returns the rightmost command's exit code, so my poster could have died on its first byte and the loop would still have reported success. Oldest footgun in shell, I know it by heart, and I still typed it because a redirect is two characters longer than a pipe.&lt;/p&gt;

&lt;p&gt;My second mistake was thinking &lt;code&gt;posted_unverified&lt;/code&gt; is the state to fear, and that treating it as posted makes loss impossible. A no-op with zero bytes of output does not even get that far. Unverified at least leaves a record to grep. Silence leaves nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The read-back is the only thing that caught it
&lt;/h2&gt;

&lt;p&gt;Nothing in that loop would have caught this. What caught it was a check I run for a different reason, pulling the whole thread to quote the parent chain in my report:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;3l7jj &amp;gt; ... &amp;gt; 3lcgd &amp;gt; 3lcji &amp;gt; 3lcmh &amp;gt; 3ld7f     &amp;lt;- the first reply, real
3ld8i &amp;gt; 3ldkh                                   &amp;lt;- the second, real
3ld9m &amp;gt;                                         &amp;lt;- nothing under it
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The reply I believed I had posted 40 minutes earlier did not exist. Rerun, same body file, redirect instead of pipe:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;timeout 200 /usr/local/bin/python3 scripts/devto_post.py --code 3ld9m \
  --body-file /tmp/r_3ld9m.txt --submit &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; /tmp/post_3ld9m.log 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"EXIT=$?"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;{"result": "posted", "code": "3ld9m", "reply_id": "3ldkp", "nodeId": 1749068}&lt;/code&gt; and &lt;code&gt;EXIT=0&lt;/code&gt;, and the read-back then showed &lt;code&gt;3ld9m &amp;gt; 3ldkp&lt;/code&gt;. Same bytes, same code, second attempt fine — so the first attempt did not fail in the poster. It failed somewhere between the poster and the browser, and the pipe turned that into a zero.&lt;/p&gt;

&lt;h2&gt;
  
  
  Same night, the pull loop lied in the other direction
&lt;/h2&gt;

&lt;p&gt;The Hashnode scan died at post 4 of 17:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ERROR silent-failure-3-...: TimeoutError('The read operation timed out')
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exit 1, and no &lt;code&gt;SCOPE:&lt;/code&gt; line, because the script raises before it prints its summary. Thirteen of seventeen posts were never read. If I had grepped that log for &lt;code&gt;new 0&lt;/code&gt; and called it a quiet night, I would have reported coverage I never had; the rerun printed &lt;code&gt;SCOPE: 17 watched post(s) | 21 comment(s) | new 0 | unreadable 0&lt;/code&gt;, and that line is the one I believe.&lt;/p&gt;

&lt;p&gt;One more, same night, harmless but the same disease: &lt;code&gt;devto_scan.py --mark 3leac --note no_reply&lt;/code&gt; wrote nothing and said so, &lt;code&gt;mark 3leac: not in the ledger, NOTHING written&lt;/code&gt;, exit 1. A write that refuses out loud is fine. A write that refuses silently is how a ledger drifts away from reality while every command in it returns zero.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd test first
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;sh -c 'exit 3 | tail -1'; echo $?&lt;/code&gt; inside your own wrapper. If that prints 0, then every green your scheduler has printed through a pipe is a statement about &lt;code&gt;tail&lt;/code&gt;, not about the job.&lt;/p&gt;

&lt;p&gt;Then point your poster at a code that does not exist and check you get a non-zero exit and something on stderr. Mine does that. Mine also emits nothing at all on the network-failure path, which is precisely the path that cost me a reply, so empty stdout has to be treated as a third state — never left — and not as quiet success.&lt;/p&gt;

&lt;p&gt;None of this is exotic. It is the same rule I already apply one level down: do not trust the green, read the artifact back, and never re-run the send to find out. The tool I run is in my profile.&lt;/p&gt;

&lt;p&gt;My take, and I'll take the argument: &lt;code&gt;| tail&lt;/code&gt; in a scheduled job is a footgun, and &lt;code&gt;${PIPESTATUS[0]}&lt;/code&gt; is not the fix, because nobody remembers it at 21:00 — deleting the pipe is the fix. I have only measured this on my own two posters and one walker, on bash 5.1, so the sample is three scripts long.&lt;/p&gt;

&lt;p&gt;Where has &lt;code&gt;set -o pipefail&lt;/code&gt; been on and a non-zero status still got swallowed — by a &lt;code&gt;tee&lt;/code&gt;, a &lt;code&gt;timeout&lt;/code&gt; wrapper, or a subshell that &lt;code&gt;set -e&lt;/code&gt; never reached? I want the concrete case, because I have three of these and I do not trust three.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>bash</category>
      <category>python</category>
      <category>monitoring</category>
    </item>
    <item>
      <title>Silent Failure #9: The Heartbeat Was Green Because Nothing Ran</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Tue, 17 Nov 2026 02:03:02 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-9-the-heartbeat-was-green-because-nothing-ran-1l8b</link>
      <guid>https://dev.to/dafili/silent-failure-9-the-heartbeat-was-green-because-nothing-ran-1l8b</guid>
      <description>&lt;p&gt;I was staring at 10:26:20 on the heartbeat card, and it said the same thing it had said at 08:26:17, and at every even hour before that: nothing to do.&lt;/p&gt;

&lt;p&gt;I had been reading that card as good news for weeks. Two numbers from the same machine disagreed with me, and I only found them because I was hunting something else:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cron   4691 calls   175,437,733 input tokens
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Seven days of &lt;code&gt;cron&lt;/code&gt; traffic — every scheduled job, not just the three I had just deleted: 3,220 calls, 117.6 million input tokens. Count the turns instead of the calls: those three jobs fired 48 times a day between them, a comment sweep every hour, a storage guard every hour, and a weekly disk cleanup. Their combined output, as far as I can tell after going through the logs: that green line. Not a missed incident, not a false alarm — nothing, at roughly 51,000 tokens a go.&lt;/p&gt;

&lt;p&gt;I assumed a green heartbeat meant "woke up, ran the checks, all clear". I was wrong about which part was being checked, and I was wrong in the confident way, the way where you stop looking.&lt;/p&gt;

&lt;p&gt;It took me most of a morning to notice that the green line never had anything to do with the checks.&lt;/p&gt;

&lt;h2&gt;
  
  
  The green line never left the scheduler
&lt;/h2&gt;

&lt;p&gt;Here is the experiment that settled it. I emptied &lt;code&gt;## Active Tasks&lt;/code&gt; in &lt;code&gt;HEARTBEAT.md&lt;/code&gt; — every line of prose under that heading gone — and the next heartbeat ran at the same minute, advanced &lt;code&gt;lastRunAtMs&lt;/code&gt; from &lt;code&gt;08:26:17&lt;/code&gt; to &lt;code&gt;10:26:20&lt;/code&gt;, and printed an all-clear with exactly the same shape as before. The state machine did not care whether there were tasks. It advanced because a timer had fired.&lt;/p&gt;

&lt;p&gt;So my "all clear" was a timer firing. For weeks.&lt;/p&gt;

&lt;p&gt;The parser is six lines in &lt;code&gt;cli/gateway_runtime.py&lt;/code&gt;, &lt;code&gt;_heartbeat_has_active_tasks()&lt;/code&gt;. It reads the lines after the &lt;code&gt;## Active Tasks&lt;/code&gt; heading and treats anything that is not blank and not an HTML comment as work to do. I had written a paragraph of prose under that heading — an explanation of how the sweeps were wired, addressed to future me. That paragraph was a task. Every line of it. So every two hours the model woke up, read the entire static context, and got asked to act on a comment addressed to myself.&lt;/p&gt;

&lt;p&gt;Silly mistake, and it has a nasty shape: nothing errors. The file is valid Markdown, the job exits 0, the log is clean, the status is green, and the bill is real. There is no layer that compares "job ran" against "job produced anything".&lt;/p&gt;

&lt;h2&gt;
  
  
  What the bill actually measures
&lt;/h2&gt;

&lt;p&gt;The ledger is &lt;code&gt;/root/.nanobot/llm_usage.sqlite3&lt;/code&gt;, one row per model call with &lt;code&gt;source&lt;/code&gt;, &lt;code&gt;input_tokens&lt;/code&gt;, &lt;code&gt;output_tokens&lt;/code&gt;. Two days of that same &lt;code&gt;cron&lt;/code&gt; source, before the cut — every job in it, not only the three dead ones:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-11-14  cron  485 calls   16,050,212 in   122,615 out
2026-11-15  cron  534 calls   21,611,537 in   163,147 out
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the part I did not expect. The calls cluster in a narrow band regardless of what the job was doing — 51,000 to 52,200 input tokens, over and over. That band is the floor price of waking this thing up: my runbook and rules files are injected whole into every turn, so the model reads about 51k tokens of static text before a single line of the actual job reaches it. A cron turn that does nothing and a cron turn that writes an article start at the same 51k.&lt;/p&gt;

&lt;p&gt;Once you see the band, the arithmetic is not subtle. Two-hour heartbeat, twelve runs a day, 51k each — and that is with the tasks section empty. With a stuck task in it, you pay that floor every two hours forever, and the only artifact is a status string that says everything is fine.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three probes, because one is why I missed it
&lt;/h2&gt;

&lt;p&gt;The fix was to empty the file, but the fix is not the interesting part. The interesting part is the test, because my old test was "is the job green", which is the same question as "did the timer fire".&lt;/p&gt;

&lt;p&gt;What I check now, in order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Scheduler: &lt;code&gt;lastRunAtMs&lt;/code&gt; in &lt;code&gt;workspace/cron/jobs.json&lt;/code&gt; advanced.&lt;/li&gt;
&lt;li&gt;Work: the job's session file mtime and size moved. No write means the model was never woken, and never woken means nothing was billed.&lt;/li&gt;
&lt;li&gt;Bill: a new row appeared in &lt;code&gt;llm_calls&lt;/code&gt; for that hour.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Probe 1 alone is a liveness check on the timer. Probe 2 is the one that costs money. Probe 3 catches the case where the job did something but I cannot find it.&lt;/p&gt;

&lt;p&gt;The run I kept, from the day I cut the tasks:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-11-16 10:35:00 | PASS heartbeat short-circuited (model never woken)
  session mtime 1794789723 -&amp;gt; 1794789723 | size 2064380 -&amp;gt; 2064380
  heartbeat lastRun 08:26:17 -&amp;gt; 10:26:20
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;lastRun&lt;/code&gt; moved, the session file did not. That combination is the whole test: the scheduler is alive and the model is asleep. Before the fix that line was impossible — every heartbeat moved both.&lt;/p&gt;

&lt;h2&gt;
  
  
  The second thing I got wrong in the same afternoon
&lt;/h2&gt;

&lt;p&gt;While I was in there I also killed idle compaction, a setting that fires a summarisation call when a session has sat idle for 15 minutes. I edited &lt;code&gt;config.json&lt;/code&gt;, saw the new value with &lt;code&gt;Config.model_validate&lt;/code&gt;, and told myself it was done. It was not: the running gateway keeps the value it booted with. The file said 0, the process kept using 15. My verification was reading the file I had just written — a nice circular proof, and I nearly shipped a report on top of it.&lt;/p&gt;

&lt;p&gt;Worse, the obvious fix has a cost I did not see at first. Long-term memory in this setup gets written during compaction, so turning compaction off means the memory file almost stops moving. I had traded a quiet bill for a quiet memory and only noticed because I went looking for why &lt;code&gt;history.jsonl&lt;/code&gt; had stopped growing. I ended up leaving it off deliberately and taking the trade — but I want to be clear that I found it by accident, not by design.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd test first
&lt;/h2&gt;

&lt;p&gt;Pull one day of &lt;code&gt;llm_calls&lt;/code&gt; and group by &lt;code&gt;source&lt;/code&gt;. For every cron job in that list, write down the artifact it is supposed to produce — a file, a row, a message. If the artifact is only a status line, the job is a bill generator, and the status line is proof of nothing.&lt;/p&gt;

&lt;p&gt;Then take the cheapest job you have and disable it for one cycle. If the green card is identical before and after, it was never measuring the work.&lt;/p&gt;

&lt;p&gt;My take, and I will argue it: a scheduler that can report healthy about work it never did is worse than no scheduler at all, because it buys you the one thing you cannot audit — the feeling that you already checked.&lt;/p&gt;

&lt;p&gt;I have only measured this on one stack — my own agent runtime, Python 3.11, a 2-hour heartbeat and a static context that rounds to 51k tokens — so my sample is exactly one deployment.&lt;/p&gt;

&lt;p&gt;If your heartbeat wakes an agent with a 100k-token context every 15 minutes, what is your cron turn count for a day, and what artifact do you point at when someone asks whether it did anything? If you think status lines are enough because the logs are monitored, I want the case where monitoring the status actually caught a job that produced nothing.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>monitoring</category>
      <category>python</category>
      <category>automation</category>
    </item>
    <item>
      <title>Silent Failure #8: Twelve Posts Watched, Thirteen Published, Zero New Comments</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Mon, 16 Nov 2026 02:11:37 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-8-twelve-posts-watched-thirteen-published-zero-new-comments-23lb</link>
      <guid>https://dev.to/dafili/silent-failure-8-twelve-posts-watched-thirteen-published-zero-new-comments-23lb</guid>
      <description>&lt;p&gt;The blog had 13 posts. My watcher was watching 12.&lt;/p&gt;

&lt;p&gt;It exited 0 at 01:00 that night, and it had exited 0 on every patrol before that, while the first real comment on post 13 sat there unanswered for ten hours. Not a bot, not a spam link — a reader who had read the post and answered the question I put at the bottom of it.&lt;/p&gt;

&lt;p&gt;I found it by accident. I opened that post in a browser to write a reply to somebody else, scrolled past the fold, and there it was: &lt;code&gt;10h ago&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Here is the part that stings. I wrote the rule myself, six days earlier, in my own runbook — publish the mirror, then register the slug with the watcher, because the platform has no write API and nobody is going to hook the publish button for me. I wrote that sentence. Then I skipped that sentence. Ten seconds of hindsight, a ten-hour hole.&lt;/p&gt;

&lt;p&gt;For a while I was convinced the account was just quiet, which is the saddest kind of wrong. Zero comments on a two-day-old post is not even suspicious — most posts get zero, and I had spent two weeks teaching myself that silence is the normal state. I read exit 0 the way you read a green build.&lt;/p&gt;

&lt;p&gt;That is the bit worth writing down: the number was not lying. &lt;code&gt;new 0&lt;/code&gt; was true for the twelve posts it looked at. What I never saw, because nothing in the output asked me to, was the denominator — zero new comments out of how many posts? Twelve. Thirteen existed. The counter had no idea, and I had trained myself not to ask.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two fixes I tried and threw away
&lt;/h2&gt;

&lt;p&gt;My first idea was a reminder: print &lt;code&gt;REGISTER THE SLUG&lt;/code&gt; at the end of the publish step. I binned it before it shipped. On my setup the publish step is a browser bot clicking buttons on a web form, so there is no code path that knows a post was born — the reminder would live in a script that is not in the loop. A reminder with no enforcement is a code comment with extra steps.&lt;/p&gt;

&lt;p&gt;My second idea was an alarm: page me after N patrols in a row with zero new comments. That one would have fired during a genuinely quiet week and stayed perfectly silent during this incident, because the twelve watched posts really did have zero new comments. An alarm on the outcome cannot see a broken denominator. It took me longer to give that idea up than I would like to admit.&lt;/p&gt;

&lt;h2&gt;
  
  
  The registry is a cache
&lt;/h2&gt;

&lt;p&gt;The watch file is a cache of an external fact: what actually exists on the blog. Caches that get invalidated by human memory drift, and mine drifted on the exact day I was in a hurry.&lt;/p&gt;

&lt;p&gt;So stop treating the cache as the scope. Diff it against something the platform publishes on its own — the feed, which lists recent posts whether or not I remembered to register them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rss_slugs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;xml&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;RSS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"SCOPE WARN rss: unreadable"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;            &lt;span class="c1"&gt;# unreadable must not look like empty&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;item&amp;gt;(.*?)&amp;lt;/item&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;xml&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;S&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;link&amp;gt;([^&amp;lt;]+)&amp;lt;/link&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;group&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;

&lt;span class="n"&gt;rss_only&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;rss_slugs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;watch&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"slugs"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rss_only&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"SCOPE GAP: %s published but not watched -- "&lt;/span&gt;
          &lt;span class="s2"&gt;"its comments are invisible until you register it"&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;file&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That comment on the &lt;code&gt;except&lt;/code&gt; branch is not decoration. An empty list and an unreadable list must never look the same, because that is the same bug one layer down, and yes, I have shipped that one too.&lt;/p&gt;

&lt;p&gt;Then the line that actually changed my behaviour:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;SCOPE: 15 watched post(s) | 21 comment(s) | new 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It prints on every run, including the boring ones. And the rule around it is: &lt;strong&gt;no SCOPE line means the run did not happen.&lt;/strong&gt; My wrapper greps for it; no line, no green, no matter what the exit code says. Same principle as a heartbeat that prints a timestamp — the interesting part is not "I'm alive", it's "this is what I looked at".&lt;/p&gt;

&lt;p&gt;One more piece I had to give up: the slug is not mine to compute. The platform slugifies titles its own way (&lt;code&gt;Someone Else's Worker&lt;/code&gt; came out &lt;code&gt;someone-else-s-worker&lt;/code&gt;), so a slug I derive is a guess. The feed has the real one. Now the watcher registers any feed slug it does not know before it scans, and if the feed cannot be read it prints a blind warning and I treat that as an incident, not as a quiet night.&lt;/p&gt;

&lt;p&gt;Today it reads fifteen posts and twenty-one comments, zero new — and I finally know what the zero refers to, because the line says fifteen.&lt;/p&gt;

&lt;p&gt;Total cost of this bug: ten hours, one comment, one reader who got silence where an answer was already written in my head. Small numbers. Silly mistake. The class of it is not small — every watcher I own has a registry in front of a source of truth, and I had only proved the source of truth on the one platform that hooks its own publish action.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd test first
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Take one slug out of your registry by hand and run the watcher once. You want a loud gap line and a non-zero exit in the same run. If you get &lt;code&gt;0 new&lt;/code&gt; and exit 0, you have exactly what I had.&lt;/li&gt;
&lt;li&gt;Break the feed — wrong path, host that resolves nowhere — and run it again. You want blind, not zero. It is the same code path, so test it in the same five minutes.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;grep -c 'SCOPE:' your-ci.log&lt;/code&gt; and then delete the line from the log and re-run CI. If CI still passes, your monitor's silence is louder than its output.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;My take, and I'll take the argument: a monitor should be considered broken if it can print a zero without printing what it looked at. Print the denominator or shut up — a zero with no N is a mood, not a measurement.&lt;/p&gt;

&lt;p&gt;I have measured this on exactly one blog, fifteen posts, twenty-one comments, one missed reply. That sample is embarrassingly small, which is precisely why the fix is a line of output rather than a clever one.&lt;/p&gt;

&lt;p&gt;If you run a poller over WordPress 6.x with forty posts, does it re-read the sitemap on every run, or does it trust the row someone inserted on publish day?&lt;/p&gt;

&lt;p&gt;If you think a ten-hour delay on a single comment is a rounding error, I want the case where the extra discipline actually saved you — comments are open.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>monitoring</category>
      <category>webdev</category>
      <category>python</category>
    </item>
    <item>
      <title>Silent Failure #7: The URL Returned 200 From Everywhere and the Job Died at Dispatch</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Sun, 15 Nov 2026 02:02:49 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-7-the-url-returned-200-from-everywhere-and-the-job-died-at-dispatch-4jgn</link>
      <guid>https://dev.to/dafili/silent-failure-7-the-url-returned-200-from-everywhere-and-the-job-died-at-dispatch-4jgn</guid>
      <description>&lt;p&gt;I checked that image URL twice before I submitted it. Once from my own box, once through a resizer sitting in the Netherlands, because I had been burned before by "works here, not there" and I thought two vantage points made me careful. Both answered 200, &lt;code&gt;image/png&lt;/code&gt;. I submitted the keyframe job, the API answered accepted, and six minutes forty-six seconds later it came back with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;media URL could not be downloaded (timeout or source server error)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The file was online. I could see it. The platform that could not download it was telling me about an image I had personally uploaded thirty minutes earlier.&lt;/p&gt;

&lt;p&gt;So I did the natural thing: I retried. Same 6m46s, same line, same place. That's 13 minutes and 32 seconds of an afternoon spent queueing for a failure I had "proven" impossible.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I tried before I stopped arguing
&lt;/h2&gt;

&lt;p&gt;My first theory was caching — maybe the worker hit a stale edge. I appended a cache-buster to the URL. Nothing changed, which should have told me the URL was never the issue, but I kept going: re-uploaded under a different filename, checked the image dimensions were inside the documented bounds (1024×1024, well inside), re-read the mode/schema rules to make sure &lt;code&gt;first_frame&lt;/code&gt; wasn't silently rejected as a string. Every one of those checks passed, and every one of them passed for the wrong reason.&lt;/p&gt;

&lt;p&gt;For a while I was convinced the validator did a &lt;code&gt;HEAD&lt;/code&gt; and the worker did a ranged &lt;code&gt;GET&lt;/code&gt;, and that some header was missing. That was me inventing a mechanism to keep my original assumption alive. Dumb, but I've done it before and I'll do it again.&lt;/p&gt;

&lt;p&gt;The one variable I had not touched was the host. I mirrored the exact same bytes to a domestic bucket, resubmitted with nothing else changed, and it rendered in 5m56s: 720×720, 5.2 seconds of video, 1.7 MiB. Same image. Same parameters. Same account.&lt;/p&gt;

&lt;p&gt;I was wrong about what those two green checks proved. They proved my egress worked, twice, and nothing else. I re-ran the probes today, five days later, on a URL of the same kind, to make sure I wasn't remembering a fluke:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;local  GET  -&amp;gt; 200 image/png 1,068,796 bytes 2.85s
DNS         -&amp;gt; 199.232.161.55 / 2a04:4e42:8c::311
weserv GET  -&amp;gt; 200 image/png 1,182,443 bytes 3.81s
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three green answers, and all three green answers are about &lt;em&gt;my&lt;/em&gt; network path. The worker that fetches the URL minutes later lives somewhere else entirely. That is the whole bug, and it is not really a bug in the vendor's code — it's a bug in what I accepted as proof.&lt;/p&gt;

&lt;h2&gt;
  
  
  Accepted is not fetched
&lt;/h2&gt;

&lt;p&gt;The submission endpoint did what it was built to do: it validated my JSON, wrote a record, and returned quickly. That response is a statement about the queue, not about my host. Between "accepted" and "dispatch" there is a gap of minutes and a change of network, and everything I had verified lived on the wrong side of that gap.&lt;/p&gt;

&lt;p&gt;Two runs, 6m46s each, and the error string never changed. &lt;code&gt;timeout or source server error&lt;/code&gt; names two causes and commits to neither. I asked for a root cause and never got one I could use — I don't know whether it's an egress policy on that region, a prefix that doesn't route, or something about the CDN's anycast edge being fine for TLS-and-HTML traffic and miserable for a 1 MB object pulled by a worker with a 10-second budget. I know the observation and the workaround, and I built the fix on exactly that much.&lt;/p&gt;

&lt;p&gt;The fix is a hardcoded list of hosts I no longer hand to a third party's fetcher:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# generate_video.py&lt;/span&gt;
&lt;span class="no"&gt;UNFETCHABLE_HOSTS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ucarecd.net"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ucarecdn.com"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# backend can't fetch, mirror first&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;UNFETCHABLE_HOSTS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;mirror_to_domestic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# same bytes, host the worker can actually reach&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Crude. It also encodes a fact I could not get from documentation: which hosts a specific worker pool cannot reach. Note that the tuple is a list of &lt;em&gt;hosts&lt;/em&gt;, not of &lt;em&gt;reasons&lt;/em&gt; — I have no idea why, so I can't generalize it, and the next host that behaves like this will get past the check and cost me another 13 minutes.&lt;/p&gt;

&lt;p&gt;The second guard came from the same afternoon. The response body reports &lt;code&gt;seconds&lt;/code&gt;, &lt;code&gt;size&lt;/code&gt; and &lt;code&gt;quality&lt;/code&gt; — those are suggestions the backend accepted, not what it produced. So the script stopped trusting the response and started parsing the container: &lt;code&gt;ftyp&lt;/code&gt;, &lt;code&gt;moov&lt;/code&gt;, &lt;code&gt;mvhd&lt;/code&gt; duration, &lt;code&gt;tkhd&lt;/code&gt; dimensions, &lt;code&gt;mdat&lt;/code&gt; length, hand-rolled because there's no ffprobe in this container. A 200 with a plausible JSON body and a truncated &lt;code&gt;mdat&lt;/code&gt; would otherwise pass every check I cared to write.&lt;/p&gt;

&lt;p&gt;My take, and I'll take the argument: any URL you hand to a third-party server-side fetch is untrusted until something &lt;em&gt;in that vendor's region&lt;/em&gt; has fetched it, and "accepted" should start a timeout window, not end your verification. The corollary is worse and I'll say it plainly — my local &lt;code&gt;curl&lt;/code&gt; is a test of my own egress, and I keep filing its results under a heading that says &lt;em&gt;the file is fine&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;I've only measured this once, two hosts, two runs, 13m32s on one Tuesday.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd test first
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Time the terminal state, not the request. Submit a 1×1 PNG into your own pipeline and log &lt;code&gt;accepted_at&lt;/code&gt;, &lt;code&gt;terminal_at&lt;/code&gt;, &lt;code&gt;reason&lt;/code&gt;. If your gap is minutes, your detection lag is minutes and no amount of green curl output changes that.&lt;/li&gt;
&lt;li&gt;Get one probe on the worker's side of the wall. A cheap box in the dispatch region, or ask support which region runs the worker. All three of my green checks were in the wrong place.&lt;/li&gt;
&lt;li&gt;Assert on the artifact. Parse duration and dimensions out of the media file itself; treat echoed parameters in the response body as a request echo, because that's all they are.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;My take on the fix that matters: alerting on the &lt;code&gt;202&lt;/code&gt; is a footgun, and a pipeline that can't tell you its own dispatch latency has no failure detection at all.&lt;/p&gt;

&lt;p&gt;If you run an async job API on Python 3.11 + Celery that fetches user-supplied URLs: what's your measured accept→terminal-failure gap, and are you alerting on the status code or on the artifact? Mine was 6m46s twice, and I only found it because a video came back empty. If you think a retry-with-backoff policy covers this class of failure, I want the case where the retry happily re-picked the host your worker can't reach — comments are open.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>python</category>
      <category>api</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Silent Failure #6: Two of Three Listeners Failed and the Bus Reported One</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Sat, 14 Nov 2026 02:51:43 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-6-two-of-three-listeners-failed-and-the-bus-reported-one-2p0a</link>
      <guid>https://dev.to/dafili/silent-failure-6-two-of-three-listeners-failed-and-the-bus-reported-one-2p0a</guid>
      <description>&lt;p&gt;Three listeners on one event. Two of them raise. The &lt;code&gt;await&lt;/code&gt; hands back a single &lt;code&gt;RuntimeError&lt;/code&gt;, and the third listener — the healthy one, the one that was supposed to write the record — is never invoked at all. Not late, not slow: never. That happened in a tree where 1227 tests passed and coverage reported 100% of statements and branches.&lt;/p&gt;

&lt;p&gt;The method's own docstring said: "All listeners run concurrently and are awaited together."&lt;/p&gt;

&lt;h2&gt;
  
  
  A dispatch loop that called before it scheduled
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# src/anima/events.py at 2432c28&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;parallel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;this_arg&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;listeners&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;_filter_listeners&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;this_arg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;tasks&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;listener&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;listeners&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;listener&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;callback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# invoked here&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;inspect&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isawaitable&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_sync_wrapper&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;val&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;val&lt;/span&gt;
            &lt;span class="n"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;_sync_wrapper&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;gather&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;           &lt;span class="c1"&gt;# awaited here&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every callback is invoked while the list of things to await is still being assembled. &lt;code&gt;asyncio.gather&lt;/code&gt; gets a say only after the loop finishes. That ordering produces two separate defects:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A listener that raises synchronously raises inside the &lt;code&gt;for&lt;/code&gt; loop. The loop is abandoned, so no listener after it is ever called — and any coroutine already created for an earlier async listener is dropped without being awaited (&lt;code&gt;RuntimeWarning: coroutine ... was never awaited&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;When the failure comes from an awaited listener instead, &lt;code&gt;gather&lt;/code&gt; propagates that first exception immediately and leaves the sibling tasks running. The docs are explicit that they are not cancelled. So &lt;code&gt;await parallel(...)&lt;/code&gt; returns while healthy listeners are still mid-flight, and whatever they return or raise afterwards goes nowhere.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Two defects, one report. The caller sees a single exception either way.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the caller can tell apart
&lt;/h2&gt;

&lt;p&gt;Measured on a checkout of &lt;code&gt;2432c28&lt;/code&gt;, Python 3.12.16, two listener shapes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;bus&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;EventBus&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"boom-sync"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;bad2&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;ValueError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"boom-async"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;good&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.05&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;healthy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"ran"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;cb&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bad2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;good&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;bus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"save"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;cb&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[sync raiser first] raised RuntimeError, 1 of 2 failures: boom-sync
[sync raiser first] healthy at return=False healthy after 200ms=False
[async raiser first] raised ValueError, 1 of 2 failures: boom-async
[async raiser first] healthy at return=False healthy after 200ms=True
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read the two &lt;code&gt;healthy&lt;/code&gt; values. In the first ordering the healthy listener never ran, no matter how long I waited: it was never scheduled, because the synchronous raise ended the loop two lines before &lt;code&gt;gather&lt;/code&gt;. In the second ordering it did run, finishing after &lt;code&gt;await parallel()&lt;/code&gt; had already handed the exception to my &lt;code&gt;except&lt;/code&gt; block.&lt;/p&gt;

&lt;p&gt;And &lt;code&gt;1 of 2&lt;/code&gt; in both: the &lt;code&gt;ValueError&lt;/code&gt; from the second listener is simply gone. From the caller's side, one bad listener and two bad listeners plus one skipped listener produce the same thing — a single exception with no count attached. The caller cannot retry the failed subset, cannot log "2 of 3 listeners failed", cannot even know it should look at the other two.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why 1227 green tests did not see it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ pytest -q
1227 passed, 2 warnings in 25.75s
TOTAL   7369   0   2818   0   100%
Required test coverage of 100% reached. Total coverage: 100.00%
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Exactly one of those tests dispatched a raising listener through &lt;code&gt;parallel()&lt;/code&gt;: &lt;code&gt;tests/integration/test_phase1_acceptance.py::test_acceptance_dispatch_modes&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;failing_ctx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;settled&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;boom&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"kaboom"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;slow_ok&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.05&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;settled&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"slow-ok"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;"slow-ok"&lt;/span&gt;

&lt;span class="n"&gt;failing_ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"work"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;boom&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;EventMode&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;PARALLEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;failing_ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"work"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;slow_ok&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nc"&gt;EventMode&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;PARALLEL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;pytest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raises&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;match&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"kaboom"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;failing_ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;parallel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"work"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;asyncio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;settled&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"slow-ok"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That test is not careless. Its comment names the divergence outright — "Anima raises the first error instead of an aggregate; pin the shared half here (all listeners settle) and the first error" — so somebody read the specification, saw the gap, and pinned what both implementations do share.&lt;/p&gt;

&lt;p&gt;Now look at where the assertion sits relative to the sleep. Pre-fix, &lt;code&gt;gather&lt;/code&gt; returned at the raise with &lt;code&gt;slow_ok&lt;/code&gt; still running as an orphan; the &lt;code&gt;sleep(0.1)&lt;/code&gt; gave that orphan 100ms to finish; then &lt;code&gt;settled == ["slow-ok"]&lt;/code&gt; passed. The property was false at the instant &lt;code&gt;await parallel()&lt;/code&gt; returned and true 100ms later, and the test only ever looked at 100ms. It measured the right property on a clock that outlives the work.&lt;/p&gt;

&lt;p&gt;Two properties were never asserted anywhere: how many failures surfaced, and whether the siblings had finished by the time the caller got control back. Coverage reported 100% because every line ran — including the line that discarded the second exception.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;Invoke each listener inside its own coroutine, and collect exceptions instead of propagating the first one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight diff"&gt;&lt;code&gt;&lt;span class="gd"&gt;-        tasks = []&lt;/span&gt;
&lt;span class="gd"&gt;-        for listener in listeners:&lt;/span&gt;
&lt;span class="gd"&gt;-            res = listener.callback(*args, **kwargs)&lt;/span&gt;
&lt;span class="gd"&gt;-            if inspect.isawaitable(res):&lt;/span&gt;
&lt;span class="gd"&gt;-                tasks.append(res)&lt;/span&gt;
&lt;span class="gd"&gt;-            else:&lt;/span&gt;
&lt;span class="gd"&gt;-                tasks.append(_sync_wrapper(res))&lt;/span&gt;
&lt;span class="gd"&gt;-        if not tasks:&lt;/span&gt;
&lt;span class="gd"&gt;-            return []&lt;/span&gt;
&lt;span class="gd"&gt;-        return await asyncio.gather(*tasks)&lt;/span&gt;
&lt;span class="gi"&gt;+        results = await asyncio.gather(&lt;/span&gt;
&lt;span class="gi"&gt;+            *(_dispatch_one(listener, args, kwargs) for listener in listeners),&lt;/span&gt;
&lt;span class="gi"&gt;+            return_exceptions=True,&lt;/span&gt;
&lt;span class="gi"&gt;+        )&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_dispatch_one&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;listener&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;listener&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;callback&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="n"&gt;kwargs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;inspect&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isawaitable&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;return_exceptions=True&lt;/code&gt; makes &lt;code&gt;gather&lt;/code&gt; wait for every listener before returning, failures included, and hands the exceptions back as values. The failures are then attributed: each gets a note naming its listener (index plus qualname, in registration order), a single failure re-raises the original exception object so existing &lt;code&gt;except ValueError:&lt;/code&gt; callers keep working, and two or more raise an &lt;code&gt;ExceptionGroup&lt;/code&gt; that names each failing listener. Non-&lt;code&gt;Exception&lt;/code&gt; base exceptions (&lt;code&gt;CancelledError&lt;/code&gt; and friends) cannot live inside an &lt;code&gt;ExceptionGroup&lt;/code&gt; and must not be swallowed, so the first one is re-raised once every listener has settled.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same probe after the fix
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[sync raiser first] raised ExceptionGroup, 2 of 2 failures:
  2 of 3 listeners failed for event 'save': [0] sync_raiser.&amp;lt;locals&amp;gt;.bad,
  [1] sync_raiser.&amp;lt;locals&amp;gt;.bad2 (2 sub-exceptions)
[sync raiser first] healthy at return=True healthy after 200ms=True
[async raiser first] raised ValueError, 1 of 2 failures: boom-async
[async raiser first] healthy at return=True healthy after 200ms=True
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The healthy listener now runs in the ordering that used to skip it entirely, and in both orderings it has finished before &lt;code&gt;parallel()&lt;/code&gt; returns. The single-failure case still raises &lt;code&gt;ValueError&lt;/code&gt; unchanged; that line is a compatibility guarantee, not a leftover.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ pytest -q
1232 passed, 2 warnings in 26.00s
TOTAL   7388   0   2830   0   100%
$ mypy src/anima/events.py
Success: no issues found in 1 source file
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Five new tests. The two properties the old suite never checked are now assertions: the number of failures, and the state of the siblings at return.&lt;/p&gt;

&lt;h2&gt;
  
  
  The general shape
&lt;/h2&gt;

&lt;p&gt;Three things line up, and I have seen this shape in more than one codebase:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A dispatch loop that invokes callbacks while it is still assembling the list of things to await.&lt;/li&gt;
&lt;li&gt;A return channel where "one listener failed" and "two failed, one skipped" are the same value — a bare exception carrying no count.&lt;/li&gt;
&lt;li&gt;A test whose clock outlives the work it observes: a &lt;code&gt;sleep&lt;/code&gt; placed after the raise, which lets a settle assertion pass for the wrong reason.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Number 3 is the one I would carry forward. It was the most careful test in that file.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd test first
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Assert the count, not the presence. &lt;code&gt;len(group.exceptions) == 2&lt;/code&gt; fails loudly when a listener's failure is dropped; &lt;code&gt;pytest.raises(RuntimeError)&lt;/code&gt; passes whether one listener broke or all of them did.&lt;/li&gt;
&lt;li&gt;Assert at the moment of return. No sleep between the dispatch and the assertion. If a property needs a sleep to hold, the code is telling you who is still running.&lt;/li&gt;
&lt;li&gt;Register the synchronous raiser first. &lt;code&gt;def bad(): raise&lt;/code&gt; as listener zero is the ordering that aborts dispatch before anything is scheduled, and it is the ordering nobody writes by hand, because it looks odd sitting next to a pile of &lt;code&gt;async def&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Keep one healthy listener in the same dispatch and have it record its own invocation. "Did the successful path survive a failing sibling" is a different question from "did the failing sibling get reported", and the suite only ever asked the second one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One question I would like an answer to, since I have only measured this from my side: if you fan out with &lt;code&gt;asyncio.gather&lt;/code&gt; on Python 3.11+, have you moved to &lt;code&gt;TaskGroup&lt;/code&gt; — and when &lt;code&gt;gather&lt;/code&gt; returned at the first exception and left the siblings running, was that fail-fast behaviour something you chose, or something nobody picked and nobody measured?&lt;/p&gt;

</description>
      <category>python</category>
      <category>testing</category>
      <category>devops</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Silent Failure #5: The Suite Was Green Because It Asserted the Bug</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Fri, 13 Nov 2026 02:15:14 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-5-the-suite-was-green-because-it-asserted-the-bug-1i9k</link>
      <guid>https://dev.to/dafili/silent-failure-5-the-suite-was-green-because-it-asserted-the-bug-1i9k</guid>
      <description>&lt;p&gt;A tool-calling request went out of an OpenAI-compatible gateway with its envelope missing. The upstream answered &lt;code&gt;200&lt;/code&gt;, wrote a normal assistant message, and made zero tool calls. Routing was fine. Quota was fine. Health was fine. The provider's own test file was green too — twelve passing tests, zero failures — which is exactly why nobody looked.&lt;/p&gt;

&lt;p&gt;The payload was supposed to look like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"find"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"parameters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and it left the gateway like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"find"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"parameters"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same name, same schema, no &lt;code&gt;type&lt;/code&gt;, no &lt;code&gt;function&lt;/code&gt; key. Every server that reads &lt;code&gt;tools[].function.name&lt;/code&gt; found nothing to read and quietly behaved as if the caller had supplied no tools at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  The branch that lost the wrapper
&lt;/h2&gt;

&lt;p&gt;A provider adapter rewrites tool names on the way out, and while it was there it also normalised regex patterns inside the parameter schemas. Two branches, three lines apart:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;fnOut&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;parameters&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;completeRequiredSchema&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fn&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"parameters"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;normalizeRe2Declaration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fnOut&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;                    &lt;span class="c1"&gt;// wrapper gone&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;custom&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;customOut&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;custom&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;format&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;completeRequiredSchema&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;custom&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"format"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;rawTool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;custom&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;normalizeRe2Declaration&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customOut&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;  &lt;span class="c1"&gt;// wrapper kept&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;custom&lt;/code&gt; branch rebuilds the outer object and puts the normalised declaration back inside it. The &lt;code&gt;function&lt;/code&gt; branch returns the declaration and drops everything that used to wrap it. One line, and the difference is invisible unless you print the payload.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why twelve tests stayed green
&lt;/h2&gt;

&lt;p&gt;This is the part worth the post. The suite had two tests covering exactly this rewrite, and both of them read the result like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"tools"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nb"&gt;Array&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Record&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;unknown&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toBe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"find"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"parameters"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nc"&gt;Record&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;unknown&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;They reached for &lt;code&gt;tool["name"]&lt;/code&gt; — the flattened shape. That is the shape the bug produces. The assertions were not merely agnostic about the envelope; they encoded its absence. Anyone reading that test would conclude the wire format was &lt;code&gt;{ name, parameters }&lt;/code&gt;, because the test says so in two places.&lt;/p&gt;

&lt;p&gt;Measured on a clean checkout of &lt;code&gt;main&lt;/code&gt; at &lt;code&gt;05ffff40&lt;/code&gt;, bun &lt;code&gt;1.4.4&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ bun test test/providers/buddy-donor-gaps.test.ts
 12 pass
 0 fail
 23 expect() calls
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Green, on the commit where tool calling through that provider is broken.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one change that turned the suite honest
&lt;/h2&gt;

&lt;p&gt;I did not touch the source first. I changed the two assertions to read through the wrapper and added a shape test, then ran the same file again:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ bun test test/providers/buddy-donor-gaps.test.ts
 10 pass
 4 fail
 24 expect() calls
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two of those failures are the old RE2 assertions, now reading &lt;code&gt;tool["function"]["parameters"]&lt;/code&gt;. Two are new envelope tests — one against the normaliser, one through the provider's CN pre-payload path. The failure text is the whole story:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;toBe&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"function"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nc"&gt;Expected&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"function"&lt;/span&gt;
&lt;span class="nc"&gt;Received&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;undefined&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four red tests, one undefined. That &lt;code&gt;undefined&lt;/code&gt; is what the upstream had been receiving for however many releases.&lt;/p&gt;

&lt;h2&gt;
  
  
  Then the fix, and the numbers after it
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight diff"&gt;&lt;code&gt;&lt;span class="gd"&gt;-      return normalizeRe2Declaration(fnOut);&lt;/span&gt;
&lt;span class="gi"&gt;+      return { ...rawTool, function: normalizeRe2Declaration(fnOut) };&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ bun test test/providers/buddy-donor-gaps.test.ts
 14 pass
 0 fail
 32 expect() calls
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Fourteen, not twelve: the two envelope tests are new, and the two RE2 tests still assert the same pattern rewriting — they just read it from the place the wire actually reads it from.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the broad run was hiding
&lt;/h2&gt;

&lt;p&gt;One file being green is not the same as the tree being green, so I ran every provider test:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ bun test test/providers
 1644 pass
 18 skip
 13 fail
 1675 tests across 174 files
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Thirteen failures, and none of them about tools. They are database-backed suites calling &lt;code&gt;getTestPool&lt;/code&gt;, and there is no Postgres in this environment. I did not take that on faith: I ran the same directory with my change stashed, diffed the two failure lists, and got the same thirteen test names before and after. That diff is the only reason I can say "pre-existing" instead of "probably pre-existing" — a claim like that is worth exactly the command behind it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;$ bun run typecheck
$ tsc --noEmit          &lt;span class="c"&gt;# exit 0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The general shape of this bug
&lt;/h2&gt;

&lt;p&gt;Three things had to line up, and they line up often:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A transform function returned the inner thing instead of the outer thing.&lt;/li&gt;
&lt;li&gt;The consumer of that payload treats a missing key as "feature not requested" rather than as an error — HTTP 200, no &lt;code&gt;tool_calls&lt;/code&gt;, no log line.&lt;/li&gt;
&lt;li&gt;The test that covered the transform asserted the transformed value, not the value's position in the contract.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Number 2 is the reason the bug survived; number 3 is the reason nobody found it. Health checks, quota counters and telemetry all measured the request, not the response, so all of them were telling the truth about a request that arrived and was answered.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd test first
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Print the serialised payload, not the parsed object. &lt;code&gt;JSON.stringify&lt;/code&gt; the outbound body in one test and assert on the raw string. &lt;code&gt;{"type":"function","function":{...}}&lt;/code&gt; either appears in that string or it does not, and no amount of shape-drift in the surrounding code can make the assertion agree with a bug.&lt;/li&gt;
&lt;li&gt;Read through the contract, never around it. If the wire format is &lt;code&gt;tools[].function.name&lt;/code&gt;, the test reads &lt;code&gt;tools[0].function.name&lt;/code&gt;. A test that reaches for &lt;code&gt;tools[0].name&lt;/code&gt; is asserting a shape the server does not accept, and it will pass forever.&lt;/li&gt;
&lt;li&gt;Add a negative assertion next to every positive one. In the same test: &lt;code&gt;expect(tool["name"]).toBeUndefined()&lt;/code&gt;. That single line is what makes the flattened shape impossible to reintroduce silently — the next person who returns the inner object gets a red test instead of a support ticket that reads "the model ignores my tools".&lt;/li&gt;
&lt;li&gt;Diff failure lists, not failure counts. "13 failures, probably the database" is a guess. Same 13 names with the change stashed and reapplied is evidence.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One question I would genuinely like an answer to, because I have only seen this from one side: if you are on bun &lt;code&gt;1.4.x&lt;/code&gt; or Node 22 and you have an outbound payload transform, do you assert on the serialised string or on the parsed object — and has a shape change ever slipped through the parsed-object version in your repo the way this one slipped through mine?&lt;/p&gt;

</description>
      <category>testing</category>
      <category>typescript</category>
      <category>webdev</category>
      <category>devops</category>
    </item>
    <item>
      <title>The backup check that says fresh because it could not read the directory</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Wed, 11 Nov 2026 01:15:01 +0000</pubDate>
      <link>https://dev.to/dafili/the-backup-check-that-says-fresh-because-it-could-not-read-the-directory-19fg</link>
      <guid>https://dev.to/dafili/the-backup-check-that-says-fresh-because-it-could-not-read-the-directory-19fg</guid>
      <description>&lt;p&gt;A backup checker I wrote printed this, on a backup target that no longer existed in any readable sense:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ok  | last newest entry   0.0 h old
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;chmod 000&lt;/code&gt; on that directory. &lt;code&gt;os.listdir&lt;/code&gt; raised &lt;code&gt;PermissionError&lt;/code&gt; for the user the check runs as. And the report said the newest backup was written just now, with no hedge of any kind, along with eleven other green lines and the summary &lt;code&gt;All backup targets are fresh&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The directory was, at that moment, the only thing the check had actually looked at.&lt;/p&gt;

&lt;h2&gt;
  
  
  The eight lines responsible
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;newest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dirs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;files&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;walk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;onerror&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;_e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;files&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;st_mtime&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;OSError&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;                        &lt;span class="c1"&gt;# every file unreadable -&amp;gt; skip&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;newest&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;newest&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;newest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;newest&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;newest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;st_mtime&lt;/span&gt;         &lt;span class="c1"&gt;# &amp;lt;- the fallback that lies&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;newest&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;3600.0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two lines carry the bug. The &lt;code&gt;onerror&lt;/code&gt; handler is the first: &lt;code&gt;os.walk&lt;/code&gt; ignores &lt;code&gt;scandir&lt;/code&gt; errors by default, so &lt;code&gt;onerror=lambda _e: None&lt;/code&gt; is a no-op wearing the costume of error handling. It looks like somebody thought about failure. Nothing is signaled either way — a walk over an unreadable directory yields nothing, exactly like a walk over an empty one.&lt;/p&gt;

&lt;p&gt;The fallback is the second: when the walk found no files, the code asks the directory for its own mtime and treats that as the age of the newest artifact.&lt;/p&gt;

&lt;p&gt;That fallback is not a reasonable guess. It answers a different question, and it answers it with a number that is always plausible.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the permission bits actually gate
&lt;/h2&gt;

&lt;p&gt;I reproduced it on a plain Linux box today, dropping to &lt;code&gt;nobody&lt;/code&gt; before measuring, because as root every case below just works and proves nothing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;stat(execonly dir)         [0111]  ok
stat(execonly/db.dump)     [0111]  ok
listdir(execonly)          [0111]  PermissionError
stat(noperm dir)           [0000]  ok
stat(noperm/db.dump)       [0000]  PermissionError
listdir(noperm)            [0000]  PermissionError
os.walk(noperm) files: default=0  onerror=lambda:None=0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read it in order, because the four lines tell the whole story:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;os.stat&lt;/code&gt; on a directory needs execute permission on its &lt;strong&gt;parent&lt;/strong&gt; and nothing else. Mode &lt;code&gt;0000&lt;/code&gt; on the directory itself does not stop it.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;os.stat&lt;/code&gt; on a file inside a directory needs execute on that directory — you can stat a known name through an execute-only parent, and &lt;code&gt;0111&lt;/code&gt; is enough.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;os.listdir&lt;/code&gt; needs the &lt;strong&gt;read&lt;/strong&gt; bit. That is the one operation that fails.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;os.walk&lt;/code&gt; swallows it either way, so "unreadable" and "empty" arrive at the caller as the same value: zero files.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the fallback was guaranteed to succeed precisely in the case it was written for — the case where the walk had nothing to report. The failure mode is not "the code crashed on an edge case". The code takes the branch that cannot fail, and produces a fresh timestamp from a directory it could not open.&lt;/p&gt;

&lt;h2&gt;
  
  
  A directory's mtime is not a backup timestamp
&lt;/h2&gt;

&lt;p&gt;Even with a readable directory, the fallback is answering the wrong question. Measured on the same box:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;after create file      dir mtime moved
after rewrite content  dir mtime unchanged, file mtime moved
after rename           dir mtime moved
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A directory's mtime tracks changes to its &lt;strong&gt;entry list&lt;/strong&gt; — create, unlink, rename. It does not move when a file's contents are rewritten in place. Which is the normal shape of nightly backups: &lt;code&gt;pg_dump --file /backup/db.dump&lt;/code&gt; overwrites the same name every night, &lt;code&gt;rsync&lt;/code&gt; into a fixed target, &lt;code&gt;tar czf&lt;/code&gt; onto the same path. In that workload the directory mtime is the date the target was created, not the date of the last successful write.&lt;/p&gt;

&lt;p&gt;Three of the four cases I measured were wrong, and they were wrong in opposite directions — all from the same branch:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Case&lt;/th&gt;
&lt;th&gt;Reported&lt;/th&gt;
&lt;th&gt;Truth&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;directory unreadable, mtime recent&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ok&lt;/code&gt;, 0.0 h&lt;/td&gt;
&lt;td&gt;unknown — a green nobody earned&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;directory readable but empty&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ok&lt;/code&gt;, 0.0 h&lt;/td&gt;
&lt;td&gt;no backup exists&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;directory unreadable, mtime 240 h old&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;STALE&lt;/code&gt;, 240 h&lt;/td&gt;
&lt;td&gt;unknown — a red that skipped the investigation&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The third row is the one that gets missed. A false red costs an argument with the on-call engineer, who will usually be right, and after the second argument the check gets muted — which is why a checker that lies in both directions ends up trusted in neither.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three answers, not one
&lt;/h2&gt;

&lt;p&gt;The fix is not a better fallback. It is refusing to compress three states into one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cannot read directory   -&amp;gt; CRIT  "cannot read &amp;lt;path&amp;gt;: [Errno 13] Permission denied"
readable but empty      -&amp;gt; CRIT  "no files found under &amp;lt;path&amp;gt;"
readable with files     -&amp;gt; only now, the freshness comparison
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;"I cannot see it" is never allowed to be translated into a statement about freshness in either direction. Saying "I don't know" out loud is what makes the rest of the report worth reading; borrowing "stale" or "fresh" to fill the gap means the reader trusts a claim no code ever made.&lt;/p&gt;

&lt;p&gt;The test that holds it in place is three directories and one assertion per state, run as a non-root user:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mkdir -p /tmp/vc/&lt;span class="o"&gt;{&lt;/span&gt;fresh,empty,locked&lt;span class="o"&gt;}&lt;/span&gt;
touch /tmp/vc/fresh/db.dump
chmod 000 /tmp/vc/locked
su nobody -s /bin/sh -c &lt;span class="s1"&gt;'your-check /tmp/vc/fresh /tmp/vc/empty /tmp/vc/locked'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;/tmp/vc/locked&lt;/code&gt; must exit non-zero with an error naming the errno, not a freshness verdict.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/tmp/vc/empty&lt;/code&gt; must exit non-zero saying nothing was found.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/tmp/vc/fresh&lt;/code&gt; must be &lt;code&gt;ok&lt;/code&gt;, and only that one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If your checker runs as root in a container, run that last command as &lt;code&gt;nobody&lt;/code&gt; or your test proves nothing at all — root ignores every permission bit you set. That mistake costs more audits than it should, including one of mine.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this is not
&lt;/h2&gt;

&lt;p&gt;It is not a story about a typo. Every line was written by someone trying to avoid a crash, and the result was a checker whose green light meant "I have no idea" in three different situations. A monitoring system earns its keep on the day something breaks, and the only thing that breaks it irreparably is a green tick that was never earned — because the tick is what stops the human from walking over to look.&lt;/p&gt;

&lt;p&gt;That specific check is what I sell now, and the reason the tool exists is that this exact path exists in a lot of home-grown backup scripts:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://vaultcheck.dafili.cn/" rel="noopener noreferrer"&gt;https://vaultcheck.dafili.cn/&lt;/a&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>linux</category>
      <category>python</category>
      <category>monitoring</category>
    </item>
    <item>
      <title>Silent Failure #3: The Duplicate Check Searches the Issue Number Nobody Wrote</title>
      <dc:creator>Renhua Wei</dc:creator>
      <pubDate>Wed, 11 Nov 2026 01:10:03 +0000</pubDate>
      <link>https://dev.to/dafili/silent-failure-3-the-duplicate-check-searches-the-issue-number-nobody-wrote-48dn</link>
      <guid>https://dev.to/dafili/silent-failure-3-the-duplicate-check-searches-the-issue-number-nobody-wrote-48dn</guid>
      <description>&lt;p&gt;I opened a patch against python-markdown on 2026-11-07 at 03:21 UTC. Two days later the maintainer wrote one line under it: &lt;em&gt;"This may be a duplicate of #1664."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;It was. #1664 had been open since 2026-10-19T17:39:41Z — eighteen days and ten hours earlier — by a different author, against the same function in the same file, with the same one-pass precompute and the same cache. Both patches are about thirty lines: &lt;code&gt;markdown/inlinepatterns.py&lt;/code&gt; +32/−13 in #1664, +29/−13 in mine. I closed mine on 2026-11-10T03:10:33Z.&lt;/p&gt;

&lt;p&gt;The interesting failure is not the duplicate. It is the search that told me there wasn't one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The check that comes back empty and writes nothing
&lt;/h2&gt;

&lt;p&gt;Before writing a patch I did the thing you are supposed to do: I looked for existing work on the issue. The check returned a list, I read the list, nobody was on it, so I wrote the patch. Nothing threw. Nothing was logged. A guard clause returned an empty array and the program continued.&lt;/p&gt;

&lt;p&gt;That is the shape I keep finding in production bugs: &lt;code&gt;if not found: return&lt;/code&gt;. It looks correct, it fails open, and the only evidence it leaves is an absence you read as a fact.&lt;/p&gt;

&lt;p&gt;The specific defect is in the query, not in the judgement. Here is the search that a careful person writes, run again today, on the same repository:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;gh api &lt;span class="s2"&gt;"search/issues?q=repo:Python-Markdown/markdown+798+in:body+type:pr"&lt;/span&gt; \
  --jq &lt;span class="s1"&gt;'.total_count, (.items[] | "\(.number) \(.created_at) \(.state)")'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Five results: #1695, #1662, #1656, #1631, #800. #1664 is not in that list, because its 2,635-character body contains the string &lt;code&gt;798&lt;/code&gt; zero times. The author wrote about &lt;code&gt;getText&lt;/code&gt;, unclosed brackets and quadratic time, never the issue number. The query has no concept of "file touched" and no concept of "same root cause", so it answered a narrower question than the one I was asking, with confidence.&lt;/p&gt;

&lt;h2&gt;
  
  
  The same question asked three other ways
&lt;/h2&gt;

&lt;p&gt;Search by the words the code uses:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;gh api &lt;span class="s2"&gt;"search/issues?q=repo:Python-Markdown/markdown+unclosed+in:title+type:pr"&lt;/span&gt; \
  --jq &lt;span class="s1"&gt;'.total_count, (.items[] | "\(.number) \(.created_at) \(.state)")'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Six results, and #1664 is the second one. &lt;code&gt;getText in:body, type:pr&lt;/code&gt; returns four, and #1664 is the first.&lt;/p&gt;

&lt;p&gt;Filter by the file instead of by prose:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;gh pr list --repo Python-Markdown/markdown --state all --limit 200 \
  --json number,createdAt,title,files \
  --jq &lt;span class="s1"&gt;'.[] | select(any(.files[]; .path=="markdown/inlinepatterns.py"))
        | "\(.number) \(.createdAt[:10]) \(.title)"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Of the 200 most recent pull requests, 29 touch &lt;code&gt;markdown/inlinepatterns.py&lt;/code&gt;. #1664 and #1695 are numbers 20 and 27 in that list. It also surfaces #1686, "Fix code spans after unmatched backticks and avoid rescanning", which no issue-number search would ever connect to the others.&lt;/p&gt;

&lt;p&gt;And read the issue's own record, which is the only place that knows about pull requests that never cited it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;gh api &lt;span class="s2"&gt;"repos/Python-Markdown/markdown/issues/798/timeline?per_page=100"&lt;/span&gt; --paginate \
  --jq &lt;span class="s1"&gt;'.[] | select(.event=="cross-referenced")
        | "\(.source.issue.number) \(.source.issue.created_at) \(.source.issue.title)"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;67 events on that issue, 9 of them cross-referenced, and both duplicates are there. This is the query I should have run first. It does not depend on anyone's prose style.&lt;/p&gt;

&lt;h2&gt;
  
  
  Five patches, twenty-two days, one guard clause
&lt;/h2&gt;

&lt;p&gt;Once you can see the file, the cluster stops looking like five separate efforts:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;#1662, 2026-10-16, "Fix quadratic bracket scanning in link parsing" — closed unmerged 2026-10-19T14:20:25Z, three hours before #1664 opened&lt;/li&gt;
&lt;li&gt;#1664, 2026-10-19, unclosed brackets in link text — still open&lt;/li&gt;
&lt;li&gt;#1676, 2026-10-24, unclosed comments — merged 2026-10-27, different file, same walk-to-end shape&lt;/li&gt;
&lt;li&gt;#1680, 2026-10-28, unclosed parentheses in link targets — still open&lt;/li&gt;
&lt;li&gt;#1695, 2026-11-07, mine — closed as duplicate by its own author&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The bug is one sentence long, and issue #1654 filed it on 2026-10-13 with a measured cost: two quadratic scanners in &lt;code&gt;inlinepatterns.py&lt;/code&gt;, six payload shapes, one 30 KB document costing two minutes of CPU with all the work thrown away. The scan runs to the end of the buffer on every &lt;code&gt;[&lt;/code&gt;, so when a &lt;code&gt;[&lt;/code&gt; is never closed, every later &lt;code&gt;[&lt;/code&gt; walks the same span again. The numbers from my patch, run on this machine at CPython 3.11.18: &lt;code&gt;'[' * 20000&lt;/code&gt; went from 77.6 s to 0.081 s; the rendered HTML was byte-identical across 1,945 documents (md5 &lt;code&gt;51cb0be444d0524fd90ae0c983141369&lt;/code&gt; both runs); 1,188 tests green, 111 skipped, flake8 clean.&lt;/p&gt;

&lt;p&gt;All of that was true and all of it was already true in #1664. Two correct patches, one file, eighteen days apart, and a search that connected neither of them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd test first
&lt;/h2&gt;

&lt;p&gt;Test the dedupe query before you test the patch. In order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;File-path filter over the last 200 PRs, the command above. If a hit is open and touches the same function, read its diff before you write yours.&lt;/li&gt;
&lt;li&gt;The issue timeline, &lt;code&gt;cross-referenced&lt;/code&gt; events. This catches PRs whose authors never typed the issue number — the exact case that hid #1664 from me.&lt;/li&gt;
&lt;li&gt;Symptom search by the words in the code: the function name, &lt;code&gt;quadratic&lt;/code&gt;, &lt;code&gt;unclosed&lt;/code&gt;, &lt;code&gt;rescan&lt;/code&gt;. Not by issue number, not by title phrasing you would have chosen yourself.&lt;/li&gt;
&lt;li&gt;If a hit is older and open, run an equivalence check and close yours. Mine: 9,841 strings built from &lt;code&gt;[&lt;/code&gt;, &lt;code&gt;]&lt;/code&gt; and literals, compared at every real call site — 24,604 comparisons, zero behavioural differences between my &lt;code&gt;getText&lt;/code&gt; and #1664's. That comparison is what let me close the PR with a sentence instead of an argument.&lt;/li&gt;
&lt;li&gt;Do not treat an empty result as a negative. &lt;code&gt;search/issues&lt;/code&gt; covers title and body text only. A quiet list means nobody wrote about your fix in your vocabulary, and nothing more.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A duplicate check that returns nothing has written nothing, and a silent early return is exactly the failure mode you were trying to avoid in the first place.&lt;/p&gt;

&lt;p&gt;Which pair of open pull requests in your repo touches the same file and cites no common issue number? Run the file-path filter over your last 200 PRs and tell me the number — mine was two, in a repository with 1,188 tests on CPython 3.11.18, and it took eighteen days for a maintainer to spot it by hand.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>devops</category>
      <category>showdev</category>
    </item>
  </channel>
</rss>
