<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: ArcticFoxz</title>
    <description>The latest articles on DEV Community by ArcticFoxz (@arcticfoxz_255bbe944bc015).</description>
    <link>https://dev.to/arcticfoxz_255bbe944bc015</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4113356%2Fc98011eb-73ef-4688-a2fd-effd9a8f3ca0.png</url>
      <title>DEV Community: ArcticFoxz</title>
      <link>https://dev.to/arcticfoxz_255bbe944bc015</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/arcticfoxz_255bbe944bc015"/>
    <language>en</language>
    <item>
      <title>Three checks that were green for the wrong reason</title>
      <dc:creator>ArcticFoxz</dc:creator>
      <pubDate>Wed, 16 Sep 2026 04:43:56 +0000</pubDate>
      <link>https://dev.to/arcticfoxz_255bbe944bc015/three-checks-that-were-green-for-the-wrong-reason-19pk</link>
      <guid>https://dev.to/arcticfoxz_255bbe944bc015/three-checks-that-were-green-for-the-wrong-reason-19pk</guid>
      <description>&lt;p&gt;I shipped a release yesterday. Four CI rounds, most of a day, and the part worth writing down isn't the release — it's that three checks in my own test suite had been passing for reasons that had nothing to do with what they claimed to test, and I only found out because a platform I don't own ran them.&lt;/p&gt;

&lt;p&gt;They're three separate mechanisms. That's what makes them worth reading about: I didn't make one mistake three times, I made three different mistakes that all produce the same symptom, which is a green check that means nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  One: the trigger that never fired
&lt;/h2&gt;

&lt;p&gt;The feature is small. A rule file in the workspace can declare &lt;code&gt;Applies to: lib/redact.py&lt;/code&gt;, and when the files it names are the ones being edited, that rule gets more room in the context block than one that declares nothing. The check compares the two and asserts the scoped one wins.&lt;/p&gt;

&lt;p&gt;It passed on macOS. It failed on Windows. I spent a while assuming that meant something about Windows.&lt;/p&gt;

&lt;p&gt;It didn't. The check builds a temp repository and commits into it — five commits. The ranking it depends on returns nothing below fifty commits, so the trigger under test had never fired, on any platform, since the check was written. What it was actually comparing was the raw length of two strings, and the scoped rule's body is longer than the unscoped one for a reason you can probably guess: it carries the &lt;code&gt;Applies to:&lt;/code&gt; line. Forty-one characters. That was the entire margin it had been passing by.&lt;/p&gt;

&lt;p&gt;Windows didn't find a Windows bug. Windows got unlucky with string lengths in the other direction and exposed a check that had never worked.&lt;/p&gt;

&lt;p&gt;Fixed by making the fixture commit past the constant it depends on, derived from the constant rather than hardcoded:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_rollup&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;MIN_COMMITS_TO_RANK&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With the trigger actually firing: scoped 541 characters, unscoped 306, elsewhere 300. Against 411 / 370 / 371 before. The feature works. Nothing had ever shown that, and it was named in the release notes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two: the check that was never the platform under test
&lt;/h2&gt;

&lt;p&gt;This one is almost funny. A detector warns when a repository contains a file named like a program the tool is about to run, because Windows searches the current directory before PATH. The check forces &lt;code&gt;sys.platform = "win32"&lt;/code&gt;, exercises the detector, restores the platform, and then asserts the detector stays quiet — "it stays silent on a platform that does not search the current directory."&lt;/p&gt;

&lt;p&gt;On a Mac that's true. On Windows the real platform &lt;em&gt;is&lt;/em&gt; the platform under test, the detector correctly fires, and the check fails.&lt;/p&gt;

&lt;p&gt;I'd written a check whose final assertion was only correct on machines that are not the thing it tests.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three: a floor finer than the clock
&lt;/h2&gt;

&lt;p&gt;CI reported this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;FAIL  late failure on 'identifiers' grows no worse than linearly:
      0.0 ms at 4 KiB, 31.2 ms at 16 KiB (x625.0)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four times the input, 625 times the CPU. That reads like a catastrophic regression in a redaction routine.&lt;/p&gt;

&lt;p&gt;The line doing the dividing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;ratio&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;big&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;small&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.05&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;process_time()&lt;/code&gt; on Windows advances on the scheduler tick, roughly every 15.6 ms. The 4 KiB case is faster than that, so it reads 0.0. The floor supplies 0.05. And 31.2 / 0.05 is 624.&lt;/p&gt;

&lt;p&gt;The tell was in the log the whole time and I didn't see it for a while: 31.2 is exactly 15.6 x 2. Both numbers were quantised to ticks. Also, the same input passed on Python 3.8 in the same run — a real quadratic blowup doesn't care which interpreter you use.&lt;/p&gt;

&lt;p&gt;The fix is autoranging, the way &lt;code&gt;timeit&lt;/code&gt; does it: repeat the small case until the total is genuinely measurable, then measure both sizes at that same repeat count and divide the totals.&lt;/p&gt;

&lt;p&gt;And then I got it wrong a second time, which is the part I actually want to write down.&lt;/p&gt;

&lt;h2&gt;
  
  
  The second mistake inside the fix
&lt;/h2&gt;

&lt;p&gt;My first version keyed the target off &lt;code&gt;time.get_clock_info("process_time").resolution&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That's a documented API returning a real number, and on Windows it is 1e-07. One hundred nanoseconds — the unit &lt;code&gt;GetProcessTimes&lt;/code&gt; returns its values in. It is not the rate at which those values change, which is the 15.6 ms tick. So the target would have been 0.002 ms, the autorange would have stopped immediately, and the fix would have been a no-op on the only platform it exists for.&lt;/p&gt;

&lt;p&gt;It would also have passed every local test, because this Mac's clock really does resolve to a microsecond and the repeat count stays at 1 here no matter what.&lt;/p&gt;

&lt;p&gt;Measure the tick instead of asking for it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;t0&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;process_time&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;t1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;process_time&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;t1&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t1&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;1000.0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Take the larger of that and the reported resolution. On macOS nothing changes. On Windows the target becomes ~312 ms and the autorange does its job.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I checked it this time
&lt;/h2&gt;

&lt;p&gt;The thing that had bitten me three times in one day was believing a green result from a mechanism that had never engaged. So before trusting the fix I forced the resolution to 15.6 ms in a scratch copy and watched what the autorange did:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;'identifiers': resolution 15.600 ms, repeated x64 to reach a 512.493 ms baseline
'digits':      resolution 15.600 ms, repeated x32 to reach a 319.907 ms baseline
'spaces':      resolution 15.600 ms, repeated x8  to reach a 312.779 ms baseline
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ten lines of Python, no VM, no second machine. The mechanism engaged, the baselines became real numbers, and the check still passed. &lt;em&gt;That&lt;/em&gt; is evidence. The earlier run where everything was green and &lt;code&gt;repeat&lt;/code&gt; was 1 everywhere was not.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd take from it
&lt;/h2&gt;

&lt;p&gt;Three mechanisms, one symptom:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a population that was never gathered (five commits against a threshold of fifty)&lt;/li&gt;
&lt;li&gt;an environment that was never entered (the check ran on the platform it wasn't about)&lt;/li&gt;
&lt;li&gt;a mechanism that never engaged (repeat stuck at 1 on a fast clock)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of them announces itself. All three look exactly like a passing test, and a passing test is the thing you least want to go and re-examine.&lt;/p&gt;

&lt;p&gt;The habit I'm trying to build from it is narrow enough to actually follow: &lt;strong&gt;before believing a check, make it fail on purpose.&lt;/strong&gt; Not "does it pass" — "have I watched this go red for the reason it exists?" It cost me ten minutes per check. It would have saved most of a day.&lt;/p&gt;

&lt;p&gt;And the part I keep turning over: none of these were bugs in the shipped code. Every one was in the apparatus I use to decide whether the shipped code is correct. The tools were fine. The instruments were lying.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>python</category>
      <category>debugging</category>
      <category>windows</category>
    </item>
    <item>
      <title>A redactor that only spoke English, and the test that would have caught it</title>
      <dc:creator>ArcticFoxz</dc:creator>
      <pubDate>Mon, 14 Sep 2026 09:16:29 +0000</pubDate>
      <link>https://dev.to/arcticfoxz_255bbe944bc015/a-redactor-that-only-spoke-english-and-the-test-that-would-have-caught-it-4e6e</link>
      <guid>https://dev.to/arcticfoxz_255bbe944bc015/a-redactor-that-only-spoke-english-and-the-test-that-would-have-caught-it-4e6e</guid>
      <description>&lt;p&gt;Two files are worth more than anything I could write here, so this post is mostly directions.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;tools/verify_release.py&lt;/code&gt;
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com/ArcticFox2029/chamnan &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;cd &lt;/span&gt;chamnan
python3 tools/verify_release.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It re-runs the whole suite on your machine and prints what happened on &lt;em&gt;yours&lt;/em&gt;. 5,081 checks, 5,081 passed, 0 failed here. Your total will land near that rather than on it — some checks need a developer setup a clone does not have, so they say they were skipped instead of running, and how many depends on your OS. The half that should be identical everywhere is the second one.&lt;/p&gt;

&lt;p&gt;The reason it exists: the tool refuses to report a result at all when the suite's totals line is missing. A run that dies half way prints no failure lines, so grepping for failures reads zero over a run that stopped early. That distinction cost me a morning once.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;code&gt;INDEX_CITED_IN_CODE.md&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Attached to the release. Every place in the shipped source where a defect that a research round found got fixed, linked to the commit that fixed it. 21 of them landed in this release; 722 in total. It means a claim in the notes can be followed to a diff instead of taken on trust.&lt;/p&gt;




&lt;p&gt;If you want the reason any of that matters, it is the thing this release leads with.&lt;/p&gt;

&lt;p&gt;A credential redactor had been catching &lt;code&gt;password&lt;/code&gt; and walking straight past &lt;code&gt;รหัสผ่าน&lt;/code&gt;, &lt;code&gt;密码&lt;/code&gt;, &lt;code&gt;비밀번호&lt;/code&gt;, &lt;code&gt;كلمة المرور&lt;/code&gt; and six more spellings of the same word. Three of thirty cases caught. It is thirty of thirty now.&lt;/p&gt;

&lt;p&gt;The vocabulary was already in the file. Somebody had added it for reading CSV header rows and never wired it to the patterns that read an assignment — same words, one path covered, the identical path beside it missed.&lt;/p&gt;

&lt;p&gt;What I got wrong in the first fix was grouping by &lt;em&gt;language&lt;/em&gt;. German, Italian, Dutch and Indonesian are not English and are plain ASCII; the thing that actually decides the rule is the &lt;strong&gt;writing system&lt;/strong&gt;. Thai, Chinese, Japanese and Korean put no spaces between words, so a keyword joined to the next word is ordinary writing there and has to be caught. Latin, Cyrillic, Arabic and Devanagari keep boundaries, so &lt;code&gt;passwordless&lt;/code&gt; has to be left alone. Two different rules, and the axis that separates them is not the one on the language dropdown.&lt;/p&gt;

&lt;p&gt;Then a second one, found by asking the same question a different way: a variable called &lt;code&gt;secret&lt;/code&gt;, assigned the word &lt;em&gt;secret&lt;/em&gt;, printed its value. So did &lt;code&gt;credential&lt;/code&gt;, &lt;code&gt;apikey&lt;/code&gt;, &lt;code&gt;passphrase&lt;/code&gt;, &lt;code&gt;auth&lt;/code&gt;, &lt;code&gt;cred&lt;/code&gt;, &lt;code&gt;keypass&lt;/code&gt;, &lt;code&gt;storepass&lt;/code&gt;, &lt;code&gt;passwd&lt;/code&gt; and &lt;code&gt;secretkey&lt;/code&gt;. Only &lt;code&gt;password&lt;/code&gt; was caught — an earlier exemption answered "that's a label" for every one of them and ran before the rule that would have caught it. The fix that closed &lt;code&gt;password&lt;/code&gt; had landed on one member of a set of twelve.&lt;/p&gt;

&lt;p&gt;The test that caught the first of those asserted one example. Its replacement derives the whole credential vocabulary from the module and asserts every word in it, so a thirteenth joins on the day it is added. That is the part I would steal if I were reading someone else's release notes: &lt;strong&gt;assert the population, not the member.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;One number in here you cannot reproduce from a clone, and I would rather say so than let you find out: the recall figure is measured against a synthetic corpus built to be hard to index, and it is not in the repository. The 5,081 is reproducible. The recall number is not.&lt;/p&gt;




&lt;p&gt;There is also a new command, &lt;code&gt;chamnan-schedule&lt;/code&gt; — name a time and your work resumes then, carrying a pointer to where the work is written down rather than a command line, because "finish what you were doing" is not something a command line can say. It is a schedule and never an auto-renew: nothing detects a limit, nothing decides on its own to resume.&lt;/p&gt;

&lt;p&gt;What are you using to catch this class of thing? I am especially interested in anyone who has found a way to test a language they do not read.&lt;/p&gt;

</description>
      <category>security</category>
      <category>python</category>
      <category>testing</category>
      <category>opensource</category>
    </item>
    <item>
      <title>A blended recall number hid an entire class of leak</title>
      <dc:creator>ArcticFoxz</dc:creator>
      <pubDate>Fri, 11 Sep 2026 06:00:13 +0000</pubDate>
      <link>https://dev.to/arcticfoxz_255bbe944bc015/a-blended-recall-number-hid-an-entire-class-of-leak-ni6</link>
      <guid>https://dev.to/arcticfoxz_255bbe944bc015/a-blended-recall-number-hid-an-entire-class-of-leak-ni6</guid>
      <description>&lt;p&gt;chamnan is a Claude Code plugin that keeps engineering context in the repository, so an agent&lt;br&gt;
stops rediscovering the same work every session. Part of that is a redactor: it scrubs what the&lt;br&gt;
tool sends out, before it reaches a model backend.&lt;/p&gt;

&lt;p&gt;It scored 98.3% recall on a labelled corpus. That number was true and it was hiding something.&lt;/p&gt;

&lt;h2&gt;
  
  
  The number was measured over a corpus missing the case it was about
&lt;/h2&gt;

&lt;p&gt;A reader — &lt;a href="https://dev.to/peterbuildssecure"&gt;Peter&lt;/a&gt; — made an argument on somebody else's&lt;br&gt;
article: &lt;em&gt;a pipeline can hit strong recall on straightforward payloads while systematically&lt;br&gt;
missing anything harder, and a blended recall number hides that completely.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;So I split the figure by the retrieval problem each case poses. A credential can be named by an&lt;br&gt;
assignment (&lt;code&gt;API_KEY=...&lt;/code&gt;), positioned under a column header, described in surrounding prose, or&lt;br&gt;
standing bare with nothing but its own characters to go on.&lt;/p&gt;

&lt;p&gt;The column-header class had &lt;strong&gt;zero cases in it.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That is the class that leaked. A password under a CSV header — &lt;code&gt;nombre,correo,contraseña&lt;/code&gt; —&lt;br&gt;
escaped in every language the redactor claimed to handle, English included. The fix shipped. The&lt;br&gt;
measurement that should have caught it had never sampled it once.&lt;/p&gt;

&lt;p&gt;An empty class does not show up in a breakdown as a zero. It does not show up at all, which is&lt;br&gt;
how it survived being looked at.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix moved the headline the wrong way
&lt;/h2&gt;

&lt;p&gt;The corpus now derives its column cases from the redactor's own list of claimed languages, so a&lt;br&gt;
language it claims is a language it is measured on. Sixteen languages, thirty-six cases.&lt;/p&gt;

&lt;p&gt;All thirty-six pass — so the blended figure went &lt;strong&gt;up&lt;/strong&gt;, 98.3% to 99.0%.&lt;/p&gt;

&lt;p&gt;That is the same defect from the other side. A headline improved by testing more of what already&lt;br&gt;
works flatters exactly as much as one that hides a weak class. So both numbers are published now:&lt;br&gt;
&lt;strong&gt;99.0% overall, 93.8% in the weakest class&lt;/strong&gt;, and a check fails if either goes missing from the&lt;br&gt;
README or SECURITY.md.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three more places a number could be confidently wrong
&lt;/h2&gt;

&lt;p&gt;Peter's other arguments turned into three more fixes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A rate without its denominator.&lt;/strong&gt; Every reported rate now carries the count it was computed
over — a rate over a shrinking, unwatched denominator produces a confident green with nothing
behind it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A third state for "cannot decide".&lt;/strong&gt; A file the parser refuses is not a file nobody
documented, and only one of those is fixable by writing a docstring. The coverage line says
which is which — and still counts them, because removing them raises the percentage, which
would mean a repository scoring better because the tool can read less of it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A list of accepted findings that could grow unnoticed.&lt;/strong&gt; When a self-scan finds something
new, the cheapest way to make the check pass is to add it to the accepted list, which silences
a real finding exactly as well as fixing one. That list has a ceiling now, and raising it has
to appear in a diff.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Checking it
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;check 4869 / 4869&lt;/code&gt;, and 1,466 of 1,466 index claims true, on Linux, macOS and Windows across two&lt;br&gt;
Python versions.&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;git clone https://github.com/ArcticFox2029/chamnan
cd chamnan &amp;amp;&amp;amp; python3 tools/verify_release.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;It runs the suite and the index-claim check on your machine and prints what happened. It refuses&lt;br&gt;
to report a result when the totals line is missing, because a run that dies mid-way prints no&lt;br&gt;
failure lines at all.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One honest limit:&lt;/strong&gt; the recall corpus is synthetic, built to be hard to index, and is not part&lt;br&gt;
of the repository — so you cannot reproduce those particular numbers from a clone. The suite and&lt;br&gt;
the index claims you can.&lt;/p&gt;

&lt;p&gt;MIT. Python standard library only. Nothing leaves the machine.&lt;/p&gt;

</description>
      <category>security</category>
      <category>python</category>
      <category>opensource</category>
      <category>ai</category>
    </item>
    <item>
      <title>The bug that only existed on the platform nobody was looking at</title>
      <dc:creator>ArcticFoxz</dc:creator>
      <pubDate>Wed, 09 Sep 2026 02:14:44 +0000</pubDate>
      <link>https://dev.to/arcticfoxz_255bbe944bc015/the-bug-that-only-existed-on-the-platform-nobody-was-looking-at-2bpl</link>
      <guid>https://dev.to/arcticfoxz_255bbe944bc015/the-bug-that-only-existed-on-the-platform-nobody-was-looking-at-2bpl</guid>
      <description>&lt;p&gt;I ship a Python tool with a regression suite of several thousand checks and CI on ubuntu, macOS and&lt;br&gt;
Windows at Python 3.8 and 3.13. Yesterday all five jobs were green. Today those same jobs turned up&lt;br&gt;
four defects that had already shipped.&lt;/p&gt;

&lt;p&gt;None of them errored. Each was a rule that was true on the machine it was written on.&lt;/p&gt;
&lt;h2&gt;
  
  
  1. A guard that compared a resolved path to an unresolved one
&lt;/h2&gt;

&lt;p&gt;The tool has a command that reads one file into the model's context. It refuses a symlink that a&lt;br&gt;
repository chose whose target sits outside that repository — a repo shipping &lt;code&gt;docs/notes.md -&amp;gt;&lt;br&gt;
~/.ssh/id_rsa&lt;/code&gt; should not get that file read aloud.&lt;/p&gt;

&lt;p&gt;The guard looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;root&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;find_root&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;                      &lt;span class="c1"&gt;# returns a RESOLVED path
&lt;/span&gt;&lt;span class="n"&gt;named_inside&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;any&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abspath&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="n"&gt;parents&lt;/span&gt;
                   &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;named_inside&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;inside&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;refuse&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;os.path.abspath&lt;/code&gt; does not resolve anything. It joins the name onto &lt;code&gt;os.getcwd()&lt;/code&gt;. On POSIX that is&lt;br&gt;
harmless, because &lt;code&gt;getcwd()&lt;/code&gt; returns a canonical path with no symlinks in it — so both sides always&lt;br&gt;
agreed and the guard always fired.&lt;/p&gt;

&lt;p&gt;Windows does not do that. &lt;code&gt;GetCurrentDirectory&lt;/code&gt; hands back whatever string the process was started&lt;br&gt;
with, short 8.3 components and all. A repository reached through &lt;code&gt;C:\Users\RUNNER~1\...&lt;/code&gt; produced&lt;br&gt;
&lt;code&gt;named_inside = False&lt;/code&gt;, and the file outside the repository was read exactly as it had been before&lt;br&gt;
the guard was written. The guard was not weaker on Windows. It was absent.&lt;/p&gt;

&lt;p&gt;The fix resolves the containing &lt;em&gt;directory&lt;/em&gt; — which has no final symlink in it, so it normalises the&lt;br&gt;
name without following the link being judged:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;named_dir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abspath&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;root_dir&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;named_inside&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;named_dir&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="n"&gt;root_dir&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="n"&gt;root_dir&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;named_dir&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;parents&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I could not reproduce this on macOS. &lt;code&gt;getcwd()&lt;/code&gt; canonicalises even if you &lt;code&gt;chdir&lt;/code&gt; through a symlink,&lt;br&gt;
so the asymmetry cannot be staged on the platform I develop on. That matters for the rest of this&lt;br&gt;
post.&lt;/p&gt;
&lt;h2&gt;
  
  
  2. A carriage return no one would ever see
&lt;/h2&gt;

&lt;p&gt;The tool installs an optional &lt;code&gt;pre-commit&lt;/code&gt; hook. It rebuilds an index, stages it, then reads back&lt;br&gt;
which files it wrote and refreshes each one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;mytool &lt;span class="nt"&gt;--written-files&lt;/span&gt; | &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; name&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;&lt;span class="nv"&gt;written&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;mytool &lt;span class="nt"&gt;--write&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | &lt;span class="nb"&gt;sed&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s1"&gt;'s/^wrote -&amp;gt; //p'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$written&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; git add &lt;span class="nt"&gt;--&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$written&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Under git's bundled shell on Windows, Python ends every line with CR LF. &lt;code&gt;read -r&lt;/code&gt; keeps the CR. So&lt;br&gt;
&lt;code&gt;name&lt;/code&gt; is &lt;code&gt;generic\r&lt;/code&gt;, and &lt;code&gt;--write "generic\r"&lt;/code&gt; is not a valid choice — argparse refuses it. The&lt;br&gt;
loop wrote nothing and staged nothing, on every commit, for as long as the loop had existed.&lt;/p&gt;

&lt;p&gt;The line &lt;em&gt;above&lt;/em&gt; it worked the whole time, because it parses no output. So the index looked healthy&lt;br&gt;
while every generated file stood still, and the README went on describing this hook as the thing&lt;br&gt;
that keeps them current.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;tr -d '\r'&lt;/code&gt; on both readings fixes it. Then I fixed it wrong: the hook body is a Python string, not&lt;br&gt;
a raw one, so writing &lt;code&gt;\r&lt;/code&gt; in the source put a real carriage return into the installed hook — the&lt;br&gt;
exact opposite of the intent, and it broke the hook on all five jobs instead of the one it was&lt;br&gt;
written for. The check I had added alongside read the source &lt;em&gt;text&lt;/em&gt;, saw two characters, and&lt;br&gt;
reported the escape as present.&lt;/p&gt;

&lt;p&gt;The check that caught it was an older one that reads the &lt;em&gt;installed file&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;That is the whole lesson of this section: assert on the artifact, not on the source that produces it.&lt;/p&gt;
&lt;h2&gt;
  
  
  3. The instruction that did nothing when pasted
&lt;/h2&gt;

&lt;p&gt;On a first run that cannot find its commands on your PATH, the tool prints the line to add:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"/path/to/bin:&lt;/span&gt;&lt;span class="nv"&gt;$PATH&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To every operating system. On Windows that is wrong three times over — &lt;code&gt;export&lt;/code&gt; is not a command,&lt;br&gt;
&lt;code&gt;$PATH&lt;/code&gt; is not the variable, &lt;code&gt;:&lt;/code&gt; is not the separator. The single instruction a new Windows user is&lt;br&gt;
given did nothing when they pasted it.&lt;/p&gt;

&lt;p&gt;And the test asserted &lt;code&gt;"export PATH=" in output&lt;/code&gt; — on Windows too. The platform where the advice was&lt;br&gt;
useless was the platform confirming it was correct.&lt;/p&gt;
&lt;h2&gt;
  
  
  4. Not a platform at all: an interpreter
&lt;/h2&gt;

&lt;p&gt;A symlink loop (&lt;code&gt;a -&amp;gt; b -&amp;gt; a&lt;/code&gt;) makes &lt;code&gt;Path.resolve()&lt;/code&gt; raise. The walk caught that and dropped the&lt;br&gt;
entry in silence — the one exit from that function that recorded nothing, while every other exit&lt;br&gt;
reports what it refused.&lt;/p&gt;

&lt;p&gt;Python 3.13 rewrote &lt;code&gt;Path.resolve()&lt;/code&gt; onto &lt;code&gt;os.path.realpath(strict=False)&lt;/code&gt;, which returns a path for&lt;br&gt;
a loop instead of raising. So the same tree was reported honestly on 3.13 and silently truncated on&lt;br&gt;
3.8 through 3.12, under a coverage bar reading 100% over a smaller set than the one it walked.&lt;/p&gt;

&lt;p&gt;The check covering it passed on the version I happen to run.&lt;/p&gt;
&lt;h2&gt;
  
  
  What actually found these
&lt;/h2&gt;

&lt;p&gt;Not cleverness. Two things:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A failing check that prints what it saw.&lt;/strong&gt; My suite's helper took a name and a boolean. A failing&lt;br&gt;
check said what was expected and nothing about what happened, which is fine when you can re-run it —&lt;br&gt;
and useless when it only fails on a platform you do not have. One round of CI was spent on four FAIL&lt;br&gt;
lines and a guess, and the guess was wrong. So:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;condition&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;saw&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;condition&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;PASSED&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;  FAIL  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;saw&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;                      &lt;span class="c1"&gt;# only on failure
&lt;/span&gt;            &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;saw&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;splitlines&lt;/span&gt;&lt;span class="p"&gt;()[:&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
                &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;        SAW  &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The very next run printed a commit stat showing the index refreshed and the generated file&lt;br&gt;
untouched. That is section 2, diagnosed in one round instead of three.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Assert the population, not the member.&lt;/strong&gt; Every one of these is the same shape: a rule applied to&lt;br&gt;
some members of a set and forgotten in the identical ones beside it. So the checks are derived from&lt;br&gt;
source rather than pinned to the line that was wrong. When I fixed the CR in the hook, the check I&lt;br&gt;
wrote was "every place this script reads output produced by a Python program strips CR", derived&lt;br&gt;
from the hook text — not "line 46 has &lt;code&gt;tr -d&lt;/code&gt;".&lt;/p&gt;

&lt;p&gt;That habit paid immediately somewhere else. GitHub's push protection refused the release over three&lt;br&gt;
test credentials written as literals. The file already had a helper that assembles fake credentials&lt;br&gt;
at runtime, forty lines above, whose docstring says &lt;em&gt;push protection blocked this repository's first&lt;br&gt;
push over exactly these lines&lt;/em&gt;. It had been used for some fixtures and not the ones beside them. I&lt;br&gt;
wrote the check over the whole file instead of the three, and it immediately found four more —&lt;br&gt;
GitLab, GitHub, Stripe, Anthropic — already in the published tree, which would refuse a fork's first&lt;br&gt;
push the same way.&lt;/p&gt;

&lt;h2&gt;
  
  
  The one that was not about platforms at all
&lt;/h2&gt;

&lt;p&gt;After the release went out I swept the published tree for something else: my own machine.&lt;/p&gt;

&lt;p&gt;Two defect comments used a real encoded path as their example — my account name and the directory&lt;br&gt;
layout of unrelated work, sitting in a public repository since a release three versions back. Not a&lt;br&gt;
secret in any scanner's sense, which is exactly why nothing stopped them. A published&lt;br&gt;
&lt;code&gt;CODE_OF_CONDUCT.md&lt;/code&gt; printed a contact address that a private security advisory link would have&lt;br&gt;
covered without one.&lt;/p&gt;

&lt;p&gt;Both are now checks over every tracked file, not fixes to the four lines that were wrong: no shipped&lt;br&gt;
file may quote a real home directory rather than an invented one, and no published document may&lt;br&gt;
print an address outside the reserved example domains a test is supposed to use. The first one&lt;br&gt;
matched its own sanity assertion on the first run, because that line contains a real-looking path;&lt;br&gt;
it builds the needle at runtime now.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four gates, and the thing that walked past all of them
&lt;/h2&gt;

&lt;p&gt;By the end of the day I had four checks standing between the repository and a leak: no real home&lt;br&gt;
directory in a tracked file, no address in a published document, no credential fixture written as a&lt;br&gt;
literal, and the project's own redactor — seventy-one credential shapes — swept over every tracked&lt;br&gt;
file against a committed baseline. The tree came back clean on all four.&lt;/p&gt;

&lt;p&gt;Then I wrote a fifth check that reads a list of terms instead of guessing a shape. Not a pattern: a&lt;br&gt;
plain list of the specific words that must never appear, kept in a file outside every repository,&lt;br&gt;
because a list of things you must not publish is the most concentrated form of the thing you must&lt;br&gt;
not publish.&lt;/p&gt;

&lt;p&gt;It failed on the first run. A test fixture in the shipped suite carried a real operational&lt;br&gt;
constraint from my own production work — an imperative sentence naming a real cluster, pasted in&lt;br&gt;
while debugging months earlier. Not a path. Not an address. Not a credential. Nothing was looking&lt;br&gt;
for it, because every check I had written looked for a &lt;em&gt;shape&lt;/em&gt;, and this was just a sentence.&lt;/p&gt;

&lt;p&gt;That is the argument for the list. Shape-matching catches the classes you have already been burned&lt;br&gt;
by. The next leak is, by definition, the one whose shape you did not think to describe — and the&lt;br&gt;
only thing that catches it is knowing, concretely, what your own secrets are called.&lt;/p&gt;

&lt;h2&gt;
  
  
  The uncomfortable part
&lt;/h2&gt;

&lt;p&gt;CI ran green on Windows for months while two of these were live. The jobs were not lying; they were&lt;br&gt;
running checks that asked the wrong question, written by someone who could only picture one platform&lt;br&gt;
while writing them. A cross-platform matrix tells you the suite passes everywhere. It does not tell&lt;br&gt;
you the suite is &lt;em&gt;asking&lt;/em&gt; the same thing everywhere.&lt;/p&gt;

&lt;p&gt;The tool is &lt;a href="https://github.com/ArcticFox2029/chamnan" rel="noopener noreferrer"&gt;chamnan&lt;/a&gt; — MIT, Python 3.8+, standard library&lt;br&gt;
only, runs entirely locally. &lt;code&gt;python3 tools/verify_release.py&lt;/code&gt; from a clean clone re-runs everything&lt;br&gt;
this post quotes and prints what happened on your machine.&lt;/p&gt;

</description>
      <category>testing</category>
      <category>python</category>
      <category>windows</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Measure a context plugin on your own repository before you install it</title>
      <dc:creator>ArcticFoxz</dc:creator>
      <pubDate>Mon, 07 Sep 2026 20:01:51 +0000</pubDate>
      <link>https://dev.to/arcticfoxz_255bbe944bc015/measure-a-context-plugin-on-your-own-repository-before-you-install-it-2hf6</link>
      <guid>https://dev.to/arcticfoxz_255bbe944bc015/measure-a-context-plugin-on-your-own-repository-before-you-install-it-2hf6</guid>
      <description>&lt;p&gt;Every plugin in this category tells you it reduces context cost. None of them let you check on your&lt;br&gt;
own code first, so I built the page that does, and it is the first thing chamnan 1.23.1 ships.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;a href="https://arcticfox2029.github.io/chamnan-measure/" rel="noopener noreferrer"&gt;arcticfox2029.github.io/chamnan-measure&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Paste a public GitHub repository URL. The page downloads the source into your browser tab, runs&lt;br&gt;
chamnan's actual modules over it through Pyodide — &lt;code&gt;mapper&lt;/code&gt;, &lt;code&gt;rollup&lt;/code&gt;, the redactor, not a&lt;br&gt;
re-implementation of any of them — and reports what would be injected into an agent's context each&lt;br&gt;
session, what the redactor would alter, and a fifty-turn simulation over that repository's own&lt;br&gt;
files.&lt;/p&gt;

&lt;p&gt;No server. No upload. No API key. The source is deleted the moment the numbers exist. It works in&lt;br&gt;
five languages.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the numbers say, and what they do not
&lt;/h2&gt;

&lt;p&gt;Thirteen repositories are measured on the page already:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;repository&lt;/th&gt;
&lt;th&gt;language&lt;/th&gt;
&lt;th&gt;source&lt;/th&gt;
&lt;th&gt;injected&lt;/th&gt;
&lt;th&gt;ratio&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;chalk/chalk&lt;/td&gt;
&lt;td&gt;JavaScript&lt;/td&gt;
&lt;td&gt;56 KB&lt;/td&gt;
&lt;td&gt;2,449 B&lt;/td&gt;
&lt;td&gt;24:1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;psf/requests&lt;/td&gt;
&lt;td&gt;Python&lt;/td&gt;
&lt;td&gt;392 KB&lt;/td&gt;
&lt;td&gt;6,345 B&lt;/td&gt;
&lt;td&gt;63:1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;rails/rails&lt;/td&gt;
&lt;td&gt;Ruby&lt;/td&gt;
&lt;td&gt;1,550 KB&lt;/td&gt;
&lt;td&gt;6,537 B&lt;/td&gt;
&lt;td&gt;243:1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;django/django&lt;/td&gt;
&lt;td&gt;Python&lt;/td&gt;
&lt;td&gt;2,209 KB&lt;/td&gt;
&lt;td&gt;6,245 B&lt;/td&gt;
&lt;td&gt;362:1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;torvalds/linux&lt;/td&gt;
&lt;td&gt;C&lt;/td&gt;
&lt;td&gt;3,028 KB&lt;/td&gt;
&lt;td&gt;6,603 B&lt;/td&gt;
&lt;td&gt;470:1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;vuejs/core&lt;/td&gt;
&lt;td&gt;TypeScript&lt;/td&gt;
&lt;td&gt;3,782 KB&lt;/td&gt;
&lt;td&gt;6,250 B&lt;/td&gt;
&lt;td&gt;620:1&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The interesting column is not the ratio. It is that the source grows nearly seventy-fold down that&lt;br&gt;
table and the injected block stays between 6.2 and 6.7 KB. &lt;strong&gt;That is a bound, not compression&lt;/strong&gt; —&lt;br&gt;
the index is rolled up to a token budget, so a bigger repository buys you a coarser map rather than&lt;br&gt;
a bigger bill.&lt;/p&gt;

&lt;p&gt;The ratio is a consequence of the bound, which is why quoting it alone would be the flattering&lt;br&gt;
version. A 25.4x figure on the published corpus is the honest headline for compression; the same&lt;br&gt;
measurement on a corpus padded with binary attachments produces 223x, and this project's own README&lt;br&gt;
records the time that number was corrected for choosing the flattering corpus.&lt;/p&gt;

&lt;h2&gt;
  
  
  The release itself is one defect, found in fourteen places
&lt;/h2&gt;

&lt;p&gt;Not fourteen unrelated bugs. One shape, repeated: &lt;strong&gt;a rule applied to some members of a set and&lt;br&gt;
forgotten in the identical ones beside it.&lt;/strong&gt; They were written months apart, found by separate&lt;br&gt;
research rounds, and naming them together is the only way the pattern is visible.&lt;/p&gt;

&lt;p&gt;A few of them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Path.glob&lt;/code&gt; raises &lt;code&gt;NotImplementedError&lt;/code&gt; — not &lt;code&gt;ValueError&lt;/code&gt;, not &lt;code&gt;OSError&lt;/code&gt; — for a pattern
beginning with &lt;code&gt;/&lt;/code&gt;. One committed rule file reading &lt;code&gt;**Check:** absent `X` in every `/etc/*`&lt;/code&gt;
ended the injected context block where it stood: the last session's handoff, open threads, the
tools index, all silently absent, every session, under a message that never named the rule. A
rule file arrives with a clone, so this needed no local access at all.&lt;/li&gt;
&lt;li&gt;A committed tools index with &lt;code&gt;"runs": "12"&lt;/code&gt; instead of &lt;code&gt;12&lt;/code&gt; crashed the same hook through
&lt;code&gt;sort(key=lambda t: -(t.get("runs") or 0))&lt;/code&gt;. The &lt;em&gt;name&lt;/em&gt; in that file was validated, because a
name becomes a path. A sort key is exactly where an untrusted field turns into arithmetic.&lt;/li&gt;
&lt;li&gt;All three context-injecting hooks failed to strip zero-width characters, each differently. Two
bypassed the sanitiser with a raw write; the third called it faithfully on the finished JSON, by
which point &lt;code&gt;json.dumps&lt;/code&gt; has escaped every smuggled code point to &lt;code&gt;\uXXXX&lt;/code&gt; text no character
filter matches and the model decodes straight back. Reproduced end to end: 44 Unicode Tag
characters in, an instruction back out.&lt;/li&gt;
&lt;li&gt;Eleven places wrapped a repository-derived name in backticks. POSIX permits a backtick in a
filename, and one closes the code span — after which the repository's data renders as the tool's
own voice. A report named two of them; the sweep found nine more.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What I would do differently, since that is the more useful half
&lt;/h2&gt;

&lt;p&gt;The Windows column of the CI matrix caught five things a Mac could not, and I spent an evening&lt;br&gt;
learning them one push at a time. Four of the five were findable locally and I did not look: a&lt;br&gt;
&lt;code&gt;#!/bin/sh&lt;/code&gt; test fixture Windows cannot execute, a path compared without normalising separators, a&lt;br&gt;
&lt;code&gt;/nonexistent-…&lt;/code&gt; path that Windows resolves against the current drive and happily creates, and a&lt;br&gt;
subprocess spawn added to the session-start path that took CI from 4m16s to 9m25s.&lt;/p&gt;

&lt;p&gt;The fifth was a real concurrency defect, and it is the one worth writing down. The file lock gave a&lt;br&gt;
waiter a fixed two seconds and then let the caller write unguarded. Four hundred turns through one&lt;br&gt;
lock drain inside two seconds only if a turn is cheap; on Windows it is not, and the counter came&lt;br&gt;
back 41 of 400 — lost, silently, on a running total nothing recomputes.&lt;/p&gt;

&lt;p&gt;I reproduced it on macOS in thirty seconds by shrinking the ceiling to 0.05 s instead of finding a&lt;br&gt;
Windows machine. &lt;strong&gt;Squeezing a constant is the same experiment as slowing the disk&lt;/strong&gt;, and it runs&lt;br&gt;
where you are already sitting. That squeeze is a permanent check now, on every platform.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rest
&lt;/h2&gt;

&lt;p&gt;MIT. Standard library only — no dependency step in CI, and if it ever needs one, that is the change&lt;br&gt;
to reject rather than the workflow to fix. 21 languages parsed, 23 adapters, tested on Ubuntu,&lt;br&gt;
macOS and Windows across Python 3.8 and 3.13.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://github.com/ArcticFox2029/chamnan" rel="noopener noreferrer"&gt;github.com/ArcticFox2029/chamnan&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>claude</category>
      <category>opensource</category>
      <category>devtools</category>
    </item>
    <item>
      <title>Built a local pre-processing plugin for Claude Code after research &amp; measuring real metrics (3,500+ tests, v1.22.1) — Looking for feedback &amp; ideas</title>
      <dc:creator>ArcticFoxz</dc:creator>
      <pubDate>Mon, 07 Sep 2026 07:11:16 +0000</pubDate>
      <link>https://dev.to/arcticfoxz_255bbe944bc015/built-a-local-pre-processing-plugin-for-claude-code-after-research-measuring-real-metrics-3500-485n</link>
      <guid>https://dev.to/arcticfoxz_255bbe944bc015/built-a-local-pre-processing-plugin-for-claude-code-after-research-measuring-real-metrics-3500-485n</guid>
      <description>&lt;p&gt;Hi everyone,&lt;/p&gt;

&lt;p&gt;I wanted to share a project I’ve been developing for my own workflow with Claude Code, and hopefully get some feedback and ideas from the community on where to take it next.&lt;/p&gt;

&lt;h3&gt;
  
  
  How it started
&lt;/h3&gt;

&lt;p&gt;Working on large-scale, complex projects, I needed better control over context, token consumption, and security. Before building my own tool, I tested various plugins in the market. As someone who likes to measure and verify metrics independently, I ran tests on actual context usage and noticed that under large, messy repository conditions, results often varied from what was expected.&lt;/p&gt;

&lt;p&gt;Since my projects involve sensitive codebases, I wanted a &lt;strong&gt;deterministic local pre-processing layer&lt;/strong&gt; that sanitizes, budgets, and organizes data &lt;em&gt;before&lt;/em&gt; anything is passed to Claude Code—ensuring zero unexpected data leaks and absolute predictability.&lt;/p&gt;

&lt;h3&gt;
  
  
  The struggle &amp;amp; research
&lt;/h3&gt;

&lt;p&gt;What started as a simple helper script turned into a serious engineering effort. I went through over 1,000 research papers and technical docs to refine context retention, token budgeting, and local redaction.&lt;/p&gt;

&lt;p&gt;Building reliability tooling is a constant battle against edge cases: silent context loss, false successes, cross-platform path quirks (like Windows process locks), and state drift. &lt;/p&gt;

&lt;p&gt;My rule became: &lt;em&gt;Never believe a bug is fixed until a test proves it.&lt;/em&gt; &lt;/p&gt;

&lt;p&gt;Today, the plugin (&lt;code&gt;chamnan&lt;/code&gt;) is at &lt;strong&gt;v1.22.1&lt;/strong&gt;. It's built in pure Python (standard library only, zero external packages) and backed by &lt;strong&gt;3,500+ automated tests&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  What it actually does (It's NOT an AI, it's a pre-processing / reliability layer)
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Deterministic Context Budgeting:&lt;/strong&gt; Pre-indexes and trims repository context before passing it to Claude Code (slashed Quick Index overhead dramatically in real-world tests).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Local Security &amp;amp; Redaction:&lt;/strong&gt; Runs local choke-point scrubbing so secrets and internal strings never leave your machine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Agent Continuity / Memory:&lt;/strong&gt; Keeps track of session decisions, lessons, and project history locally inside &lt;code&gt;.chamnan/&lt;/code&gt; so Claude Code doesn't waste tokens re-discovering architecture across sessions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Monorepo &amp;amp; Multi-adapter support:&lt;/strong&gt; Designed to handle nested checkouts, messy directory trees, and OS-specific edge cases cleanly.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Looking for community feedback
&lt;/h3&gt;

&lt;p&gt;I dogfood this daily, but since I built it to share with other developers working on large or privacy-conscious projects, I'd love to hear your thoughts:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;What edge cases or repository structures have broken your Claude Code workflows in the past?&lt;/li&gt;
&lt;li&gt;What safety verifications or context features would you like to see in a local helper plugin?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you're interested in the architecture or test setup, you can check out the repository here: &lt;a href="https://github.com/ArcticFox2029/chamnan" rel="noopener noreferrer"&gt;https://github.com/ArcticFox2029/chamnan&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Appreciate any feedback!&lt;br&gt;
&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fchzf4ia0mqjel66pmz1r.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fchzf4ia0mqjel66pmz1r.png" alt=" " width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>showdev</category>
      <category>security</category>
    </item>
  </channel>
</rss>
