<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Jianzhujisuan</title>
    <description>The latest articles on DEV Community by Jianzhujisuan (@jianzhujisuan).</description>
    <link>https://dev.to/jianzhujisuan</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4215661%2Ffc715d8d-5ae7-4f51-ba83-4ce606f7d9f3.png</url>
      <title>DEV Community: Jianzhujisuan</title>
      <link>https://dev.to/jianzhujisuan</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jianzhujisuan"/>
    <language>en</language>
    <item>
      <title>I unit-tested the 131 conversion equations in my own docs. The tolerance was the bug.</title>
      <dc:creator>Jianzhujisuan</dc:creator>
      <pubDate>Fri, 20 Nov 2026 07:15:48 +0000</pubDate>
      <link>https://dev.to/jianzhujisuan/i-unit-tested-the-131-conversion-equations-in-my-own-docs-the-tolerance-was-the-bug-1ol0</link>
      <guid>https://dev.to/jianzhujisuan/i-unit-tested-the-131-conversion-equations-in-my-own-docs-the-tolerance-was-the-bug-1ol0</guid>
      <description>&lt;p&gt;The check I added to my build last week has printed &lt;code&gt;0 arithmetic errors&lt;/code&gt; on every single run. Then I scrolled a live page and read one row of a conversion table:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;C30 concrete = 30 MPa = 30 N/mm² = 300 kgf/cm²
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;30 MPa is 305.9 kgf/cm². That row is off by 1.93%, and the check that exists specifically to catch wrong numbers had waved it through.&lt;/p&gt;

&lt;p&gt;Not a regex bug. Not a missing test case. One line of tolerance arithmetic — the part everybody writes in thirty seconds and never looks at again.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the check actually does
&lt;/h2&gt;

&lt;p&gt;The site publishes 242 pages in each of two languages, about 2.7 million characters of visible text. Buried in that prose are sentences like &lt;code&gt;1 in = 25.4 mm&lt;/code&gt; and &lt;code&gt;1 ft³ = 0.0283168 m³&lt;/code&gt;. Unit conversions are the one class of statement where correctness is decided by arithmetic alone — no editor, no source, no judgement. Multiply, compare, done.&lt;/p&gt;

&lt;p&gt;So the check pulls every &lt;code&gt;number unit = number unit&lt;/code&gt; out of the rendered HTML, converts both sides to SI, and compares:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="no"&gt;FAC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"mm"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1e-3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"m"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"in"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.0254&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ft"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.3048&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ft³"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.028316846592&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;calc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;num_a&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="no"&gt;FAC&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;unit_a&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="no"&gt;FAC&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;unit_b&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;num_b&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;calc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;tolerance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;num_b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;flag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everything hinges on that last function, and the obvious choice is a percentage:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;tolerance&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.01&lt;/span&gt;      &lt;span class="c1"&gt;# ±1%, "close enough"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It reads as reasonable. It is wrong in both directions at once, and the two failures are not symmetric — one is loud and one is silent.&lt;/p&gt;

&lt;h2&gt;
  
  
  A percentage is the wrong shape
&lt;/h2&gt;

&lt;p&gt;Run the same 131 equations under three rules:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Rule&lt;/th&gt;
&lt;th&gt;Flagged&lt;/th&gt;
&lt;th&gt;Correct values rejected&lt;/th&gt;
&lt;th&gt;Real errors caught&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;±1% of the value&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;±0.1% of the value&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;half of the last printed digit&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The ±0.1% column is the interesting one. Six of the seven things it flagged are correct:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;3/4 in  = 19 mm       (true 19.05)
1.5 in  = 38 mm       (true 38.10)
3 in    = 76 mm       (true 76.20)
80 ft³  = 2.96 yd³    (true 2.96296)
4 in    = 0.333 ft    (true 0.333333)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are not errors. They are a table rounded to whole millimetres, which is what a table of construction dimensions is supposed to look like. A relative tolerance treats precision as a property of the &lt;em&gt;magnitude&lt;/em&gt;: bigger number, more slack. Printed precision does not work that way. It is a property of how the author typed it.&lt;/p&gt;

&lt;p&gt;The same 1% window means wildly different things depending on where you apply it:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Equation&lt;/th&gt;
&lt;th&gt;1% of it&lt;/th&gt;
&lt;th&gt;Half the last printed digit&lt;/th&gt;
&lt;th&gt;Ratio&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;1 lb = 453.59237 g&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;4.54 g&lt;/td&gt;
&lt;td&gt;0.000005 g&lt;/td&gt;
&lt;td&gt;907,094×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;1 ft³ = 0.0283168 m³&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;0.00028 m³&lt;/td&gt;
&lt;td&gt;0.00000005 m³&lt;/td&gt;
&lt;td&gt;5,663×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;1 in = 25.4 mm&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;0.254 mm&lt;/td&gt;
&lt;td&gt;0.05 mm&lt;/td&gt;
&lt;td&gt;5×&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the first row, a 1% tolerance will happily accept an error of four and a half grams on a value that was printed to eight significant figures. Across the corpus, the median 1% window is 5× the last printed digit; the 90th percentile is 13,788×; 58 of the 135 equations get a window at least ten times too wide. That is the silent failure — nothing is flagged, so nothing looks wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  And the "last digit" rule has a hole of its own
&lt;/h2&gt;

&lt;p&gt;So derive the tolerance from the printed form: half of the last digit shown. &lt;code&gt;25.4&lt;/code&gt; allows 0.05, &lt;code&gt;0.0283168&lt;/code&gt; allows 5×10⁻⁸. Write more digits, you have to be more correct. That rule flagged nothing in the corpus, which felt like a win until it also failed to flag the 300 kgf/cm² row.&lt;/p&gt;

&lt;p&gt;Because &lt;code&gt;300&lt;/code&gt; is ambiguous. It can mean 300 ± 50, or 3.0×10² ± 5. Nothing in the string tells you which, and the rule I wrote assumes the worst case:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;"."&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;decimals&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;decimals&lt;/span&gt;
&lt;span class="n"&gt;sig&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"0"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;sig&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For &lt;code&gt;1000&lt;/code&gt;, that returns &lt;strong&gt;±500&lt;/strong&gt;. Thirty of the 135 equations end in a zero, so their tolerance is enormous — &lt;code&gt;1 m = 1000 mm&lt;/code&gt; would pass as &lt;code&gt;1500&lt;/code&gt;, and 300 passes as 305.9. The strictest-looking rule in the file is the one with the widest hole.&lt;/p&gt;

&lt;p&gt;There is no clever fix here. Printed precision is a claim, and a bare trailing zero declines to make it. The workable response is to stop guessing: either the copy prints significant figures explicitly (&lt;code&gt;3.0×10²&lt;/code&gt;, or &lt;code&gt;300.0&lt;/code&gt;), or the value gets queued for a human instead of counted as verified. My checker now reports three buckets — verified, failed, undecidable — and I look at the third. It is 30 rows. It takes ten minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four ways the parser lies, and how much it costs
&lt;/h2&gt;

&lt;p&gt;The tolerance argument is the intellectual part. The parser is where you lose coverage without noticing, because every fix for a false positive is also a fix that quietly stops checking something.&lt;/p&gt;

&lt;p&gt;I broke the checker on purpose, four ways, against the same 484 pages:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Broken on purpose&lt;/th&gt;
&lt;th&gt;Verified&lt;/th&gt;
&lt;th&gt;Unverifiable&lt;/th&gt;
&lt;th&gt;False errors&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Correct&lt;/td&gt;
&lt;td&gt;131&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unit table sorted alphabetically, not by length&lt;/td&gt;
&lt;td&gt;115&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No fraction support (&lt;code&gt;3/4 in&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;129&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No "deliberate counter-example" filter&lt;/td&gt;
&lt;td&gt;131&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Unit ordering.&lt;/strong&gt; &lt;code&gt;re&lt;/code&gt; tries alternatives left to right, so with &lt;code&gt;m&lt;/code&gt; before &lt;code&gt;mm&lt;/code&gt;, the string &lt;code&gt;25.4 mm&lt;/code&gt; matches &lt;code&gt;25.4 m&lt;/code&gt; and the rest is noise. Sixteen equations stop being checked and the error list stays empty. Sort your unit table by descending length. This one is invisible in review and invisible in CI unless you assert the count.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fractions.&lt;/strong&gt; &lt;code&gt;3/4 in = 19 mm&lt;/code&gt; is a quarter-inch value written the way a carpenter writes it. Naively, &lt;code&gt;4 in = 19 mm&lt;/code&gt; is off by 101.6 mm and you get a spectacular false positive. The guard that prevents it — &lt;code&gt;(?&amp;lt;![\w./])&lt;/code&gt; — also means the equation never matches at all, so it silently stops being verified. Two equations in my corpus are in that state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Deliberate counter-examples.&lt;/strong&gt; One page teaches the classic mistake: &lt;em&gt;"people assume &lt;code&gt;1 m² = 100 cm²&lt;/code&gt;"&lt;/em&gt;. It is correct content and a wrong equation. A checker without a negative-context filter reports two errors (English and Chinese) on a page that is doing its job.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cross-family equations.&lt;/strong&gt; &lt;code&gt;1 m³ = 1 t&lt;/code&gt; is true for water and false for concrete. Ten equations in the corpus relate volume to mass; none of them can be verified without knowing the material, so they are listed for reading, not scoring.&lt;/p&gt;

&lt;p&gt;The lesson I keep re-learning: &lt;strong&gt;assert the number of checks, not just the number of failures.&lt;/strong&gt; A linter that checks nothing passes everything. Mine now fails the build if the verified count drops more than 20% between runs, which is how I noticed the alphabetical-sort experiment instead of shipping it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two errors it did catch
&lt;/h2&gt;

&lt;p&gt;Both were found in the first week, both in the same family — precision drifting between two places that are supposed to agree.&lt;/p&gt;

&lt;p&gt;A table cell where three-quarter inch had been rounded to &lt;strong&gt;75 mm&lt;/strong&gt;, while the cells directly above and below it were rounded to 19 and 38. Same table, three different rounding conventions, one cell wrong by the width of a millimetre.&lt;/p&gt;

&lt;p&gt;And an intermediate value printed as &lt;code&gt;0.771&lt;/code&gt; (true 0.7715) that the next formula in the chain multiplied as if it were 0.7715 — the printed derivation and the computed result disagreed in the third digit. Neither error is visible to a reader skimming for plausibility. Both are trivially visible to a machine that recomputes.&lt;/p&gt;

&lt;p&gt;That is the whole case for this kind of check: not that it is clever, but that it is exhaustive on the one class of claim where exhaustiveness is cheap.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running it on your own docs
&lt;/h2&gt;

&lt;p&gt;You do not need my unit table — you need four things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The rendered output&lt;/strong&gt;, not the templates. Half of my equations only exist after variable interpolation, and the HTML tags sit between the number and its unit in ways the source never shows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A unit table sorted by string length&lt;/strong&gt;, with every unit's SI factor.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A tolerance derived from the printed form&lt;/strong&gt;, plus an explicit "undecidable" bucket for trailing zeros and for anything cross-family.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A count assertion in CI.&lt;/strong&gt; &lt;code&gt;verified &amp;lt; previous * 0.8&lt;/code&gt; should fail the build. This is the part that catches the parser silently going blind.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The script is about 120 lines of Python, no dependencies, and runs in under a second over 2.7 million characters.&lt;/p&gt;

&lt;p&gt;I maintain a set of construction-quantity calculators, which is why my prose is unusually full of conversion factors — the &lt;a href="https://jianzhujisuan.com/en/construction-unit-table?src=devto" rel="noopener noreferrer"&gt;unit conversion table&lt;/a&gt; is the page that generated the most equations, and the one where a wrong digit costs somebody a truckload of concrete. But the pattern is not specific to construction. Any documentation with prices, benchmarks, tolerances, or version-dated numbers in it has this property: the numbers are testable, and almost nobody tests them.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;0 arithmetic errors&lt;/code&gt; line in my build log was not evidence of correctness. It was evidence that the check ran. Those are different claims, and only one of them is worth putting in a build log.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>testing</category>
      <category>python</category>
      <category>tooling</category>
    </item>
    <item>
      <title>I unit-tested the 131 conversion equations in my own docs. The tolerance was the bug.</title>
      <dc:creator>Jianzhujisuan</dc:creator>
      <pubDate>Sat, 14 Nov 2026 14:03:37 +0000</pubDate>
      <link>https://dev.to/jianzhujisuan/i-unit-tested-the-131-conversion-equations-in-my-own-docs-the-tolerance-was-the-bug-5gak</link>
      <guid>https://dev.to/jianzhujisuan/i-unit-tested-the-131-conversion-equations-in-my-own-docs-the-tolerance-was-the-bug-5gak</guid>
      <description>&lt;p&gt;The check I added to my build last week has printed &lt;code&gt;0 arithmetic errors&lt;/code&gt; on every single run. Then I scrolled a live page and read one row of a conversion table:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;C30 concrete = 30 MPa = 30 N/mm² = 300 kgf/cm²
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;30 MPa is 305.9 kgf/cm². That row is off by 1.93%, and the check that exists specifically to catch wrong numbers had waved it through.&lt;/p&gt;

&lt;p&gt;Not a regex bug. Not a missing test case. One line of tolerance arithmetic — the part everybody writes in thirty seconds and never looks at again.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the check actually does
&lt;/h2&gt;

&lt;p&gt;The site publishes 242 pages in each of two languages, about 2.7 million characters of visible text. Buried in that prose are sentences like &lt;code&gt;1 in = 25.4 mm&lt;/code&gt; and &lt;code&gt;1 ft³ = 0.0283168 m³&lt;/code&gt;. Unit conversions are the one class of statement where correctness is decided by arithmetic alone — no editor, no source, no judgement. Multiply, compare, done.&lt;/p&gt;

&lt;p&gt;So the check pulls every &lt;code&gt;number unit = number unit&lt;/code&gt; out of the rendered HTML, converts both sides to SI, and compares:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="no"&gt;FAC&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"mm"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1e-3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"m"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"in"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.0254&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ft"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.3048&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"ft³"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mf"&gt;0.028316846592&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;calc&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;num_a&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="no"&gt;FAC&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;unit_a&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="no"&gt;FAC&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;unit_b&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;num_b&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;calc&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;tolerance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;num_b&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;flag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Everything hinges on that last function, and the obvious choice is a percentage:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;tolerance&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;lambda&lt;/span&gt; &lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.01&lt;/span&gt;      &lt;span class="c1"&gt;# ±1%, "close enough"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It reads as reasonable. It is wrong in both directions at once, and the two failures are not symmetric — one is loud and one is silent.&lt;/p&gt;

&lt;h2&gt;
  
  
  A percentage is the wrong shape
&lt;/h2&gt;

&lt;p&gt;Run the same 131 equations under three rules:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Rule&lt;/th&gt;
&lt;th&gt;Flagged&lt;/th&gt;
&lt;th&gt;Correct values rejected&lt;/th&gt;
&lt;th&gt;Real errors caught&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;±1% of the value&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;±0.1% of the value&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;half of the last printed digit&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The ±0.1% column is the interesting one. Six of the seven things it flagged are correct:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;3/4 in  = 19 mm       (true 19.05)
1.5 in  = 38 mm       (true 38.10)
3 in    = 76 mm       (true 76.20)
80 ft³  = 2.96 yd³    (true 2.96296)
4 in    = 0.333 ft    (true 0.333333)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those are not errors. They are a table rounded to whole millimetres, which is what a table of construction dimensions is supposed to look like. A relative tolerance treats precision as a property of the &lt;em&gt;magnitude&lt;/em&gt;: bigger number, more slack. Printed precision does not work that way. It is a property of how the author typed it.&lt;/p&gt;

&lt;p&gt;The same 1% window means wildly different things depending on where you apply it:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Equation&lt;/th&gt;
&lt;th&gt;1% of it&lt;/th&gt;
&lt;th&gt;Half the last printed digit&lt;/th&gt;
&lt;th&gt;Ratio&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;1 lb = 453.59237 g&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;4.54 g&lt;/td&gt;
&lt;td&gt;0.000005 g&lt;/td&gt;
&lt;td&gt;907,094×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;1 ft³ = 0.0283168 m³&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;0.00028 m³&lt;/td&gt;
&lt;td&gt;0.00000005 m³&lt;/td&gt;
&lt;td&gt;5,663×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;1 in = 25.4 mm&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;0.254 mm&lt;/td&gt;
&lt;td&gt;0.05 mm&lt;/td&gt;
&lt;td&gt;5×&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the first row, a 1% tolerance will happily accept an error of four and a half grams on a value that was printed to eight significant figures. Across the corpus, the median 1% window is 5× the last printed digit; the 90th percentile is 13,788×; 58 of the 135 equations get a window at least ten times too wide. That is the silent failure — nothing is flagged, so nothing looks wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  And the "last digit" rule has a hole of its own
&lt;/h2&gt;

&lt;p&gt;So derive the tolerance from the printed form: half of the last digit shown. &lt;code&gt;25.4&lt;/code&gt; allows 0.05, &lt;code&gt;0.0283168&lt;/code&gt; allows 5×10⁻⁸. Write more digits, you have to be more correct. That rule flagged nothing in the corpus, which felt like a win until it also failed to flag the 300 kgf/cm² row.&lt;/p&gt;

&lt;p&gt;Because &lt;code&gt;300&lt;/code&gt; is ambiguous. It can mean 300 ± 50, or 3.0×10² ± 5. Nothing in the string tells you which, and the rule I wrote assumes the worst case:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;"."&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;decimals&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"."&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;decimals&lt;/span&gt;
&lt;span class="n"&gt;sig&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;rstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"0"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;printed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;sig&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For &lt;code&gt;1000&lt;/code&gt;, that returns &lt;strong&gt;±500&lt;/strong&gt;. Thirty of the 135 equations end in a zero, so their tolerance is enormous — &lt;code&gt;1 m = 1000 mm&lt;/code&gt; would pass as &lt;code&gt;1500&lt;/code&gt;, and 300 passes as 305.9. The strictest-looking rule in the file is the one with the widest hole.&lt;/p&gt;

&lt;p&gt;There is no clever fix here. Printed precision is a claim, and a bare trailing zero declines to make it. The workable response is to stop guessing: either the copy prints significant figures explicitly (&lt;code&gt;3.0×10²&lt;/code&gt;, or &lt;code&gt;300.0&lt;/code&gt;), or the value gets queued for a human instead of counted as verified. My checker now reports three buckets — verified, failed, undecidable — and I look at the third. It is 30 rows. It takes ten minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four ways the parser lies, and how much it costs
&lt;/h2&gt;

&lt;p&gt;The tolerance argument is the intellectual part. The parser is where you lose coverage without noticing, because every fix for a false positive is also a fix that quietly stops checking something.&lt;/p&gt;

&lt;p&gt;I broke the checker on purpose, four ways, against the same 484 pages:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Broken on purpose&lt;/th&gt;
&lt;th&gt;Verified&lt;/th&gt;
&lt;th&gt;Unverifiable&lt;/th&gt;
&lt;th&gt;False errors&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Correct&lt;/td&gt;
&lt;td&gt;131&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unit table sorted alphabetically, not by length&lt;/td&gt;
&lt;td&gt;115&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No fraction support (&lt;code&gt;3/4 in&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;129&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No "deliberate counter-example" filter&lt;/td&gt;
&lt;td&gt;131&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Unit ordering.&lt;/strong&gt; &lt;code&gt;re&lt;/code&gt; tries alternatives left to right, so with &lt;code&gt;m&lt;/code&gt; before &lt;code&gt;mm&lt;/code&gt;, the string &lt;code&gt;25.4 mm&lt;/code&gt; matches &lt;code&gt;25.4 m&lt;/code&gt; and the rest is noise. Sixteen equations stop being checked and the error list stays empty. Sort your unit table by descending length. This one is invisible in review and invisible in CI unless you assert the count.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fractions.&lt;/strong&gt; &lt;code&gt;3/4 in = 19 mm&lt;/code&gt; is a quarter-inch value written the way a carpenter writes it. Naively, &lt;code&gt;4 in = 19 mm&lt;/code&gt; is off by 101.6 mm and you get a spectacular false positive. The guard that prevents it — &lt;code&gt;(?&amp;lt;![\w./])&lt;/code&gt; — also means the equation never matches at all, so it silently stops being verified. Two equations in my corpus are in that state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Deliberate counter-examples.&lt;/strong&gt; One page teaches the classic mistake: &lt;em&gt;"people assume &lt;code&gt;1 m² = 100 cm²&lt;/code&gt;"&lt;/em&gt;. It is correct content and a wrong equation. A checker without a negative-context filter reports two errors (English and Chinese) on a page that is doing its job.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cross-family equations.&lt;/strong&gt; &lt;code&gt;1 m³ = 1 t&lt;/code&gt; is true for water and false for concrete. Ten equations in the corpus relate volume to mass; none of them can be verified without knowing the material, so they are listed for reading, not scoring.&lt;/p&gt;

&lt;p&gt;The lesson I keep re-learning: &lt;strong&gt;assert the number of checks, not just the number of failures.&lt;/strong&gt; A linter that checks nothing passes everything. Mine now fails the build if the verified count drops more than 20% between runs, which is how I noticed the alphabetical-sort experiment instead of shipping it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two errors it did catch
&lt;/h2&gt;

&lt;p&gt;Both were found in the first week, both in the same family — precision drifting between two places that are supposed to agree.&lt;/p&gt;

&lt;p&gt;A table cell where three-quarter inch had been rounded to &lt;strong&gt;75 mm&lt;/strong&gt;, while the cells directly above and below it were rounded to 19 and 38. Same table, three different rounding conventions, one cell wrong by the width of a millimetre.&lt;/p&gt;

&lt;p&gt;And an intermediate value printed as &lt;code&gt;0.771&lt;/code&gt; (true 0.7715) that the next formula in the chain multiplied as if it were 0.7715 — the printed derivation and the computed result disagreed in the third digit. Neither error is visible to a reader skimming for plausibility. Both are trivially visible to a machine that recomputes.&lt;/p&gt;

&lt;p&gt;That is the whole case for this kind of check: not that it is clever, but that it is exhaustive on the one class of claim where exhaustiveness is cheap.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running it on your own docs
&lt;/h2&gt;

&lt;p&gt;You do not need my unit table — you need four things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The rendered output&lt;/strong&gt;, not the templates. Half of my equations only exist after variable interpolation, and the HTML tags sit between the number and its unit in ways the source never shows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A unit table sorted by string length&lt;/strong&gt;, with every unit's SI factor.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A tolerance derived from the printed form&lt;/strong&gt;, plus an explicit "undecidable" bucket for trailing zeros and for anything cross-family.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A count assertion in CI.&lt;/strong&gt; &lt;code&gt;verified &amp;lt; previous * 0.8&lt;/code&gt; should fail the build. This is the part that catches the parser silently going blind.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The script is about 120 lines of Python, no dependencies, and runs in under a second over 2.7 million characters.&lt;/p&gt;

&lt;p&gt;I maintain a set of construction-quantity calculators, which is why my prose is unusually full of conversion factors — the &lt;a href="https://jianzhujisuan.com/en/construction-unit-table" rel="noopener noreferrer"&gt;unit conversion table&lt;/a&gt; is the page that generated the most equations, and the one where a wrong digit costs somebody a truckload of concrete. But the pattern is not specific to construction. Any documentation with prices, benchmarks, tolerances, or version-dated numbers in it has this property: the numbers are testable, and almost nobody tests them.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;0 arithmetic errors&lt;/code&gt; line in my build log was not evidence of correctness. It was evidence that the check ran. Those are different claims, and only one of them is worth putting in a build log.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>testing</category>
      <category>python</category>
      <category>tooling</category>
    </item>
    <item>
      <title>Your analytics said 32 visitors yesterday. Nine of them were people.</title>
      <dc:creator>Jianzhujisuan</dc:creator>
      <pubDate>Sun, 08 Nov 2026 16:34:13 +0000</pubDate>
      <link>https://dev.to/jianzhujisuan/your-analytics-said-32-visitors-yesterday-nine-of-them-were-people-49dd</link>
      <guid>https://dev.to/jianzhujisuan/your-analytics-said-32-visitors-yesterday-nine-of-them-were-people-49dd</guid>
      <description>&lt;p&gt;Yesterday my dashboard said 32 visitors, 20 unique. I spent the evening taking it apart. Nine of them were people.&lt;/p&gt;

&lt;p&gt;Not nine out of twenty. Nine out of thirty-two page views. The unique-visitor number was inflated by 55% and the page-view number by 34%, on a day with no marketing push, no post, nothing that would attract a spike.&lt;/p&gt;

&lt;p&gt;The interesting part is not that bots were in the numbers. That is old news. The interesting part is &lt;em&gt;why my bot filter missed them&lt;/em&gt;, because the answer is a problem every site now has and almost nobody has fixed.&lt;/p&gt;

&lt;h2&gt;
  
  
  What my filter actually did
&lt;/h2&gt;

&lt;p&gt;The backend classifies each hit by user agent:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;UA_BOT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sr"&gt;/bot|crawl|spider|slurp|headless|phantom|puppeteer|playwright|selenium|curl&lt;/span&gt;&lt;span class="se"&gt;\/&lt;/span&gt;&lt;span class="sr"&gt;|wget|python-|go-http-client|java&lt;/span&gt;&lt;span class="se"&gt;\/&lt;/span&gt;&lt;span class="sr"&gt;|okhttp|axios|node-fetch|scrapy|semrush|ahrefs|mj12|dotbot|petalbot|yandex|facebookexternalhit|applebot|discordbot|telegrambot|whatsapp|uptime|monitor|scanner|nmap|masscan|googleother|google-inspectiontool|apis-google|mediapartners|feedfetcher|builtwith|dataforseo|lighthouse/i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;classifyUA&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ua&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;?&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;UA_BOT&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;bot&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;UA_TB&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;tb&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;UA_MB&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;mb&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;UA_PC&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;pc&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;?&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That regex is long. It catches Applebot, GoogleOther, facebookexternalhit, headless Chrome, and about thirty things I have never seen on my own traffic. On paper it looks thorough.&lt;/p&gt;

&lt;p&gt;Here is a hit it let through, straight from the log:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51
Mobile Safari/537.36 MicroMessenger/7.0.1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read it word by word. &lt;code&gt;Android 6.0&lt;/code&gt;. &lt;code&gt;Nexus 5&lt;/code&gt;. &lt;code&gt;Chrome/99&lt;/code&gt;. &lt;code&gt;MicroMessenger/7.0.1&lt;/code&gt; — it claims to be WeChat on a phone. There is no &lt;code&gt;bot&lt;/code&gt;, no &lt;code&gt;crawl&lt;/code&gt;, no &lt;code&gt;headless&lt;/code&gt;, nothing my regex could possibly match. It walked in as &lt;code&gt;mb&lt;/code&gt; — mobile human.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Thirty-one different IP addresses sent that exact string over twelve days.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;WeChat has over a billion users. The odds that thirty-one of them are on a Nexus 5 running Android 6.0, sharing a build number from 2015, are not small — they are zero. That UA is the default device profile of a browser automation tool, and whoever runs it knows the filter only reads words.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap I nearly fell into myself
&lt;/h2&gt;

&lt;p&gt;Then there was the opposite mistake, which is worse.&lt;/p&gt;

&lt;p&gt;Another fourteen hits in a single day carried a version number ending in three zeros:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;Chrome/145.0.0.0&lt;/code&gt;. My first instinct was: nobody's real Chrome reports &lt;code&gt;.0.0.0&lt;/code&gt;, that is obviously forged.&lt;/p&gt;

&lt;p&gt;Wrong. Since Chrome 110, the browser &lt;strong&gt;reduces&lt;/strong&gt; its own UA. &lt;code&gt;145.0.0.0&lt;/code&gt; is what genuine Chrome sends. There are no minor or patch digits any more, for privacy. My instinct would have deleted real Mac users from the report and I would have called it a job well done.&lt;/p&gt;

&lt;p&gt;So the string is real, and simultaneously useless: every Chrome 145 user on macOS now presents the identical UA. Real visitors and fake ones are indistinguishable by fingerprint alone.&lt;/p&gt;

&lt;p&gt;That is the actual problem. UA reduction did not just remove entropy from the header — it gave crawlers a crowd of legitimate users to hide in. A UA blacklist is now structurally unable to separate them, no matter how many keywords you add.&lt;/p&gt;

&lt;h2&gt;
  
  
  The question that does work
&lt;/h2&gt;

&lt;p&gt;Stop asking &lt;em&gt;"is this UA a robot?"&lt;/em&gt; Ask &lt;em&gt;"is this UA too popular for the number of IPs using it?"&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Real people are metered two ways: their IP is stable (one address, many page views) or they appear once and leave (one address, one view). A fleet rotating through proxies looks like neither — it looks like &lt;strong&gt;many addresses, each visiting exactly once, all presenting the same header&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So for every UA string I compute three numbers across the whole dataset:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;ipn&lt;/code&gt; — how many distinct IP addresses used it&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pv&lt;/code&gt; — total page views from it&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;per&lt;/code&gt; — &lt;code&gt;pv / ipn&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;and flag it when &lt;code&gt;ipn &amp;gt;= 3&lt;/code&gt; &lt;strong&gt;and&lt;/strong&gt; &lt;code&gt;per &amp;lt;= 1.6&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Cloudflare Worker / D1 — run once, cache the result&lt;/span&gt;
&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;fingerprintReuse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;prepare&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`
    SELECT ua,
           COUNT(*)                       AS pv,
           COUNT(DISTINCT ip)             AS ipn,
           COUNT(DISTINCT d)              AS days
    FROM hits
    WHERE ua IS NOT NULL
    GROUP BY ua
  `&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;fleet&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;results&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;per&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;pv&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ipn&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nx"&gt;fleet&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;ipn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ipn&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;pv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;pv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="nx"&gt;per&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="na"&gt;squad&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ipn&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;per&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mf"&gt;1.6&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;fleet&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then the per-hit decision becomes a lookup instead of a regex:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;fleet&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ua&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;toLowerCase&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;meta-webindexer&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;bot&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;// Facebook preview, not in any list&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;f&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;fleet&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;f&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;squad&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;bot&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;                    &lt;span class="c1"&gt;// fleet fingerprint&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;classifyUA&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;                             &lt;span class="c1"&gt;// fall back to keywords&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The two conditions are doing different work. &lt;code&gt;ipn &amp;gt;= 3&lt;/code&gt; requires the coincidence to happen more than twice. &lt;code&gt;per &amp;lt;= 1.6&lt;/code&gt; is what separates a fleet from a shared office: ten colleagues behind one NAT gateway is &lt;em&gt;one&lt;/em&gt; IP with hundreds of views, so &lt;code&gt;per&lt;/code&gt; goes up, not down.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the data looked like after
&lt;/h2&gt;

&lt;p&gt;The top offenders on a site with a few hundred views a day:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;UA fingerprint&lt;/th&gt;
&lt;th&gt;IPs&lt;/th&gt;
&lt;th&gt;Views&lt;/th&gt;
&lt;th&gt;Days&lt;/th&gt;
&lt;th&gt;Views per IP&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;Chrome/145.0.0.0&lt;/code&gt; on &lt;code&gt;Macintosh&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;56&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;56&lt;/td&gt;
&lt;td&gt;13&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;Nexus 5 build/MRA58N&lt;/code&gt; + &lt;code&gt;MicroMessenger/7.0.1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;31&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;43&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;1.39&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CriOS/144.0.7559.95&lt;/code&gt; on iPhone&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;HD1900&lt;/code&gt; + &lt;code&gt;Chrome/75.0.3770.156&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;1.17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;Chrome/125.0.0.0&lt;/code&gt; on &lt;code&gt;Macintosh&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;Chrome/117.0.5938.132&lt;/code&gt; on &lt;code&gt;Windows&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;1.00&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Fifty-six addresses, fifty-six page views, thirteen days, exactly one view each. No human population behaves like that. Every one of those hits had already passed the keyword filter.&lt;/p&gt;

&lt;p&gt;A second signal shows up in the same table if you look at the version numbers. My real visitors arrive on Chrome 145 and 156. The fleet arrives on Chrome &lt;strong&gt;75, 79, 83, 88, 92, 99&lt;/strong&gt; — build numbers from 2019, on three separate IPs each, one view per IP. Twelve crawler fingerprints, not one of them running anything newer than 2021.&lt;/p&gt;

&lt;p&gt;That is a cheap heuristic on its own: &lt;em&gt;when a UA claims a browser version that has been obsolete for four years, and only three addresses ever used it, believe neither.&lt;/em&gt; A slow-moving UA library is the fingerprint of automation, because browsers auto-update and people do not stay on Chrome 79 for four years.&lt;/p&gt;

&lt;p&gt;The catch is that this cannot be your only rule either — a real visitor on a six-year-old Android phone looks exactly the same. Reuse across IPs is the part that survives that objection; version staleness is a supporting signal, not a verdict.&lt;/p&gt;

&lt;p&gt;Re-running the classifier with fingerprint reuse, day by day:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Day&lt;/th&gt;
&lt;th&gt;Reported views&lt;/th&gt;
&lt;th&gt;Real views&lt;/th&gt;
&lt;th&gt;Real visitors&lt;/th&gt;
&lt;th&gt;Dropped&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Nov 04&lt;/td&gt;
&lt;td&gt;37&lt;/td&gt;
&lt;td&gt;22&lt;/td&gt;
&lt;td&gt;14&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nov 05&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nov 06&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nov 07&lt;/td&gt;
&lt;td&gt;111&lt;/td&gt;
&lt;td&gt;88&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;20&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;23&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Nov 08&lt;/td&gt;
&lt;td&gt;32&lt;/td&gt;
&lt;td&gt;21&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;9&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;11&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Across the whole dataset: &lt;strong&gt;452 reported page views became 300. 245 reported unique visitors became 110.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Nearly every day my dashboard had been off by a third, and the one day it reported 42 unique visitors the true number was 20. For two weeks I had been making decisions about content and distribution on a number that was, on average, half fiction.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this approach still fails
&lt;/h2&gt;

&lt;p&gt;Two honest limits, because both will bite you.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It cannot see a crawler that keeps one IP.&lt;/strong&gt; A single-address scraper has &lt;code&gt;ipn = 1&lt;/code&gt; and never triggers. If that matters, the only reliable signal is request rate per IP per minute, which means keeping raw request logs, not just visits.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It will occasionally misclassify a real person.&lt;/strong&gt; Someone on a mobile carrier with CGNAT, browsing once, on a version of Chrome where every user shares a UA string, is structurally identical to a fleet member. My threshold (&lt;code&gt;ipn &amp;gt;= 3&lt;/code&gt;) is set to be conservative — it leaves some bots in rather than deleting real readers. If your traffic is large enough that &lt;code&gt;ipn &amp;gt;= 3&lt;/code&gt; is common for genuine users, raise the threshold and inspect the strings by hand before trusting the filter.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Aggregate across days, not within one.&lt;/strong&gt; A fleet may only send two hits a day. Judged on a single day it looks harmless; judged over two weeks the reuse is obvious. My first attempt ran the query per-day and flagged nothing, because 56 addresses spread over 13 days is only 4 per day.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I bothered
&lt;/h2&gt;

&lt;p&gt;I run a construction-calculator site — concrete volumes, rebar weights, formwork areas. It is a small site, a few hundred page views a day, and the only thing I have to steer by is where the visitors come from and what they look at. If a third of that is noise, every conclusion drawn from it is noise too: which page to expand, which channel to drop, whether that flurry on Tuesday was worth repeating.&lt;/p&gt;

&lt;p&gt;The number on the dashboard is not traffic. It is a claim about traffic, and it is worth checking about as often as you check a supplier's invoice.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;The site is &lt;a href="https://jianzhujisuan.com?src=devto" rel="noopener noreferrer"&gt;jianzhujisuan.com&lt;/a&gt; — Chinese-language calculators for construction quantity takeoff. The analytics code above runs on a Cloudflare Worker with D1 behind it; no third-party script, no cookie, which is also why there was nowhere else to look when the numbers went wrong.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>analytics</category>
      <category>javascript</category>
      <category>cloudflare</category>
    </item>
    <item>
      <title>Two rebar weights that look 8% apart are actually 61% apart</title>
      <dc:creator>Jianzhujisuan</dc:creator>
      <pubDate>Thu, 05 Nov 2026 03:07:24 +0000</pubDate>
      <link>https://dev.to/jianzhujisuan/two-rebar-weights-that-look-8-apart-are-actually-61-apart-4ddh</link>
      <guid>https://dev.to/jianzhujisuan/two-rebar-weights-that-look-8-apart-are-actually-61-apart-4ddh</guid>
      <description>&lt;p&gt;A rebar takeoff landed on my desk that balanced to the last bar and was still wrong by two tonnes.&lt;/p&gt;

&lt;p&gt;Every bar was the right length. Every count matched the drawing. The error was upstream of the arithmetic: two schedules had been converted from metric to imperial by someone who compared the &lt;em&gt;numbers&lt;/em&gt; instead of the &lt;em&gt;units&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;This is the least glamorous failure mode in quantity takeoff, and it survives every review because the arithmetic downstream is perfect. Wrong inputs, clean math, confident total.&lt;/p&gt;

&lt;p&gt;Here are the three conversions where it happens, how much each one costs, and the check that catches them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 1: the bar substitution that adds 61%
&lt;/h2&gt;

&lt;p&gt;A slab schedule says &lt;strong&gt;10 mm bars at 200 mm centres, both ways&lt;/strong&gt;. A fabricator in a US market quotes &lt;strong&gt;#4 at 8 inches&lt;/strong&gt; and calls it equivalent.&lt;/p&gt;

&lt;p&gt;The supporting logic is usually something like this: 10 mm is about 3/8 inch, #4 is 1/2 inch, close enough. That is already two approximations deep, and the second one is not close at all.&lt;/p&gt;

&lt;p&gt;The unit weights look deceptively similar, which is why nobody questions them:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;10 mm metric bar: &lt;strong&gt;0.617 kg/m&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;#4 bar: &lt;strong&gt;0.668 lb/ft&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;0.668 and 0.617 differ by 8%. That looks like rounding noise between two published tables. It is not. Different units, so the digits are not comparable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;0.668 lb/ft × 1.4882 kg/m per lb/ft = 0.994 kg/m
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The real gap is &lt;strong&gt;0.994 vs 0.617&lt;/strong&gt;, which is 61% heavier per metre. Same nominal "one bar per 200 mm" grid, 61% more steel.&lt;/p&gt;

&lt;p&gt;The area check says the same thing without any conversion table:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bar&lt;/th&gt;
&lt;th&gt;Nominal diameter&lt;/th&gt;
&lt;th&gt;Area&lt;/th&gt;
&lt;th&gt;Unit weight&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;10 mm metric&lt;/td&gt;
&lt;td&gt;10.0 mm&lt;/td&gt;
&lt;td&gt;78.5 mm²&lt;/td&gt;
&lt;td&gt;0.617 kg/m&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;#3 (3/8")&lt;/td&gt;
&lt;td&gt;9.5 mm&lt;/td&gt;
&lt;td&gt;71 mm²&lt;/td&gt;
&lt;td&gt;0.560 kg/m&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;#4 (1/2")&lt;/td&gt;
&lt;td&gt;12.7 mm&lt;/td&gt;
&lt;td&gt;129 mm²&lt;/td&gt;
&lt;td&gt;0.994 kg/m&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;129 / 78.5 = 1.64. &lt;strong&gt;#4 carries 64% more area than 10 mm.&lt;/strong&gt; It is not the soft-metric equivalent of a 10 mm bar; #3 is the nearest US size, and even that is 10% under on area.&lt;/p&gt;

&lt;h3&gt;
  
  
  Worked example: 500 m² of slab
&lt;/h3&gt;

&lt;p&gt;Both directions at 200 mm centres gives 5 running metres per square metre per direction:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;10 mm:  10 m/m² × 0.617 kg/m = 6.17 kg/m²  → 500 m² = 3,085 kg = 3.09 t
#4:     10 m/m² × 0.994 kg/m = 9.94 kg/m²  → 500 m² = 4,970 kg = 4.97 t
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;+1.88 t, +61%.&lt;/strong&gt; If the spacing is converted honestly (8 in = 203.2 mm, so slightly fewer bars), it drops to +58%. Either way, the line item is over by more than half and the slab is over-reinforced against a design that was checked for 10 mm.&lt;/p&gt;

&lt;p&gt;The reverse substitution is the one that gets people hurt: swapping #4 for 10 mm to "save steel" cuts the area by 39% on a section that was designed, checked and approved at 78.5 mm².&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 2: scaling a squared formula linearly
&lt;/h2&gt;

&lt;p&gt;The standard metric unit weight is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;w = 0.00617 × d²        (kg/m, d in millimetres)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It comes straight out of density and area:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;7850 kg/m³ × π/4 × (d/1000)² = 0.006165 d²
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exponent is the part people drop. Given a table entry for Φ10 at 0.617 kg/m, a Φ12 bar gets estimated as &lt;code&gt;0.617 × 12/10 = 0.740 kg/m&lt;/code&gt;. The correct value is 0.888 kg/m — the linear guess is &lt;strong&gt;16.7% low&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Worked example: 1,200 bars of Φ12, 6.0 m each
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Correct:  0.888 kg/m × 6.0 m × 1,200 = 6,394 kg = 6.39 t
Linear:   0.740 kg/m × 6.0 m × 1,200 = 5,328 kg = 5.33 t
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Short by 1.07 t on a 6.4 t package. That is not a rounding difference at the invoice; it is a second delivery, usually on a minimum-order charge, usually on the day the pour is scheduled.&lt;/p&gt;

&lt;p&gt;Two more variants of the same bug, both of which I have seen survive review:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Diameter in centimetres.&lt;/strong&gt; &lt;code&gt;0.00617 × 1.2² = 0.0089 kg/m&lt;/code&gt;. Off by a factor of 100. This one is almost always caught, because the tonnage collapses to something absurd.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Diameter in inches.&lt;/strong&gt; &lt;code&gt;0.00617 × 0.5² = 0.0015&lt;/code&gt;. Also caught, but only if someone sanity-checks against a known bar.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The squared term punishes interpolation too. Between Φ12 (0.888 kg/m) and Φ25 (3.856 kg/m), the midpoint of the weight column is 2.372. The actual Φ18 and Φ19 bars inside that gap weigh 2.00 and 2.23 kg/m. Averaging two neighbours overstates the middle of the range by roughly 12% — produced by the most reasonable-looking step in the whole process. If the diameter is missing from the table, square it; do not average the weights.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 3: tonne, ton, and the 10% that hides in a word
&lt;/h2&gt;

&lt;p&gt;Rebar is quoted per tonne in most of the world and per &lt;strong&gt;short ton&lt;/strong&gt; in the US. One letter apart:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 short ton  = 2,000 lb = 907.2 kg
1 metric tonne = 1,000 kg
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A 40-unit package invoiced in short tons delivers &lt;strong&gt;36.3 tonnes&lt;/strong&gt; of actual steel. On a 40 t order, 3.7 t is missing — roughly one floor's slab of a small building — and every document says "40".&lt;/p&gt;

&lt;p&gt;Concrete has the same shape of problem, in cubic yards:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;1 yd³ = 0.7646 m³      1 m³ = 1.3079 yd³
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;14.4 m³ of slab is 18.83 yd³. Rounding the factor to 0.75 gives 19.2 yd³ — 2% over, which is inside normal waste allowance and therefore invisible. Rounding to 1.3 instead of 1.3079 goes the other way and under-orders by 0.6%.&lt;/p&gt;

&lt;h2&gt;
  
  
  The conversion table worth taping to the wall
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Quantity&lt;/th&gt;
&lt;th&gt;The number people reuse&lt;/th&gt;
&lt;th&gt;What it actually is&lt;/th&gt;
&lt;th&gt;Error if used as-is&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;#4 unit weight&lt;/td&gt;
&lt;td&gt;0.668 (as kg/m)&lt;/td&gt;
&lt;td&gt;0.994 kg/m&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;−33%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;#3 unit weight&lt;/td&gt;
&lt;td&gt;0.376 (as kg/m)&lt;/td&gt;
&lt;td&gt;0.560 kg/m&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;−33%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1 lb/ft&lt;/td&gt;
&lt;td&gt;1.5 kg/m&lt;/td&gt;
&lt;td&gt;1.4882 kg/m&lt;/td&gt;
&lt;td&gt;+0.8%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1 short ton&lt;/td&gt;
&lt;td&gt;1,000 kg&lt;/td&gt;
&lt;td&gt;907.2 kg&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;+10%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1 yd³&lt;/td&gt;
&lt;td&gt;0.75 m³&lt;/td&gt;
&lt;td&gt;0.7646 m³&lt;/td&gt;
&lt;td&gt;−1.9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1 in&lt;/td&gt;
&lt;td&gt;25 mm&lt;/td&gt;
&lt;td&gt;25.4 mm&lt;/td&gt;
&lt;td&gt;−1.6%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Sort that column by magnitude and the priority is obvious. Two of these will blow a budget line. Three of them are noise you can ignore without losing sleep — and knowing which is which is most of the skill.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually catches these
&lt;/h2&gt;

&lt;p&gt;Three habits, none of them clever:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Carry units in the cell, not in your head.&lt;/strong&gt; &lt;code&gt;0.668 lb/ft&lt;/code&gt; and &lt;code&gt;0.617 kg/m&lt;/code&gt; are different objects. A schedule row that says only "0.668" invites the next reader to assume the unit they were already thinking about.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cross-check every weight against area, not against another weight.&lt;/strong&gt; &lt;code&gt;A = πd²/4&lt;/code&gt;, then &lt;code&gt;× 7850&lt;/code&gt;. If two numbers disagree, the area check tells you which one is wrong, because diameter is the one input you can measure with a caliper.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sanity-check the total against steel per cubic metre.&lt;/strong&gt; Ordinary reinforced concrete sits around 80–150 kg/m³. A slab that comes out at 40 or 260 is not a clever optimisation; it is a unit error.&lt;/p&gt;

&lt;p&gt;I keep the arithmetic in a calculator rather than in a spreadsheet cell, mostly so the unit stays attached to the input: &lt;a href="https://jianzhujisuan.com/en/rebar-weight?src=devto" rel="noopener noreferrer"&gt;rebar weight calculator&lt;/a&gt; takes diameter, length, count and waste allowance, and returns kg/m, per-bar weight and total tonnage. There are sibling pages for lap length, bend deduction and stirrup cutting length built the same way — one input per row, unit stated, formula printed on the page so the result can be checked by hand.&lt;/p&gt;

&lt;p&gt;None of this is difficult mathematics. It is bookkeeping, and bookkeeping fails quietly.&lt;/p&gt;

&lt;p&gt;The takeoff that started this post was off by two tonnes because two numbers eight percent apart were assumed to be the same number. They were never the same quantity. A unit is not a label you drop when the arithmetic gets going — it is the thing that decides whether 0.668 means a bar weighs 33% less than it does.&lt;/p&gt;

</description>
      <category>construction</category>
      <category>engineering</category>
      <category>math</category>
      <category>productivity</category>
    </item>
    <item>
      <title>A bot loaded my CSS, ran my JS, and still wasn't human</title>
      <dc:creator>Jianzhujisuan</dc:creator>
      <pubDate>Thu, 05 Nov 2026 02:42:26 +0000</pubDate>
      <link>https://dev.to/jianzhujisuan/a-bot-loaded-my-css-ran-my-js-and-still-wasnt-human-6el</link>
      <guid>https://dev.to/jianzhujisuan/a-bot-loaded-my-css-ran-my-js-and-still-wasnt-human-6el</guid>
      <description>&lt;p&gt;My daily report has one job: tell me how many actual people visited. Not requests, not hits — people.&lt;/p&gt;

&lt;p&gt;For a while it said five. I went to the raw logs to write a summary and found one of them had made &lt;strong&gt;1,743 requests to my origin&lt;/strong&gt; that day.&lt;/p&gt;

&lt;p&gt;Five page views. One thousand seven hundred and forty-three requests. A ratio of 350:1.&lt;/p&gt;

&lt;p&gt;That is not a person. That is a fetch loop with a Chrome user agent.&lt;/p&gt;

&lt;h2&gt;
  
  
  The detection stack, and where it failed
&lt;/h2&gt;

&lt;p&gt;My classifier has four layers, applied in order:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Signal&lt;/th&gt;
&lt;th&gt;What it catches&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;User agent string&lt;/td&gt;
&lt;td&gt;Declared crawlers, obvious scanners&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Render completeness (did it fetch CSS and JS?)&lt;/td&gt;
&lt;td&gt;Headless browsers, prefetchers, link previews&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;IP ownership (&lt;code&gt;hosting: true&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Cloud VMs, proxies, datacenter egress&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Request-to-pageview ratio&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Real browsers driven by something that isn't a person&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Layer 2 is the one I trusted most. The logic is sound on paper: a page that renders in the browser cannot show anything without its stylesheet and its calculation script. A crawler that only wants the HTML source never asks for them. So if an IP fetched both, something rendered the page, and rendering means a browser, and a browser means a person.&lt;/p&gt;

&lt;p&gt;Every step in that chain is wrong, and the last one is the dangerous one. &lt;strong&gt;Rendering proves a browser ran. It does not prove a human was watching.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Headless Chrome fetches CSS. So does Playwright. So does every screenshot service, every SEO audit tool, every "AI browsing" agent, and every scraper author who bothered to add &lt;code&gt;page.waitForLoadState('networkidle')&lt;/code&gt; because their selectors weren't working without it.&lt;/p&gt;

&lt;p&gt;Layer 2 tells you the page was executed. It says nothing about intent, dwell time, or how many times the same document got pulled.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the numbers actually looked like
&lt;/h2&gt;

&lt;p&gt;The IP resolved to a residential ISP in Jiangsu — consumer broadband, not a datacenter, so layer 3 waved it through. Full desktop Chrome user agent, so layer 1 waved it through. Fetched CSS and JS, so layer 2 waved it through.&lt;/p&gt;

&lt;p&gt;Then the edge logs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;requests:      1,743
pageviews:         5
ratio:          350:1
distinct pages:    5
window:        ~19 hours
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For comparison, here is what an ordinary visit looks like on the same site — every real visitor that day, measured the same way:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;requests:      4, 5, 10, 10, 17
pageviews:     2, 2,  2,  1,  1
ratio:         2:1, 2.5:1, 5:1, 10:1, 17:1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A person opening a calculator page pulls the document, one stylesheet, one script, maybe a favicon, maybe a share image. Ten to thirty requests for a one-page visit. Click through to a second page and it doubles. The whole population sits between 2:1 and 20:1.&lt;/p&gt;

&lt;p&gt;The outlier sat at 350:1, and its pageviews were five &lt;em&gt;distinct&lt;/em&gt; pages — so it wasn't one page reloaded 350 times. It was pulling every asset, tracking script, and API endpoint over and over, while only five documents were ever counted as views.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why pageview counts missed it
&lt;/h2&gt;

&lt;p&gt;Because pageviews are self-reported.&lt;/p&gt;

&lt;p&gt;The tracking script only fires if JavaScript runs &lt;em&gt;and&lt;/em&gt; the request reaches the analytics endpoint. A scraper that loads the page, extracts what it wants, and tears down the browser before &lt;code&gt;navigator.sendBeacon&lt;/code&gt; resolves produces zero pageviews — while still hammering the origin for every byte it could reach.&lt;/p&gt;

&lt;p&gt;So the two counters measure different things:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Edge requests&lt;/strong&gt; count what the network saw. Nothing can hide from this.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Pageviews&lt;/strong&gt; count what the client admitted to. Easy to skip, accidentally or otherwise.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Comparing them is the only cross-check that doesn't depend on the client's cooperation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: two thresholds, both required
&lt;/h2&gt;

&lt;p&gt;A single threshold doesn't work. High request counts alone are normal — corporate NAT puts forty people behind one IP, and forty people browsing genuinely produce a lot of requests. Low ratios alone are normal too, because everyone's ratio is low.&lt;/p&gt;

&lt;p&gt;What is abnormal is &lt;strong&gt;both at once&lt;/strong&gt;: enough absolute volume to rule out a quiet visitor, and a ratio high enough to rule out a busy shared connection.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;REQ_SUSPECT&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;300&lt;/span&gt;   &lt;span class="c1"&gt;# absolute floor: below this, shared NAT can look weird
&lt;/span&gt;&lt;span class="n"&gt;RATIO_SUSPECT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;    &lt;span class="c1"&gt;# ceiling on requests per pageview
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;looks_like_bulk&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reqs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pageviews&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;pv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pageviews&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;reqs&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;REQ_SUSPECT&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;reqs&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;pv&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;RATIO_SUSPECT&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both numbers came from the data rather than from taste:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;300&lt;/strong&gt; sits well above the observed maximum for a genuine multi-page session on this site (the busiest real visitor that day: 17 requests), and well below the outlier (1,743). It leaves room for a shared connection with several real people behind it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;60&lt;/strong&gt; is three times the highest ratio any confirmed human produced (20:1) and roughly a sixth of the outlier's. Anywhere in that gap works; 60 is the round number in the middle.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The function returns a boolean, and the caller routes the IP into the "automation" bucket — not deleted, not hidden, just not counted as a person. If the classification is ever wrong, the evidence is still in the report with a flag next to it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;183.34.x.x   1 pv   residential ISP   [1743 edge requests, suspected bulk]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That line matters more than the count. A number I can audit beats a number I can trust.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it changed
&lt;/h2&gt;

&lt;p&gt;One IP moved from "confirmed human" to "automation." The day's human count went from 5 to 4 — a 20% overstatement, on a day with a sample size of five.&lt;/p&gt;

&lt;p&gt;Small sample, large relative error. That is the whole problem with low-traffic analytics: a single misclassified IP is a fifth of your data. You cannot average it away, so the classifier has to be right, and "right" has to mean "conservative and auditable" rather than "clever."&lt;/p&gt;

&lt;p&gt;I would rather undercount humans by one than report a bot as a reader. Undercounting costs me nothing I can act on. Overcounting produces decisions — "traffic is up, keep doing this" — based on nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this check is wrong
&lt;/h2&gt;

&lt;p&gt;Two honest failure modes:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cloudflare Workers and edge functions.&lt;/strong&gt; An IP running a Worker that proxies user requests will show elevated request counts against its own pageviews. If you proxy your own analytics, exclude your own egress ranges explicitly rather than relying on the ratio.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Link-preview generators and screenshot bots.&lt;/strong&gt; They fetch HTML, CSS and JS, produce one request burst, and often land under 300 requests. Ratio check misses them; the ratio is low because they only did it once. Those still get caught by the render-completeness heuristic, so keep both — they cover different failure modes and neither is sufficient alone.&lt;/p&gt;

&lt;p&gt;And one thing this absolutely cannot do: distinguish a determined scraper that deliberately spaces its requests and stays under every threshold. That is not a detection problem, that is an arms race, and it is not worth fighting on a site with four visitors a day.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three things I'd tell someone building this
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Never trust a client-reported metric as your denominator without a server-side cross-check.&lt;/strong&gt; Pageviews, sessions, scroll depth — all of it is a claim, not an observation. The edge log is the observation.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Fingerprints degrade; behavior ratios don't.&lt;/strong&gt; User agents get spoofed in one line. Headless detection gets patched weekly. The relationship between two independent counters stays meaningful as long as they measure different things.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Set thresholds from your own distribution, and write down what produced them.&lt;/strong&gt; "300 because the busiest human did 17" is a decision someone can audit and override. "300 because it felt right" is a magic number, and magic numbers are how you end up with a coefficient you can't explain six months later.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The site this runs on is &lt;a href="https://jianzhujisuan.com?src=devto" rel="noopener noreferrer"&gt;jianzhujisuan.com&lt;/a&gt; — 382 static construction calculators, single developer, which is exactly why analytics noise hurts: there is no traffic to hide it behind.&lt;/p&gt;

&lt;p&gt;If your dashboard says you had five visitors yesterday, check the request counts before you believe it. One of them might be a browser with no one inside.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>analytics</category>
      <category>cloudflare</category>
      <category>debugging</category>
    </item>
    <item>
      <title>1,713 404s in three days — and roughly 70% weren't my site's fault</title>
      <dc:creator>Jianzhujisuan</dc:creator>
      <pubDate>Sat, 31 Oct 2026 07:57:43 +0000</pubDate>
      <link>https://dev.to/jianzhujisuan/1713-404s-in-three-days-and-roughly-70-werent-my-sites-fault-2o55</link>
      <guid>https://dev.to/jianzhujisuan/1713-404s-in-three-days-and-roughly-70-werent-my-sites-fault-2o55</guid>
      <description>&lt;p&gt;I opened the Cloudflare logs on a Tuesday and saw 1,713 404s over three days, spread across 200 distinct paths.&lt;/p&gt;

&lt;p&gt;My site has 382 pages. Every one of them returns 200 when I check it. So where were these 404s coming from?&lt;/p&gt;

&lt;p&gt;The top path was &lt;code&gt;/wp-admin/install.php&lt;/code&gt;, hit 226 times. I have never installed WordPress. There is no PHP on this site at all — it is a static build on Cloudflare Pages.&lt;/p&gt;

&lt;p&gt;That was the moment I stopped treating "number of 404s" as a health metric.&lt;/p&gt;

&lt;h2&gt;
  
  
  The list looks like a disaster until you cluster it
&lt;/h2&gt;

&lt;p&gt;Here are the actual top paths from those three days, grouped by what they are:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Family&lt;/th&gt;
&lt;th&gt;Example paths&lt;/th&gt;
&lt;th&gt;What it is&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;WordPress probes&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;/wp-admin/install.php&lt;/code&gt;, &lt;code&gt;/wp-includes/wlwmanifest.xml&lt;/code&gt;, &lt;code&gt;/wp-json/wp/v2/users&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;A bot checking whether you run WordPress&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Config leaks&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;/.env&lt;/code&gt;, &lt;code&gt;/.git/config&lt;/code&gt;, &lt;code&gt;/.aws/credentials&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;A bot looking for credentials you left in the web root&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Backups&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;/backup.zip&lt;/code&gt;, &lt;code&gt;/db.sql&lt;/code&gt;, &lt;code&gt;/www.zip&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Same bot, guessing filenames&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Admin panels&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;/admin.php&lt;/code&gt;, &lt;code&gt;/phpmyadmin/&lt;/code&gt;, &lt;code&gt;/manager/html&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Same bot, different target&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Real traffic&lt;/td&gt;
&lt;td&gt;a short list&lt;/td&gt;
&lt;td&gt;Things a human actually asked for&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Roughly seven in ten of the 1,713 requests belonged to the first four families. None of them can ever succeed on a static site, and none of them indicate a problem with the site. They are a background radiation level that every public host sees.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The first rule: sort before you count.&lt;/strong&gt; Group by path family, not by path. 200 alarming rows collapse into a list short enough to read line by line.&lt;/p&gt;

&lt;h2&gt;
  
  
  The second trap: crawler user-agents are trivially spoofed
&lt;/h2&gt;

&lt;p&gt;While I was in there I filtered for &lt;code&gt;Googlebot&lt;/code&gt; and got a list that included a scan from a residential IP hitting &lt;code&gt;/.env&lt;/code&gt;. That is not Google.&lt;/p&gt;

&lt;p&gt;So I stopped filtering on user-agent alone and matched the UA against Google's published IP ranges:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;GOOGLE_RANGES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;66.249.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;64.233.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;72.14.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                 &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;74.125.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;209.85.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;is_real_googlebot&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ip&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Googlebot&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;ua&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;ip&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;GOOGLE_RANGES&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The result over one week:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Real&lt;/strong&gt; Googlebot: 3,446 requests, 400 distinct content pages out of 382 — it re-crawls more than the whole site, multiple times.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fake&lt;/strong&gt; Googlebot UA: 112 requests, all from IPs outside those ranges.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Same story for Bing, which was the surprise. My SEO dashboard said Bing was crawling almost nothing. When I filtered correctly, bingbot really had made 36 requests to 6 pages. The other 17 "bingbot" hits were scanners wearing the UA.&lt;/p&gt;

&lt;p&gt;Two things came out of that exercise:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Never trust a UA string on its own.&lt;/strong&gt; A "Googlebot" hit from a residential IP is a scanner, and if you act on it you will spend a weekend fixing a bug that does not exist.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bing's mobile UA shares the same prefix as Google's mobile UA.&lt;/strong&gt; They only diverge at the trailing &lt;code&gt;bingbot/2.0&lt;/code&gt;. If you match with &lt;code&gt;startswith("Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X")&lt;/code&gt; you will silently count Google's phone crawler as Bing and conclude that Bing loves you.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;One more noise source worth naming: a pile of 504s in my logs all carried &lt;code&gt;User-Agent: *early hints*&lt;/code&gt;. That is not a timeout or an overloaded origin. It is a browser opening an HTTP/1.0 preconnect to warm the connection for a 103 Early Hints response. It never expects a body. If you alert on 504 rate without excluding that UA, you will page yourself at 3am over a browser being efficient.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the remaining ~30% actually was
&lt;/h2&gt;

&lt;p&gt;After removing scanner traffic and spoofed crawlers, what was left fit on one screen. Three problems were real.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Malformed URLs from chat apps.&lt;/strong&gt; People copy text out of WeChat or Slack and paste it into the address bar. Chinese full-width parentheses in a sentence next to a link mean the parenthetical gets swallowed into the URL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/zh/service（英文版        → 404
/zh/service                → 200
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The link was fine. My formatting was the bug — I had written the explanation on the same line as the URL, so copying the line copied the explanation too.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bare paths with no locale prefix.&lt;/strong&gt; Old bookmarks and shared links arrive as &lt;code&gt;/contact&lt;/code&gt; instead of &lt;code&gt;/zh/service&lt;/code&gt;. Six one-line 301s fixed all of them, and they were worth adding because those are links real people saved.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;/favicon.ico&lt;/code&gt; at the domain root.&lt;/strong&gt; Browsers request it automatically even when your HTML declares an icon elsewhere. Empty root, 404, about twelve times a day. A 32×32 icon generated with nothing but &lt;code&gt;struct&lt;/code&gt; (no Pillow on this machine) made it a non-issue.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I actually changed — and what I deliberately did not
&lt;/h2&gt;

&lt;p&gt;I did &lt;strong&gt;not&lt;/strong&gt; add redirects for &lt;code&gt;/wp-admin/install.php&lt;/code&gt;. Redirecting scanner probes to your homepage is a small, free gift: it confirms the site is live and tells the bot its request pattern reached something. A plain 404 is cheaper.&lt;/p&gt;

&lt;p&gt;I did add a client-side correction on the 404 page itself, because malformed URLs from chat clients are a real human path:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;location&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;pathname&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;match&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="se"&gt;\/(&lt;/span&gt;&lt;span class="sr"&gt;zh|en&lt;/span&gt;&lt;span class="se"&gt;)\/([&lt;/span&gt;&lt;span class="sr"&gt;a-z0-9-&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;+&lt;/span&gt;&lt;span class="se"&gt;)&lt;/span&gt;&lt;span class="sr"&gt;/&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="nx"&gt;slug&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
  &lt;span class="c1"&gt;// if slug is a real page, send them there instead of showing 404&lt;/span&gt;
  &lt;span class="nx"&gt;location&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This recovers the &lt;code&gt;/zh/service（英文版&lt;/code&gt; case without me knowing in advance which punctuation a given chat app will append.&lt;/p&gt;

&lt;p&gt;And I verified the whole site the boring way, because spot checks lie:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-s&lt;/span&gt; https://example.com/sitemap.xml &lt;span class="se"&gt;\&lt;/span&gt;
  | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="s1"&gt;'&amp;lt;loc&amp;gt;[^&amp;lt;]*'&lt;/span&gt; | &lt;span class="nb"&gt;sed&lt;/span&gt; &lt;span class="s1"&gt;'s/&amp;lt;loc&amp;gt;//'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  | xargs &lt;span class="nt"&gt;-P12&lt;/span&gt; &lt;span class="nt"&gt;-I&lt;/span&gt;&lt;span class="o"&gt;{}&lt;/span&gt; curl &lt;span class="nt"&gt;-s&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; /dev/null &lt;span class="nt"&gt;-w&lt;/span&gt; &lt;span class="s2"&gt;"%{http_code} {}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;{}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;382 URLs, all 200, zero surprises. That one command is worth more than any dashboard, because it tests what the crawler sees rather than what I think it sees.&lt;/p&gt;

&lt;p&gt;The site is &lt;a href="https://jianzhujisuan.com?src=devto" rel="noopener noreferrer"&gt;jianzhujisuan.com&lt;/a&gt; — a free set of construction calculators, 382 pages, all static, all rendered in the browser.&lt;/p&gt;

&lt;h2&gt;
  
  
  The triage order I now follow
&lt;/h2&gt;

&lt;p&gt;Before touching a single redirect:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Cluster by path family.&lt;/strong&gt; WP probes, config leaks, backup guesses, admin panels. Anything in those buckets is background radiation — ignore it permanently.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Validate crawler UAs against published IP ranges.&lt;/strong&gt; Separate real bots from scanners before you draw any conclusion about crawl coverage. A spoofed UA can make a healthy crawl look broken, or a broken one look healthy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Only then look at what's left.&lt;/strong&gt; What remains is usually short, and usually falls into: links copied out of chat apps, URLs missing a locale prefix, and things the browser requests on its own.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The number that matters is not "how many 404s". It is "how many distinct human-intent paths 404'd". On this site, over three days, that number was three. The other 1,710 were the internet doing its usual sweep of every host on the planet.&lt;/p&gt;

&lt;p&gt;If you are sitting on a scary-looking 404 report right now, the odds are good that most of it is not yours to fix. Sort first. Then fix the six.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>seo</category>
      <category>cloudflare</category>
      <category>debugging</category>
    </item>
    <item>
      <title>Googlebot ran my analytics script — my unique visitor count was 14x too high</title>
      <dc:creator>Jianzhujisuan</dc:creator>
      <pubDate>Fri, 30 Oct 2026 04:37:18 +0000</pubDate>
      <link>https://dev.to/jianzhujisuan/googlebot-ran-my-analytics-script-my-unique-visitor-count-was-14x-too-high-13am</link>
      <guid>https://dev.to/jianzhujisuan/googlebot-ran-my-analytics-script-my-unique-visitor-count-was-14x-too-high-13am</guid>
      <description>&lt;p&gt;For three weeks, my site's dashboard told a nice story. Traffic was growing. Unique visitors hit 87 on the best day, then 94, then a hundred-odd. New site, indexing starting to happen, numbers going up — exactly what you want to see.&lt;/p&gt;

&lt;p&gt;Then I looked at the actual requests and found &lt;strong&gt;six humans&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Not six hundred. Six. And two of those six were probably me.&lt;/p&gt;

&lt;p&gt;This post is about where the other 81 came from, why the usual bot filters miss them, and the four checks that finally made the number honest. If you run a small site with a hand-rolled analytics endpoint, you have the same problem — you just don't know it yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  The mechanism: rendered pages run JavaScript
&lt;/h2&gt;

&lt;p&gt;I built the analytics myself. A tiny script posts to &lt;code&gt;/api/pv&lt;/code&gt; on page load; a Cloudflare Worker writes a row into D1 with path, referrer, UA, and IP. Fifty lines, no vendor, no cookie banner.&lt;/p&gt;

&lt;p&gt;That design assumes a visit is a person. It isn't.&lt;/p&gt;

&lt;p&gt;Googlebot fetches a page, renders it in headless Chromium, and executes the JavaScript — my beacon included. So &lt;strong&gt;every page Google crawls generates a "visit"&lt;/strong&gt;. On a 380-page site, one crawl pass is up to 380 hits. And because crawlers don't reuse a connection the way a browser session does, each batch can arrive from a different egress IP, so &lt;code&gt;COUNT(DISTINCT ip)&lt;/code&gt; treats them as hundreds of separate people.&lt;/p&gt;

&lt;p&gt;That's the whole illusion. Nothing was broken. My endpoint correctly recorded what happened. It just recorded the wrong thing as a visitor.&lt;/p&gt;

&lt;p&gt;Worth knowing the scale before you go hunting: on my site, Googlebot had crawled 372 of 380 pages, while bingbot had crawled 5. Both are "search engine traffic" in a naive dashboard. One of them is 70x the other, and neither is a person.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fix attempt #1: filter on user agent (insufficient)
&lt;/h2&gt;

&lt;p&gt;The obvious move — classify at ingest time, store the result, and never reinterpret it later:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;UA_BOT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sr"&gt;/bot|crawl|spider|slurp|headless|phantom|puppeteer|playwright|selenium|curl&lt;/span&gt;&lt;span class="se"&gt;\/&lt;/span&gt;&lt;span class="sr"&gt;|wget|python-|go-http-client|java&lt;/span&gt;&lt;span class="se"&gt;\/&lt;/span&gt;&lt;span class="sr"&gt;|okhttp|axios|node-fetch|scrapy|semrush|ahrefs|dataforseo|lighthouse/i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;classifyUA&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ua&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ua&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;?&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;UA_BOT&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;bot&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/ipad|tablet|silk&lt;/span&gt;&lt;span class="se"&gt;\/&lt;/span&gt;&lt;span class="sr"&gt;/i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;tb&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/android|iphone|mobile/i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;mb&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;pc&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This caught the honest bots immediately — &lt;code&gt;Googlebot&lt;/code&gt;, &lt;code&gt;bingbot&lt;/code&gt;, &lt;code&gt;AhrefsBot&lt;/code&gt;. It also caught a few things I hadn't thought of:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;GoogleOther&lt;/code&gt;, &lt;code&gt;Google-InspectionTool&lt;/code&gt;, &lt;code&gt;APIs-Google&lt;/code&gt;, &lt;code&gt;FeedFetcher&lt;/code&gt; — Google's non-search fetchers. A regex containing only &lt;code&gt;bot&lt;/code&gt; misses all of them.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;BuiltWith&lt;/code&gt;, &lt;code&gt;DataForSEO&lt;/code&gt;, &lt;code&gt;Lighthouse&lt;/code&gt; — audit tools that behave exactly like a crawler.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;HeadlessChrome&lt;/code&gt; — the giveaway inside otherwise-normal Chrome UAs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The &lt;code&gt;mediapartners&lt;/code&gt; and &lt;code&gt;google-inspectiontool&lt;/code&gt; entries were the ones that actually moved my numbers. Add them even if you've never heard of them.&lt;/p&gt;

&lt;p&gt;But UA filtering alone fails on two cases, and both were present on my site.&lt;/p&gt;

&lt;h2&gt;
  
  
  The case that fooled me three times: fake WeChat traffic
&lt;/h2&gt;

&lt;p&gt;I kept seeing a cluster of "WeChat" visitors. Mobile, from Tencent Cloud IP ranges, hundreds of hits. On a Chinese-language engineering site, WeChat referrals are the single most plausible traffic source, so I believed it — three separate times, over three weeks, before I stopped believing it.&lt;/p&gt;

&lt;p&gt;What finally exposed it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Mozilla/5.0 (Linux; Android 5.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/74.0.3729.131 Mobile Safari/537.36
MicroMessenger/7.0.1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three independent tells:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;MicroMessenger/7.0.1&lt;/code&gt;&lt;/strong&gt; — that's a 2018 client. Real WeChat traffic in 2026 reports &lt;code&gt;8.x&lt;/code&gt; and carries a build identifier like &lt;code&gt;(0x28002d3a)&lt;/code&gt;. A version with no &lt;code&gt;(0x&lt;/code&gt; segment is not the real client.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;Nexus 5 Build/MRA58N&lt;/code&gt;&lt;/strong&gt; — that is Puppeteer's default device descriptor, straight out of the library's device list. Real Android devices in the wild report their own build string.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tencent Cloud IP ranges&lt;/strong&gt; — &lt;code&gt;81.71.&lt;/code&gt;, &lt;code&gt;183.60.&lt;/code&gt;, &lt;code&gt;42.194.&lt;/code&gt;, &lt;code&gt;14.152.&lt;/code&gt;, &lt;code&gt;43.248.&lt;/code&gt; are datacenter, not mobile carriers.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A single tell is weak. Three together is conclusive. The lesson generalises: &lt;strong&gt;a UA string is a claim, not evidence.&lt;/strong&gt; Check whether the claim is internally consistent before you count it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The second blind spot: crawlers wearing a normal Chrome UA
&lt;/h2&gt;

&lt;p&gt;The other failure mode is worse, because the UA looks perfect. I had a stream of &lt;code&gt;Macintosh; Intel Mac OS X&lt;/code&gt; Safari UAs from &lt;code&gt;17.166.*&lt;/code&gt; — roughly two requests per page, and 159 distinct IPs in a single day. Nothing in the UA string says "bot".&lt;/p&gt;

&lt;p&gt;The behaviour does:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;exactly two requests per page&lt;/strong&gt; (HTML + one asset), never the full asset set&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;no referrer, ever&lt;/strong&gt;, and no session continuity&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;159 IPs, all on the same cloud range&lt;/strong&gt;, all doing the same thing within hours&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;zero interaction&lt;/strong&gt; — no search query, no second page view, no scroll-triggered event&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's the profile of a distributed fetcher, not of people. Two requests per page means "I parsed your HTML", not "I read your page".&lt;/p&gt;

&lt;p&gt;Here's the uncomfortable part: &lt;strong&gt;rendering completeness is not a reliable human signal any more.&lt;/strong&gt; Googlebot loads CSS, executes JS, and would happily pass a "did they load the assets" check. I used to treat "loaded everything" as proof of humanity. It isn't. It only rules out the lazy crawlers.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually worked: four layers, in this order
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Classify at write time, store the verdict.&lt;/strong&gt; A &lt;code&gt;dev&lt;/code&gt; column (&lt;code&gt;bot&lt;/code&gt; / &lt;code&gt;tb&lt;/code&gt; / &lt;code&gt;mb&lt;/code&gt; / &lt;code&gt;pc&lt;/code&gt;) written by the Worker. Doing this at query time means every new bot name invalidates three months of history.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Named crawlers get IP verification, not just UA matching.&lt;/strong&gt; Google publishes its ranges; match &lt;code&gt;66.249.&lt;/code&gt;, &lt;code&gt;64.233.&lt;/code&gt;, &lt;code&gt;72.14.&lt;/code&gt;, &lt;code&gt;74.125.&lt;/code&gt;, &lt;code&gt;209.85.&lt;/code&gt; alongside the UA. This also removes the impostors: I had a &lt;code&gt;bingbot&lt;/code&gt; UA from &lt;code&gt;209.112.91.75&lt;/code&gt;, which is a vulnerability scanner, not Bing. Real Bing crawls from &lt;code&gt;40.77.167.&lt;/code&gt;, &lt;code&gt;52.167.144.&lt;/code&gt;, &lt;code&gt;207.46.13.&lt;/code&gt;, &lt;code&gt;157.55.39.&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Consistency checks on the claim.&lt;/strong&gt; WeChat version vs build number vs IP ownership. Datacenter IP claiming to be a phone is a bot, no matter what the UA says.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Request-shape checks for the rest.&lt;/strong&gt; Assets loaded, referrer present, page-depth, inter-request timing. Any one alone is weak; three together are decisive.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Then every downstream number gets a single filter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;DISTINCT&lt;/span&gt; &lt;span class="n"&gt;ip&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;humans&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;hits&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;hits&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;COALESCE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'bot'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="nb"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'+8 hours'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;date&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'now'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'+8 hours'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That &lt;code&gt;COALESCE(dev, '') &amp;lt;&amp;gt; 'bot'&lt;/code&gt; has to be in &lt;strong&gt;every&lt;/strong&gt; query — dashboard, admin panel, the JSON endpoint the frontend calls. I missed it in two places and got a "traffic surged" alert about my own crawler.&lt;/p&gt;

&lt;p&gt;One more trap in the same query: my timestamps are stored in UTC. Comparing "today so far" against "yesterday's full day" made the dashboard report &lt;strong&gt;-93%&lt;/strong&gt; on a completely normal morning. Six visits by 11:35 versus thirty-eight across the whole previous day isn't a decline, it's arithmetic. If your period comparison can produce a number that large, the bug is in the comparison, not the site.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the honest number looks like
&lt;/h2&gt;

&lt;p&gt;After filtering, on a 380-page site with 168 bilingual calculators:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Before&lt;/th&gt;
&lt;th&gt;After&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Best-day unique visitors&lt;/td&gt;
&lt;td&gt;87&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Traffic source&lt;/td&gt;
&lt;td&gt;"growing"&lt;/td&gt;
&lt;td&gt;1 real organic click&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Crawl coverage (Googlebot)&lt;/td&gt;
&lt;td&gt;unknown&lt;/td&gt;
&lt;td&gt;372 / 380 pages&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Crawl coverage (bingbot)&lt;/td&gt;
&lt;td&gt;unknown&lt;/td&gt;
&lt;td&gt;5 / 380 pages&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The last row is the one that changed my priorities. With bots stripped out, I could finally see that Google was indexing aggressively and Bing essentially wasn't crawling at all — a fact completely invisible in the inflated dashboard, where both were just "traffic".&lt;/p&gt;

&lt;p&gt;You can't diagnose a funnel you can't measure. My dashboard was measuring machine labour and calling it demand.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaway
&lt;/h2&gt;

&lt;p&gt;If your analytics are hand-rolled, assume they're counting crawlers until you've proven otherwise. The cheap version of the proof:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Log the UA and IP at write time — you can't retroactively classify what you didn't record.&lt;/li&gt;
&lt;li&gt;Treat every UA as a claim, then check it against IP ownership and request shape.&lt;/li&gt;
&lt;li&gt;Add &lt;code&gt;&amp;lt;&amp;gt; 'bot'&lt;/code&gt; to every query, not just the pretty one.&lt;/li&gt;
&lt;li&gt;Compare like periods, in the same timezone, or your percentages will lie to you.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The site in question is &lt;a href="https://jianzhujisuan.com?src=devto" rel="noopener noreferrer"&gt;jianzhujisuan.com&lt;/a&gt; — 168 construction-material calculators, 380 static pages, no framework. If you want to see what the crawling looks like from the other side, pick a page and view source; the result table is already in the HTML, which is why crawlers execute it and why my visitor count was fiction for three weeks.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>analytics</category>
      <category>javascript</category>
      <category>cloudflare</category>
    </item>
    <item>
      <title>Your sitemap's lastmod is lying — one CSS tweak marked all 380 pages as updated</title>
      <dc:creator>Jianzhujisuan</dc:creator>
      <pubDate>Thu, 29 Oct 2026 17:00:23 +0000</pubDate>
      <link>https://dev.to/jianzhujisuan/your-sitemaps-lastmod-is-lying-one-css-tweak-marked-all-380-pages-as-updated-4pdi</link>
      <guid>https://dev.to/jianzhujisuan/your-sitemaps-lastmod-is-lying-one-css-tweak-marked-all-380-pages-as-updated-4pdi</guid>
      <description>&lt;p&gt;Last winter I added one line to a stylesheet on my site — &lt;code&gt;letter-spacing: -0.01em&lt;/code&gt; on a heading — and redeployed. Nothing else changed. And yet, according to my sitemap, &lt;strong&gt;all 380 pages had been updated that day&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;bingbot, being reasonable, went off to recrawl 380 pages. Almost none of them had changed.&lt;/p&gt;

&lt;p&gt;This is the quiet way a static site wastes its crawl budget: not through broken links or slow responses, but through a &lt;code&gt;lastmod&lt;/code&gt; that lies.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why &lt;code&gt;lastmod&lt;/code&gt; matters more than people think
&lt;/h2&gt;

&lt;p&gt;Google has said for years that it mostly ignores &lt;code&gt;&amp;lt;lastmod&amp;gt;&lt;/code&gt; unless it can trust it. Bing is blunter about it: crawl scheduling uses change history per URL. If your sitemap claims everything changed today, every day, then either:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the crawler recrawls a lot of unchanged pages, or&lt;/li&gt;
&lt;li&gt;it learns your &lt;code&gt;lastmod&lt;/code&gt; is noise and stops using it — so genuinely new pages wait behind nothing.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Either outcome is bad, and both are invisible in your analytics because nothing errors. You just notice, months later, that a batch of pages never got indexed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The naive fix, and why it backfires
&lt;/h2&gt;

&lt;p&gt;The obvious improvement over file mtimes is to hash the &lt;em&gt;rendered output&lt;/em&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;lastmod_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;digest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha1&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;rb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setdefault&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;today&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;digest&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's roughly what I shipped first. It's correct in principle and useless in practice, because the bytes of a rendered page are not stable even when the content is.&lt;/p&gt;

&lt;p&gt;Two things move on every single deploy:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Cache-busting query strings.&lt;/strong&gt; My asset references look like &lt;code&gt;/assets/style.css?v=3f9a1c&lt;/code&gt;. The build rewrites that hash whenever the stylesheet changes. So a one-line CSS edit changes the bytes of &lt;em&gt;every page that links the stylesheet&lt;/em&gt; — which is all of them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. The analytics beacon.&lt;/strong&gt; A small script tag with a site token gets injected into &lt;code&gt;&amp;lt;head&amp;gt;&lt;/code&gt;. Harmless, identical on every page — but if the injection order or whitespace ever shifts (a template refactor, a new minifier version, a different Python release writing attributes in a different order), the hash changes for the whole site at once.&lt;/p&gt;

&lt;p&gt;Neither is a content change. Both are pure output noise. A hash of the raw file can't tell the difference.&lt;/p&gt;

&lt;h2&gt;
  
  
  Hash the meaning, not the bytes
&lt;/h2&gt;

&lt;p&gt;The fix is to normalise before hashing — strip the parts that are allowed to vary without the page meaning anything different:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;

&lt;span class="n"&gt;NOISE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
    &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;\?v=[0-9a-zA-Z]+&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;                 &lt;span class="c1"&gt;# cache-busting hashes
&lt;/span&gt;    &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;script[^&amp;gt;]*data-analytics[^&amp;gt;]*&amp;gt;.*?&amp;lt;/script&amp;gt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;\s+&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;                              &lt;span class="c1"&gt;# whitespace
&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fingerprint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;pat&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;NOISE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;html&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;pat&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;hashlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sha1&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;utf-8&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;hexdigest&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then keep a state file mapping path → fingerprint, and only touch &lt;code&gt;lastmod&lt;/code&gt; when the fingerprint moves:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;state&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;_lastmod.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;changed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;built_pages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;fp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;fingerprint&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{}).&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;fp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fp&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;lastmod&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;today&lt;/span&gt;&lt;span class="p"&gt;()}&lt;/span&gt;
        &lt;span class="n"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dump&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;_lastmod.json&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;w&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; pages changed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;lastmod&lt;/code&gt; values in the sitemap now come from that state file, not from the clock.&lt;/p&gt;

&lt;h2&gt;
  
  
  The test that keeps it honest
&lt;/h2&gt;

&lt;p&gt;A fingerprint system is easy to write and easy to break silently. The assertion I run in CI is deliberately dumb:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Build twice in a row. The second build must report zero changed pages.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;If it reports one page, something in the pipeline is non-deterministic. If it reports 380, the normaliser stopped working — usually because someone added a new injected block and didn't add it to &lt;code&gt;NOISE&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That single assertion has caught three real regressions for me:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;a minifier that started emitting a random class-name suffix&lt;/li&gt;
&lt;li&gt;an analytics snippet that included a build timestamp&lt;/li&gt;
&lt;li&gt;a Python upgrade that changed attribute order in the HTML writer&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;All three would have shipped an "everything changed" sitemap, and none of them would have raised anything in a normal test suite, because the pages were &lt;em&gt;correct&lt;/em&gt;. They were just dishonest about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the state file lives
&lt;/h2&gt;

&lt;p&gt;One trap worth calling out: if &lt;code&gt;_lastmod.json&lt;/code&gt; lives only in CI and the cache is wiped, every page looks new again — the failure mode you were trying to fix.&lt;/p&gt;

&lt;p&gt;Two options that work:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;commit it to the repo (small, boring, effective — 380 lines of JSON for my site)&lt;/li&gt;
&lt;li&gt;store it as a build artefact with a long retention and restore it before building&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What does &lt;em&gt;not&lt;/em&gt; work: recomputing it from git history. Git tells you when a source file changed, not when a rendered page's output changed — and one shared layout file touches all 380 outputs.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually changed
&lt;/h2&gt;

&lt;p&gt;On my site, &lt;a href="https://jianzhujisuan.com?src=devto" rel="noopener noreferrer"&gt;jianzhujisuan.com&lt;/a&gt; — 168 bilingual engineering calculators, 380 static pages — the effect was immediate and measurable in the logs rather than in rankings:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;deploys that change one article now mark &lt;strong&gt;1&lt;/strong&gt; page as updated instead of 380&lt;/li&gt;
&lt;li&gt;the second build in any CI run reports &lt;strong&gt;0&lt;/strong&gt; changed pages&lt;/li&gt;
&lt;li&gt;crawlers spend their budget on the pages that actually moved&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No ranking jump. No dramatic story. Just a sitemap that stopped lying, which is what &lt;code&gt;lastmod&lt;/code&gt; was supposed to be all along.&lt;/p&gt;

&lt;p&gt;If you want to see the output side of this, open any tool page and view source — say &lt;a href="https://jianzhujisuan.com/en/brick-count?src=devto" rel="noopener noreferrer"&gt;the brick count calculator&lt;/a&gt; — the result table is in the HTML, with a fingerprint attribute that's the same on every build until the formula changes.&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>javascript</category>
      <category>performance</category>
    </item>
    <item>
      <title>Your calculator returns nothing to crawlers. Here's how I fixed it with Playwright.</title>
      <dc:creator>Jianzhujisuan</dc:creator>
      <pubDate>Thu, 29 Oct 2026 16:19:41 +0000</pubDate>
      <link>https://dev.to/jianzhujisuan/your-calculator-returns-nothing-to-crawlers-heres-how-i-fixed-it-with-playwright-108o</link>
      <guid>https://dev.to/jianzhujisuan/your-calculator-returns-nothing-to-crawlers-heres-how-i-fixed-it-with-playwright-108o</guid>
      <description>&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;If your site is "user types numbers → JavaScript computes a result", your HTML source probably looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;div&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"result"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&amp;lt;/div&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;script &lt;/span&gt;&lt;span class="na"&gt;src=&lt;/span&gt;&lt;span class="s"&gt;"calc.js"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&amp;lt;/script&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Users see nothing wrong. &lt;strong&gt;Crawlers download 30KB of HTML containing no answer at all.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Googlebot does execute JavaScript, but only when it decides the page is worth rendering — and the render queue can lag days to weeks. &lt;strong&gt;bingbot and most other crawlers do not execute JavaScript at all.&lt;/strong&gt; They get the blank div, index it, and your page competes for "concrete volume calculator" with zero matching text.&lt;/p&gt;

&lt;p&gt;I hit this building &lt;a href="https://jianzhujisuan.com?src=devto" rel="noopener noreferrer"&gt;jianzhujisuan.com&lt;/a&gt; — 168 construction calculators, 380 static pages, every result computed in the browser.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: don't solve it at runtime, solve it at build time
&lt;/h2&gt;

&lt;p&gt;One extra build step: open each page in a real browser with default inputs, wait for the JS to finish, capture &lt;code&gt;#result&lt;/code&gt;'s innerHTML, write it back into the same container in the HTML file.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;chromium&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;playwright&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;prerender&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;file&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;newPage&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;goto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;file://&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;file&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;        &lt;span class="c1"&gt;// local file, no server needed&lt;/span&gt;
  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;waitForSelector&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;#result&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="c1"&gt;// Note the (expr)() — see trap 1&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;inner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`(&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;config&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;capture&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;)()`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="nf"&gt;injectIntoContainer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;file&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;inner&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;         &lt;span class="c1"&gt;// into the container, see trap 2&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight html"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;div&lt;/span&gt; &lt;span class="na"&gt;id=&lt;/span&gt;&lt;span class="s"&gt;"result"&lt;/span&gt;&lt;span class="nt"&gt;&amp;gt;&lt;/span&gt;
  &lt;span class="c"&gt;&amp;lt;!-- prerender:start fp=a3f8c91b2e4d7a60 --&amp;gt;&lt;/span&gt;
  &lt;span class="nt"&gt;&amp;lt;table&amp;gt;&amp;lt;tr&amp;gt;&amp;lt;td&amp;gt;&lt;/span&gt;Concrete volume&lt;span class="nt"&gt;&amp;lt;/td&amp;gt;&amp;lt;td&amp;gt;&lt;/span&gt;3.24 m³&lt;span class="nt"&gt;&amp;lt;/td&amp;gt;&amp;lt;/tr&amp;gt;&amp;lt;/table&amp;gt;&lt;/span&gt;
  &lt;span class="c"&gt;&amp;lt;!-- prerender:end --&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;/div&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;For visitors&lt;/strong&gt;: &lt;code&gt;calc.js&lt;/code&gt; loads and overwrites &lt;code&gt;#result&lt;/code&gt;'s innerHTML — the static copy disappears, nothing changes for them.&lt;br&gt;
&lt;strong&gt;For crawlers&lt;/strong&gt;: no JS execution, but the table is right there in the source.&lt;/p&gt;

&lt;p&gt;Single acceptance test — open the page with &lt;code&gt;javaScriptEnabled: false&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;browser&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;newContext&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;javaScriptEnabled&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;newPage&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;goto&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;textContent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;#result&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;  &lt;span class="c1"&gt;// must contain real numbers&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Trap 1: &lt;code&gt;page.evaluate("() =&amp;gt; x")&lt;/code&gt; evaluates, doesn't call
&lt;/h2&gt;

&lt;p&gt;I used a "formula fingerprint" to decide whether a page needs re-rendering — hash the compute function's source, compare against a stored fingerprint, skip if unchanged.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// WRONG&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;fp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;() =&amp;gt; window.CALC.fp&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The fingerprint came back empty every time. When &lt;code&gt;page.evaluate&lt;/code&gt; receives a string, it &lt;strong&gt;evaluates the expression and returns the resulting function object&lt;/strong&gt; — it never calls it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;callExpr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;`(&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;)()`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;     &lt;span class="c1"&gt;// becomes (() =&amp;gt; x)()&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;fp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;callExpr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;() =&amp;gt; window.CALC.fp&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same trap exists in &lt;code&gt;waitForFunction&lt;/code&gt;, and it's worse there: a string that evaluates truthy means &lt;strong&gt;you waited for nothing&lt;/strong&gt; and captured a half-rendered result.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 2: injecting before &lt;code&gt;&amp;lt;/body&amp;gt;&lt;/code&gt; shows users two copies
&lt;/h2&gt;

&lt;p&gt;My first version inserted the block before &lt;code&gt;&amp;lt;/body&amp;gt;&lt;/code&gt;. Locally, with JS enabled, users saw &lt;strong&gt;two result tables&lt;/strong&gt; — the static one sat outside &lt;code&gt;#result&lt;/code&gt;, so the live innerHTML overwrite never removed it.&lt;/p&gt;

&lt;p&gt;You have to splice into the container's &lt;em&gt;interior&lt;/em&gt;, matching tag depth:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;spliceInner&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;html&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;inner&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;open&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;RegExp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&amp;lt;div[^&amp;gt;]*id=["']&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;["'][^&amp;gt;]*&amp;gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;open&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exec&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;html&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;index&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;m&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;tagRe&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sr"&gt;/&amp;lt;div&lt;/span&gt;&lt;span class="se"&gt;\b&lt;/span&gt;&lt;span class="sr"&gt;|&amp;lt;&lt;/span&gt;&lt;span class="se"&gt;\/&lt;/span&gt;&lt;span class="sr"&gt;div&amp;gt;/g&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nx"&gt;tagRe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;lastIndex&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

  &lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;tagRe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exec&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;html&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;&amp;lt;/div&amp;gt;&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;depth&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;html&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;i&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;inner&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;html&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;index&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the static block lives inside &lt;code&gt;#result&lt;/code&gt; and gets replaced along with everything else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 3: re-running stacks blocks
&lt;/h2&gt;

&lt;p&gt;A build step must be idempotent. Inserting before &lt;code&gt;&amp;lt;/body&amp;gt;&lt;/code&gt; gives you three copies after three runs. Wrap the block in markers and replace on re-run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;BLOCK_RE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sr"&gt;/&amp;lt;!--&lt;/span&gt;&lt;span class="se"&gt;\s&lt;/span&gt;&lt;span class="sr"&gt;*prerender:start&lt;/span&gt;&lt;span class="se"&gt;[^&lt;/span&gt;&lt;span class="sr"&gt;&amp;gt;&lt;/span&gt;&lt;span class="se"&gt;]&lt;/span&gt;&lt;span class="sr"&gt;*--&amp;gt;&lt;/span&gt;&lt;span class="se"&gt;[\s\S]&lt;/span&gt;&lt;span class="sr"&gt;*&lt;/span&gt;&lt;span class="se"&gt;?&lt;/span&gt;&lt;span class="sr"&gt;&amp;lt;!--&lt;/span&gt;&lt;span class="se"&gt;\s&lt;/span&gt;&lt;span class="sr"&gt;*prerender:end&lt;/span&gt;&lt;span class="se"&gt;\s&lt;/span&gt;&lt;span class="sr"&gt;*--&amp;gt;/&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;after&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;BLOCK_RE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;before&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;before&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;BLOCK_RE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;block&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;inject&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;before&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;block&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Second run should report &lt;code&gt;0 rendered, 1 unchanged&lt;/code&gt;. &lt;strong&gt;If every run reports "all rendered", your fingerprint is empty — go back to trap 1.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  One more thing: lastmod ordering
&lt;/h2&gt;

&lt;p&gt;If your sitemap carries &lt;code&gt;lastmod&lt;/code&gt;, &lt;strong&gt;the prerender injection must run before lastmod is computed.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Otherwise adding prerender marks all 380 pages as "modified today". Run it again the next day — all 380 pages modified again. From the crawler's perspective your site claims a full-site rewrite every single day, which destroys your sitemap's credibility.&lt;/p&gt;

&lt;p&gt;If you must run it daily, strip the injected block from the content fingerprint — the block changed, the page's substance didn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;380 pages, on a single reused browser instance:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Build time: noticeably longer, but it's a one-off cost with a reused single browser instance — never in the user request path&lt;/li&gt;
&lt;li&gt;Every page returns its full result table with JS disabled&lt;/li&gt;
&lt;li&gt;Median real body content: 1,655 CJK characters (was ~400 characters of template skeleton)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What prerendering does &lt;em&gt;not&lt;/em&gt; fix
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;It's worth being blunt here.&lt;/strong&gt; Prerendering fixes "can the crawler see the content". It does not fix "does the search engine want to index you".&lt;/p&gt;

&lt;p&gt;I assumed fixing the blank-div problem would fix indexing. It didn't. Two weeks later, bingbot had fetched 2 pages out of 380 and indexed zero. The reason had nothing to do with my HTML — it was a new domain with zero inbound links, and no crawler algorithm allocates crawl budget for a site nobody mentions.&lt;/p&gt;

&lt;p&gt;Content visibility and site trust are two separate problems. Prerender solves the first one properly. The second one is a marketing problem, and you solve it by writing things people link to.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code and verification
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The code&lt;/strong&gt;: &lt;a href="https://gitee.com/the-art-of-strategic-planning/calc-prerender" rel="noopener noreferrer"&gt;calc-prerender&lt;/a&gt; — MIT licensed, ~180 lines of Node, a runnable example, and the three traps written up in the README. It is not a demo: this is the script that runs in the production build.&lt;/p&gt;

&lt;p&gt;Production case: &lt;a href="https://jianzhujisuan.com?src=devto" rel="noopener noreferrer"&gt;jianzhujisuan.com&lt;/a&gt; — 168 bilingual construction calculators, 380 pages, all built this way. The whole implementation (~180 lines) is what runs in that build pipeline.&lt;/p&gt;

&lt;p&gt;Quick verification: open a page like &lt;a href="https://jianzhujisuan.com/en/brick-count?src=devto" rel="noopener noreferrer"&gt;brick count calculator&lt;/a&gt;, disable JavaScript, refresh. If the result table is still there, it worked. If you see an empty div, prerendering never ran.&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>seo</category>
      <category>playwright</category>
      <category>webdev</category>
    </item>
  </channel>
</rss>
