<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Mahmut Gündüzalp</title>
    <description>The latest articles on DEV Community by Mahmut Gündüzalp (@mahmut_gndzalp_c736ac4b).</description>
    <link>https://dev.to/mahmut_gndzalp_c736ac4b</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2031144%2F507c6d99-33de-4ad3-b203-16e7eba3fc01.png</url>
      <title>DEV Community: Mahmut Gündüzalp</title>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mahmut_gndzalp_c736ac4b"/>
    <language>en</language>
    <item>
      <title>Our Restaurant TV Menu Never Showed Page 5: A Polling Refresh, a Rotating Carousel and Array Identity in React</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Fri, 13 Nov 2026 07:24:55 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/our-restaurant-tv-menu-never-showed-page-5-a-polling-refresh-a-rotating-carousel-and-array-4iik</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/our-restaurant-tv-menu-never-showed-page-5-a-polling-refresh-a-rotating-carousel-and-array-4iik</guid>
      <description>&lt;p&gt;We build a point-of-sale and QR menu product for cafes and restaurants. One small feature is a TV menu: a public URL that the restaurant types into the browser of the smart TV hanging in the dining room. It shows the whole menu in large type, product names on the left, prices on the right. No ordering, no clicks. Customers look at it and tell the waiter what they want.&lt;/p&gt;

&lt;p&gt;Two requirements shaped the screen. First, the TV must never need a human. If the owner changes a price in the admin panel, the TV has to pick it up on its own, so the page polls the menu endpoint every 60 seconds. Second, the menu must never scroll. A cafe with 20 items gets one screen with big letters. A restaurant with 300 items gets split into pages, and the pages rotate every 15 seconds.&lt;/p&gt;

&lt;p&gt;Both requirements worked in testing. Together they produced a bug that only shows up with a long menu: the TV never showed anything after page 4.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;Stripped down, the component looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;setSections&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;useState&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Section&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;page&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;setPage&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;useState&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// 1. Poll the menu every 60 s&lt;/span&gt;
&lt;span class="nf"&gt;useEffect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;load&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;menu&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;api&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fullMenu&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nf"&gt;setSections&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;menu&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;setInterval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;load&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;60_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;clearInterval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// 2. Measure rendered heights, split into pages&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;useMemo&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;paginate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;measurement&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;measurement&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// 3. Data or layout changed: go back to the first page&lt;/span&gt;
&lt;span class="nf"&gt;useEffect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nf"&gt;setPage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;sections&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;columns&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;measurement&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// 4. Rotate pages&lt;/span&gt;
&lt;span class="nf"&gt;useEffect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;setInterval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;setPage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;15_000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nf"&gt;clearInterval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;pages&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Effect 3 is reasonable on its own. If the owner deletes a category and the menu shrinks from 9 pages to 6, the current page index might point past the end. Resetting to page 0 when the data changes is the simple, safe answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually happened
&lt;/h2&gt;

&lt;p&gt;Every 60 seconds, effect 1 called &lt;code&gt;setSections&lt;/code&gt; with a freshly parsed array from the API. The content was identical to the previous response, but the array was new. React compares dependencies with &lt;code&gt;Object.is&lt;/code&gt;, so to React the sections had changed.&lt;/p&gt;

&lt;p&gt;That set off a chain:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;sections&lt;/code&gt; changed, so the measurement effect re-ran and produced a new &lt;code&gt;measurement&lt;/code&gt; object.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;sections&lt;/code&gt; and &lt;code&gt;measurement&lt;/code&gt; changed, so effect 3 ran and set the page back to 0.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;pages.length&lt;/code&gt; was still 9, so the rotation interval was not restarted. It just kept ticking from wherever it was.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;With a 15-second page duration and a 60-second refresh, the timeline looked like this:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;What the TV shows&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;0 s&lt;/td&gt;
&lt;td&gt;page 1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;15 s&lt;/td&gt;
&lt;td&gt;page 2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;30 s&lt;/td&gt;
&lt;td&gt;page 3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;45 s&lt;/td&gt;
&lt;td&gt;page 4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;60 s&lt;/td&gt;
&lt;td&gt;refresh: back to page 1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;75 s&lt;/td&gt;
&lt;td&gt;page 2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A full tour of 9 pages takes 135 seconds. The refresh fired every 60. Pages 5 to 9 were never displayed. Ever. Small menus were fine because they fit on 4 pages or fewer, and so did our test data.&lt;/p&gt;

&lt;p&gt;The bug was invisible in development for the same reason. Nobody sits and watches a carousel for two minutes. And when you reload the page to check something, you start at page 1 anyway, which looks correct.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the usual checks missed it
&lt;/h2&gt;

&lt;p&gt;There was nothing to catch. No error, no warning, no failed request. The network tab showed a successful request every minute. React DevTools showed the component re-rendering once a minute, which is what polling is supposed to do. Each piece did what it was written to do. The bug lived in the interaction between two timers with different periods and a reset rule that treated "new object" as "new data".&lt;/p&gt;

&lt;p&gt;For a restaurant, that means whatever sits at the end of the menu, often desserts and drinks, simply does not exist on the TV.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: only touch state when the content changes
&lt;/h2&gt;

&lt;p&gt;The fix was to stop handing React a new array when nothing had changed. We build a signature from the fields that actually affect what the TV draws and compare it with the last one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight tsx"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;signatureRef&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;useRef&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;load&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;menu&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;api&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fullMenu&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;slug&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;next&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;menu&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sections&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="n"&gt;signature&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;JSON&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stringify&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;image&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;products&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;price&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;isAvailable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;soldOut&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;priceOptions&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;price&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;signature&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="n"&gt;signatureRef&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;signatureRef&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;current&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;signature&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nf"&gt;setSections&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now an unchanged response does nothing. No new array, no new measurement, no reset. The carousel keeps rotating through all 9 pages. When the owner changes a price, the signature changes, the menu re-renders with the new price and the reset to page 1 is the correct behaviour, because the layout may have changed.&lt;/p&gt;

&lt;p&gt;A few notes on the choice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Why not compare the whole response?&lt;/strong&gt; The API also returns things the TV ignores, such as timestamps and fields used by the ordering screen. Including them would make the signature change for reasons the customer cannot see. The signature lists exactly what is drawn.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Why &lt;code&gt;JSON.stringify&lt;/code&gt; and not a deep-equal library?&lt;/strong&gt; The menu is a few hundred products at most. Stringifying it once a minute costs less than a millisecond on a TV browser, and the code is obvious to the next person who reads it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Why a ref and not state?&lt;/strong&gt; The signature is bookkeeping. Storing it in state would itself cause a render.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The general lesson
&lt;/h2&gt;

&lt;p&gt;The pattern is common: a polling hook or a WebSocket feed sets state on every message, and somewhere downstream an effect treats "the dependency changed" as "the data changed". In most components this costs an extra render and nobody notices. It becomes a bug when the downstream effect does something visible or stateful: resets a page, clears a form, scrolls to the top, restarts an animation, refetches something else.&lt;/p&gt;

&lt;p&gt;What we check now when a component both polls and holds UI state:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Does the fetch set state unconditionally?&lt;/strong&gt; If so, ask whether an unchanged response should be a no-op. It usually should.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Which effects depend on the polled data, and do any of them reset something?&lt;/strong&gt; Reset effects are where identity changes turn into user-visible behaviour.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Are there two timers with different periods?&lt;/strong&gt; A refresh interval and a rotation interval, a refresh and a debounce, a refresh and a toast timeout. Write down the timeline for the longest realistic case, like we did in the table above. Ten seconds with a pencil would have found this bug.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test with production-sized data.&lt;/strong&gt; A short test menu fits on a page or two. The bug needed more than four.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Libraries like TanStack Query help here because they keep referential identity for unchanged data through structural sharing. If you write the polling yourself, you get to do that part yourself.&lt;/p&gt;

&lt;h2&gt;
  
  
  A related design choice
&lt;/h2&gt;

&lt;p&gt;The same screen does not hide sold-out products. They stay in place, dimmed, with a "SOLD OUT" label. Removing them would reflow the whole menu and move items between pages every time stock changed, which is the same kind of surprise from the customer's side: the thing you were reading a moment ago is somewhere else now.&lt;/p&gt;

&lt;p&gt;We write about the products behind this, restaurant POS and QR menus among them, at &lt;a href="https://alestaweb.com/adisyon-programi" rel="noopener noreferrer"&gt;alestaweb.com&lt;/a&gt;. The TV screen itself is one URL and a smart TV browser, which is about as low-tech as a feature gets. It still managed to hide more than half of a long menu.&lt;/p&gt;

</description>
      <category>react</category>
      <category>javascript</category>
      <category>webdev</category>
      <category>frontend</category>
    </item>
    <item>
      <title>The Cleanup Script Was Right About the Name and Wrong About the File: Dynamic Includes and Leftover-File Hunts</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Sat, 07 Nov 2026 13:03:03 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/the-cleanup-script-was-right-about-the-name-and-wrong-about-the-file-dynamic-includes-and-28lp</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/the-cleanup-script-was-right-about-the-name-and-wrong-about-the-file-dynamic-includes-and-28lp</guid>
      <description>&lt;p&gt;Every few weeks we run a maintenance pass over the servers we look after. One step is a hunt for leftover files in web roots: the &lt;code&gt;fix_something.php&lt;/code&gt;, &lt;code&gt;test2.php&lt;/code&gt;, &lt;code&gt;debug_login.php&lt;/code&gt; and &lt;code&gt;install.php&lt;/code&gt; files that pile up on any server that has been alive long enough. They are worth hunting. On one pass we found a debug script that printed a payment provider's API credentials to anyone who opened it, an installer that still created an &lt;code&gt;admin/admin123&lt;/code&gt; account, and a "test" file that dumped server details.&lt;/p&gt;

&lt;p&gt;So the hunt is useful. It is also how we broke a live product page on a customer site, and the way it happened is worth writing down, because the check we relied on looked solid and was not.&lt;/p&gt;

&lt;h2&gt;
  
  
  The rule we had
&lt;/h2&gt;

&lt;p&gt;The rule was reasonable on paper:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Find files whose names match a leftover pattern: &lt;code&gt;fix*&lt;/code&gt;, &lt;code&gt;test*&lt;/code&gt;, &lt;code&gt;debug*&lt;/code&gt;, &lt;code&gt;check*&lt;/code&gt;, &lt;code&gt;verify*&lt;/code&gt;, &lt;code&gt;*_backup.php&lt;/code&gt;, &lt;code&gt;*-old*.php&lt;/code&gt;, &lt;code&gt;setup*.php&lt;/code&gt;, &lt;code&gt;phpinfo*&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;For each match, search the code base for references to the file name.&lt;/li&gt;
&lt;li&gt;If nothing references it, move it to a quarantine directory instead of deleting it.&lt;/li&gt;
&lt;li&gt;Never call a candidate over HTTP to "see what it does". Calling a &lt;code&gt;fix_*.php&lt;/code&gt; script runs it, and some of them change data.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Step 2 is the safety net. If &lt;code&gt;fix_totals.php&lt;/code&gt; is included by &lt;code&gt;cron.php&lt;/code&gt;, the search finds the name and the file stays.&lt;/p&gt;

&lt;h2&gt;
  
  
  The file that matched
&lt;/h2&gt;

&lt;p&gt;On an older PHP catalogue site, the pattern &lt;code&gt;fix*&lt;/code&gt; matched a file called something like &lt;code&gt;product/&amp;lt;brand&amp;gt;/fixpro.php&lt;/code&gt;. Nobody had heard of it. The reference search over the whole document root returned nothing. The file went to quarantine, and some weeks later the quarantine was cleared.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;fixpro&lt;/code&gt; was not a fix. It was a product name. The file was the product detail page for a hardware item called "FixPro", and the site loaded product pages like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="c1"&gt;// detail.php&lt;/span&gt;
&lt;span class="nv"&gt;$brand&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$_GET&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'brand'&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;   &lt;span class="c1"&gt;// validated against a list&lt;/span&gt;
&lt;span class="nv"&gt;$prog&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nv"&gt;$_GET&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'prog'&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;    &lt;span class="c1"&gt;// validated against a list&lt;/span&gt;
&lt;span class="k"&gt;require&lt;/span&gt; &lt;span class="s2"&gt;"product/&lt;/span&gt;&lt;span class="nv"&gt;$brand&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="nv"&gt;$prog&lt;/span&gt;&lt;span class="s2"&gt;.php"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The string &lt;code&gt;fixpro&lt;/code&gt; appeared nowhere in the code. It existed only in a database row, in a rewrite rule pattern and in incoming URLs. A grep for the file name could never find it, because the reference was built at runtime.&lt;/p&gt;

&lt;p&gt;The page started returning 500. The off-site backups only kept a few days of revisions for that file, so by the time anyone noticed, the old copy was gone too. The text and images of one product page had to be recreated by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the reference search lied
&lt;/h2&gt;

&lt;p&gt;A static reference search answers "does any source file contain this string?". It does not answer "does anything load this file?". The gap between those two questions is wider than it looks in old PHP code:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Variable includes:&lt;/strong&gt; &lt;code&gt;require "pages/$page.php"&lt;/code&gt;, &lt;code&gt;include $module . '/index.php'&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rewrite rules:&lt;/strong&gt; &lt;code&gt;RewriteRule ^urun/([a-z]+)/([a-z0-9-]+)$ detail.php?brand=$1&amp;amp;prog=$2&lt;/code&gt;. The rule mentions the directory pattern, not the file.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Database-driven routing:&lt;/strong&gt; a &lt;code&gt;pages&lt;/code&gt; table with a &lt;code&gt;template&lt;/code&gt; column whose values are file names.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Autoloaders:&lt;/strong&gt; class names mapped to paths. &lt;code&gt;FixProController&lt;/code&gt; becomes &lt;code&gt;Controllers/FixProController.php&lt;/code&gt; without the path ever being written down.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cron and external callers:&lt;/strong&gt; a hosting panel's scheduled task or a payment provider's callback URL can point straight at a file that nothing in the code base mentions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Direct links from outside:&lt;/strong&gt; a PDF, a QR code, a newsletter, an old social media post.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Name patterns are a decent way to find candidates. They are a bad way to decide anything, because the name is the least reliable thing about the file.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we check now
&lt;/h2&gt;

&lt;p&gt;A file that matches a leftover pattern is only a candidate. Before it can be quarantined, three more checks have to come back empty. If any of them finds something, the file stays.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Does it have siblings that look like content?
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;dirname&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-50&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If &lt;code&gt;product/led/&lt;/code&gt; contains &lt;code&gt;strip5050.php&lt;/code&gt;, &lt;code&gt;panel60.php&lt;/code&gt; and &lt;code&gt;driver24.php&lt;/code&gt;, then &lt;code&gt;product/brandx/fixpro.php&lt;/code&gt; is almost certainly one more content page, whatever its name looks like. A directory of similar files is a strong sign of runtime includes.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Does any rewrite rule or router mention its directory or name stem?
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;stem&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;basename&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; .php&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;basename&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;dirname&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rnE&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$stem&lt;/span&gt;&lt;span class="s2"&gt;|&lt;/span&gt;&lt;span class="nv"&gt;$dir&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; .htaccess &lt;span class="k"&gt;*&lt;/span&gt;/.htaccess 2&amp;gt;/dev/null
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rnE&lt;/span&gt; &lt;span class="s2"&gt;"(require|include)(_once)?&lt;/span&gt;&lt;span class="se"&gt;\s&lt;/span&gt;&lt;span class="s2"&gt;*[&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s2"&gt;'(].*&lt;/span&gt;&lt;span class="se"&gt;\\\$&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--include&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;*&lt;/span&gt;.php &lt;span class="nb"&gt;.&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second grep lists every include that uses a variable. If there are any, the reference search from the old rule is no longer enough on its own, and the file needs one of the other checks to be cleared.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Has anyone requested it recently?
&lt;/h3&gt;

&lt;p&gt;This is the check that would have saved the page:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# last 30 days of access logs, any status&lt;/span&gt;
zgrep &lt;span class="nt"&gt;-h&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;basename&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; /path/to/access.log&lt;span class="k"&gt;*&lt;/span&gt; | &lt;span class="nb"&gt;awk&lt;/span&gt; &lt;span class="s1"&gt;'{print $9}'&lt;/span&gt; | &lt;span class="nb"&gt;sort&lt;/span&gt; | &lt;span class="nb"&gt;uniq&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A product page gets requests. A forgotten fix script usually does not. A 200 in the log means something reaches the file, through a route you may not have found. A 404 or no line at all does not prove the file is dead, but it removes the strongest reason to keep it.&lt;/p&gt;

&lt;p&gt;For URL-routed files the file name may not appear in the log at all, because the URL is &lt;code&gt;/urun/brandx/fixpro&lt;/code&gt; and not &lt;code&gt;/product/brandx/fixpro.php&lt;/code&gt;. So we search for the stem, not only the full file name.&lt;/p&gt;

&lt;h3&gt;
  
  
  And then: keep the quarantine longer than the backups
&lt;/h3&gt;

&lt;p&gt;The second mistake was clearing the quarantine sooner than the backups could cover. We now keep quarantined files for at least 30 days and never clear quarantine before checking that the off-site backup for that site actually holds an older revision. A quarantine you delete in a week is just a slower &lt;code&gt;rm&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  A small script
&lt;/h2&gt;

&lt;p&gt;This is roughly what the candidate filter does now. It prints a verdict per file and never moves anything itself.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="c"&gt;# usage: leftover-check.sh &amp;lt;docroot&amp;gt; &amp;lt;access-log-glob&amp;gt;&lt;/span&gt;
&lt;span class="nv"&gt;root&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nv"&gt;logs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;
find &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$root&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-type&lt;/span&gt; f &lt;span class="nt"&gt;-name&lt;/span&gt; &lt;span class="s1"&gt;'*.php'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="se"&gt;\(&lt;/span&gt; &lt;span class="nt"&gt;-iname&lt;/span&gt; &lt;span class="s1"&gt;'fix*'&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nt"&gt;-iname&lt;/span&gt; &lt;span class="s1"&gt;'test*'&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nt"&gt;-iname&lt;/span&gt; &lt;span class="s1"&gt;'debug*'&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nt"&gt;-iname&lt;/span&gt; &lt;span class="s1"&gt;'check*'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
     &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nt"&gt;-iname&lt;/span&gt; &lt;span class="s1"&gt;'*_backup.php'&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nt"&gt;-iname&lt;/span&gt; &lt;span class="s1"&gt;'*-old*.php'&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="nt"&gt;-iname&lt;/span&gt; &lt;span class="s1"&gt;'setup*.php'&lt;/span&gt; &lt;span class="se"&gt;\)&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-not&lt;/span&gt; &lt;span class="nt"&gt;-path&lt;/span&gt; &lt;span class="s1"&gt;'*/vendor/*'&lt;/span&gt; | &lt;span class="k"&gt;while &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; f&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
    &lt;/span&gt;&lt;span class="nv"&gt;stem&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;basename&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; .php&lt;span class="si"&gt;)&lt;/span&gt;
    &lt;span class="nb"&gt;dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;dirname&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
    &lt;span class="nv"&gt;reason&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;
    &lt;span class="nv"&gt;siblings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;find &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$dir&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-maxdepth&lt;/span&gt; 1 &lt;span class="nt"&gt;-name&lt;/span&gt; &lt;span class="s1"&gt;'*.php'&lt;/span&gt; | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$siblings&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-gt&lt;/span&gt; 3 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; reason+&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"siblings=&lt;/span&gt;&lt;span class="nv"&gt;$siblings&lt;/span&gt;&lt;span class="s2"&gt; "&lt;/span&gt;
    &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rqF&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$stem&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--include&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;*&lt;/span&gt;.php &lt;span class="nt"&gt;--include&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;.htaccess &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$root&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
      &lt;span class="nt"&gt;--exclude&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;basename&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; reason+&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"referenced "&lt;/span&gt;
    &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-qE&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;basename&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$dir&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$root&lt;/span&gt;&lt;span class="s2"&gt;/.htaccess"&lt;/span&gt; 2&amp;gt;/dev/null &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; reason+&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"rewrite "&lt;/span&gt;
    &lt;span class="nv"&gt;hits&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;zgrep &lt;span class="nt"&gt;-hc&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$stem&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nv"&gt;$logs&lt;/span&gt; 2&amp;gt;/dev/null | &lt;span class="nb"&gt;paste&lt;/span&gt; &lt;span class="nt"&gt;-sd&lt;/span&gt;+ | bc&lt;span class="si"&gt;)&lt;/span&gt;
    &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;hits&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;0&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-gt&lt;/span&gt; 0 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; reason+&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"log_hits=&lt;/span&gt;&lt;span class="nv"&gt;$hits&lt;/span&gt;&lt;span class="s2"&gt; "&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$reason&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"KEEP  &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;  (&lt;/span&gt;&lt;span class="nv"&gt;$reason&lt;/span&gt;&lt;span class="s2"&gt;)"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;else &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"CAND  &lt;/span&gt;&lt;span class="nv"&gt;$f&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;fi
done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;CAND&lt;/code&gt; still means "a person looks at it", not "delete it".&lt;/p&gt;

&lt;h2&gt;
  
  
  What did not change
&lt;/h2&gt;

&lt;p&gt;We still hunt leftovers, and we still find bad ones. The debug script that printed payment credentials was real, and a cautious rule that never removes anything would have left it online. Security findings like that one go to quarantine immediately, because the cost of leaving them is higher than the cost of a broken page.&lt;/p&gt;

&lt;p&gt;The change is in how the decision is made. A file name is a reason to look at a file. Siblings, rewrite rules, access logs and the age of the newest backup are the reasons to keep it or let it go.&lt;/p&gt;

&lt;p&gt;If you run maintenance on old PHP sites, the one-line version is: grep for the variable includes before you grep for the file name. If there are any, your reference search is incomplete, and the access log is the only witness you have left.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;We run maintenance like this for the sites we build and host at &lt;a href="https://alestaweb.com" rel="noopener noreferrer"&gt;Alesta WEB&lt;/a&gt;. The rule changes above came out of a real incident; the file and brand names here are changed.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>php</category>
      <category>devops</category>
      <category>security</category>
    </item>
    <item>
      <title>The Site Moved. The Scheduler Didn't: 19 Days of Silent Background Jobs After a Migration</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Fri, 30 Oct 2026 06:44:18 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/the-site-moved-the-scheduler-didnt-19-days-of-silent-background-jobs-after-a-migration-ja8</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/the-site-moved-the-scheduler-didnt-19-days-of-silent-background-jobs-after-a-migration-ja8</guid>
      <description>&lt;p&gt;We moved a busy local news site to a new server. The migration checklist was long and we were proud of it: files, database, media, DNS, TLS, redirects, image paths, mail, a performance pass. The home page loaded in half a second instead of four. Editors logged in, wrote stories, and published them. Every check we had was green.&lt;/p&gt;

&lt;p&gt;Nineteen days later, during a routine look at table sizes, one number stood out. A small table that normally holds a few thousand rows had close to 600,000. Nothing was broken from the outside. The site was fast, pages returned 200, nobody had complained.&lt;/p&gt;

&lt;p&gt;The table was big because the job that trims it had not run once since the migration. Neither had any other background job. The application's scheduler, a single crontab line, had never been added on the new server.&lt;/p&gt;

&lt;p&gt;This post is about why that kind of failure is invisible, what it actually cost us, and the three checks we now treat as part of every move.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the scheduler works in this application
&lt;/h2&gt;

&lt;p&gt;The CMS has one scheduler entry point. The operating system runs it every minute from cron:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;* * * * * /path/to/php /path/to/app/scheduler.php &amp;gt; /dev/null 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That one script reads a list of jobs from the database, each with its own interval, and runs whichever are due. The list is long: publishing posts that were scheduled for a future time, rolling raw visit logs up into hourly statistics, deleting old rows from logging and rate-limit tables, rebuilding sitemaps, refreshing small data widgets, sending queued notifications.&lt;/p&gt;

&lt;p&gt;This design is convenient. Adding a job is a database row, not a server change. It is also a single point of failure that lives outside the application's code, in the system crontab of whatever user the site runs as. Copy the files and the database, and you have copied the job list but not the thing that runs it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why nobody noticed for 19 days
&lt;/h2&gt;

&lt;p&gt;Look at what each job does and ask who would see it fail.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Scheduled publishing.&lt;/strong&gt; Editors who schedule a story for 06:00 would notice when it doesn't appear. On this site the newsroom publishes immediately; nobody had scheduled anything. We checked afterwards: no posts were stuck in a scheduled state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Statistics roll-up.&lt;/strong&gt; The admin dashboard reads today's numbers from the raw log, so "today" looked fine. Only longer ranges, built from the hourly summaries, were going stale, and nobody had opened a monthly report yet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cleanup jobs.&lt;/strong&gt; Their whole effect is that something &lt;em&gt;doesn't&lt;/em&gt; happen: a table doesn't grow. A table that is growing gives no signal until it is big enough to slow a query or fill a disk.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Data widgets.&lt;/strong&gt; Currency rates, weather and football fixtures were the only visibly stale things. We had actually noticed those during the migration and fixed them by adding a few dedicated cron lines, one per widget. That fix made the problem harder to see: the visible symptoms went away, so the missing scheduler line looked like it had been handled.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Our uptime monitor checked the home page and an article page. Both were fine. A health check that only asks "does the page render" will never see a job that didn't run, because the page doesn't depend on it today.&lt;/p&gt;

&lt;p&gt;There is a general rule hiding here: &lt;strong&gt;the failure of a background job is invisible until the job's absence becomes a foreground problem.&lt;/strong&gt; Cleanup jobs take weeks. Aggregation jobs take until someone opens a report. Publishing jobs take until someone schedules a post.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it cost
&lt;/h2&gt;

&lt;p&gt;In this case, less than it could have. No content was lost and nothing went down. The damage was:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A rate-limit and request log table with roughly 594,000 rows that should have held a few days of data. Inserts into it happen on every request, so it was slowly making every request a little more expensive.&lt;/li&gt;
&lt;li&gt;Nineteen days without hourly statistics summaries. Summaries can only be rebuilt from raw logs that still exist. Here the raw logs were still there, because the cleanup hadn't run either. If the roll-up had stopped while the cleanup kept running, the raw data behind those reports would have been deleted on schedule.&lt;/li&gt;
&lt;li&gt;A few days of security log retention that was longer than our own policy says it should be.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The more uncomfortable cost is what would have happened on a different site. On a site that relies on scheduled publishing, the morning's stories would have stayed drafts, and the newsroom would have discovered it at 06:01 on a weekday.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix itself was one line
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;crontab &lt;span class="nt"&gt;-u&lt;/span&gt; siteuser &lt;span class="nt"&gt;-l&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; crontab-before.txt   &lt;span class="c"&gt;# keep the old state&lt;/span&gt;
&lt;span class="o"&gt;(&lt;/span&gt; crontab &lt;span class="nt"&gt;-u&lt;/span&gt; siteuser &lt;span class="nt"&gt;-l&lt;/span&gt; 2&amp;gt;/dev/null&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'* * * * * /path/to/php -d date.timezone=Europe/Istanbul /path/to/app/scheduler.php &amp;gt; /dev/null 2&amp;gt;&amp;amp;1'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="o"&gt;)&lt;/span&gt; | crontab &lt;span class="nt"&gt;-u&lt;/span&gt; siteuser -
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details mattered more than the line:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The per-widget lines had to go.&lt;/strong&gt; They were now duplicates of jobs the scheduler also runs. Two copies of the same job, running at slightly different times, is its own source of subtle bugs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The site sat behind a CDN proxy, and some jobs call the site's own URLs.&lt;/strong&gt; From the server, those requests went out to the CDN and back in, picking up rate limits and bot checks on the way. Pointing the site's own hostnames at the local machine in &lt;code&gt;/etc/hosts&lt;/code&gt; keeps self-requests local.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Before turning the scheduler on, we checked for posts with a publish time in the past that were still waiting. A scheduler that has been off for weeks will publish all of them in its first minute. On a migrated site those can be old test posts, so look before you enable.&lt;/p&gt;

&lt;p&gt;After enabling it, the job table told us within two minutes whether it worked:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;last_run&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;last_status&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;scheduled_jobs&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;last_run&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The jobs started showing fresh &lt;code&gt;last_run&lt;/code&gt; values, and the oversized table was left to the regular nightly cleanup instead of a manual &lt;code&gt;DELETE&lt;/code&gt; during business hours.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three checks we now run on every migration
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Diff the crontabs, not just the files
&lt;/h3&gt;

&lt;p&gt;Before the move, dump every crontab that touches the site: the site user's, root's, and any system cron file that mentions the site's path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;crontab &lt;span class="nt"&gt;-u&lt;/span&gt; siteuser &lt;span class="nt"&gt;-l&lt;/span&gt;
crontab &lt;span class="nt"&gt;-l&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-F&lt;/span&gt; &lt;span class="s1"&gt;'/path/to/app'&lt;/span&gt;
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rlF&lt;/span&gt; &lt;span class="s1"&gt;'/path/to/app'&lt;/span&gt; /etc/cron.d /etc/crontab 2&amp;gt;/dev/null
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After the move, run the same commands on the new server and compare. Any line that exists on the old server and not the new one is either a deliberate decision, written down, or a bug. We had migrated the files with &lt;code&gt;rsync&lt;/code&gt;, the database with a dump, and the crontab with memory. Memory lost.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Monitor "time since last run", not "is the site up"
&lt;/h3&gt;

&lt;p&gt;The scheduler already wrote a &lt;code&gt;last_run&lt;/code&gt; timestamp for every job. Nothing was reading it. We added a check to our regular server health script that asks one question per site:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;stale&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;scheduled_jobs&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;enabled&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;last_run&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;NOW&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;INTERVAL&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt; &lt;span class="k"&gt;MINUTE&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;run_every_minutes&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a job that should run every few minutes has not run in ten, the check warns. It also warns if the site has no scheduler line in any crontab at all. This turns an invisible failure into a visible one within minutes instead of weeks.&lt;/p&gt;

&lt;p&gt;If your application doesn't record job runs, the cheapest version is a heartbeat: have the scheduler touch a file or ping a URL every run, and alert when the file's age passes a threshold. Dead-man's-switch services exist for exactly this, and a ten-line script against a timestamp does the same job.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Put the scheduler in the definition of "migrated"
&lt;/h3&gt;

&lt;p&gt;Our checklist said "copy cron jobs" somewhere in the middle. It now says, as the last step before anyone signs off:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The scheduler is running on the new server. Proof: the job table shows a &lt;code&gt;last_run&lt;/code&gt; within the last two minutes for every enabled job.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A checklist item that needs a timestamp as evidence is hard to tick by accident. "Copy cron jobs" was easy to tick after adding the six widget lines, because cron jobs had, in fact, been copied.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to take from this
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A scheduler that lives in the system crontab is part of your deployment, even if every job it runs lives in your database.&lt;/li&gt;
&lt;li&gt;Background job failures are silent by design. Plan for that by monitoring their timestamps, because their effects will show up late and somewhere else.&lt;/li&gt;
&lt;li&gt;When you fix a symptom during a migration, ask what produced it. We fixed stale widgets one by one and missed that they all had the same cause.&lt;/li&gt;
&lt;li&gt;Before re-enabling a scheduler that has been off, look at the queue it will process. Days of backlog can run in one minute.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We run a small fleet of news sites on the same CMS at &lt;a href="https://alestaweb.com" rel="noopener noreferrer"&gt;Alesta WEB&lt;/a&gt;, and the stale-job check now runs against all of them as part of regular server maintenance. It is a few lines of SQL, and it would have caught this on day one.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>linux</category>
      <category>webdev</category>
      <category>monitoring</category>
    </item>
    <item>
      <title>PHP2Shell (CVE-2026-109618 / CVE-2026-109619): A Practical Patch, Mitigation and Incident Checklist</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Wed, 21 Oct 2026 22:04:44 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/php2shell-cve-2026-109618-cve-2026-109619-a-practical-patch-mitigation-and-incident-checklist-53ml</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/php2shell-cve-2026-109618-cve-2026-109619-a-practical-patch-mitigation-and-incident-checklist-53ml</guid>
      <description>&lt;p&gt;On 21 October 2026 at 14:00 UTC the PHP project shipped fixes for two critical bugs in the PHP engine itself, nicknamed &lt;strong&gt;PHP2Shell&lt;/strong&gt;. Both allow unauthenticated remote code execution over plain HTTP. Details leaked a day early, the release was pulled forward from 22 October, and many organisations took PHP sites offline as a precaution while packages were still being built.&lt;/p&gt;

&lt;p&gt;This post is a working checklist: what is affected, how to update on common distributions, panels and containers, how to confirm the fix is actually running, what temporary mitigations do and do not cover, and what to look for if you could not patch in time. Everything here comes from the official advisories and vendor notes listed at the end. Where sources disagree, I say so.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the two bugs are
&lt;/h2&gt;

&lt;p&gt;PHP aborts a request when it hits &lt;code&gt;memory_limit&lt;/code&gt;. If that abort happens at the wrong moment, internal state is left half-initialised or still points at freed memory, and the &lt;strong&gt;next request handled by the same worker&lt;/strong&gt; uses it.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CVE-2026-109618&lt;/strong&gt; ("RCE via OOM corruption", GHSA-w8hw-w9fx-j79w). PHP leaves the request with &lt;code&gt;longjmp()&lt;/code&gt; on memory exhaustion. If request memory is only partly initialised at that point, shutdown processes invalid state. Several multipart parsing issues (no usable content length, unlimited headers per part, nested empty-file checks) can push a request into OOM while staying under &lt;code&gt;post_max_size&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CVE-2026-109619&lt;/strong&gt; ("RCE via stale global state", GHSA-6wmf-4rv5-p3rp). Request-scoped global state can outlive the request, and some code runs before normal request startup and reads it: a heap use-after-free. The confirmed paths are the &lt;strong&gt;session upload progress&lt;/strong&gt; lifecycle (mod_php and FPM) and, on PHP 8.5, &lt;code&gt;fatal_error_backtraces&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The trigger is a crafted &lt;code&gt;multipart/form-data&lt;/code&gt; request. No login is needed, and the application code does not have to run at all. Researchers also described a memory disclosure path where reclaimed heap content comes back in the &lt;code&gt;PHPSESSID&lt;/code&gt; response cookie, even if the app never calls &lt;code&gt;session_start()&lt;/code&gt;. That leaked memory can hold other users' passwords, session tokens and API keys.&lt;/p&gt;

&lt;p&gt;A note on CVE mapping: Wiz and Sansec group the upload-progress bugs under 109618, while the official PHP advisories put session upload progress under 109619. Use the GHSA advisories as the reference.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Severity:&lt;/strong&gt; CVSS 4.0 &lt;strong&gt;9.2&lt;/strong&gt; per the PHP advisory. The 10.0 that circulated with the leak is wrong according to the PHP Foundation. NVD shows CVSS 3.1 9.8 and Red Hat rates it CVSS 3 8.1, so you will see all three numbers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Your app framework updates do not fix this.&lt;/strong&gt; The bug is in the interpreter. A CMS, plugin or framework update leaves you exactly as exposed as before. You have to update PHP on the server.&lt;/p&gt;

&lt;h2&gt;
  
  
  Patch matrix
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Branch&lt;/th&gt;
&lt;th&gt;Status&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;8.5.x&lt;/td&gt;
&lt;td&gt;Fixed in &lt;strong&gt;8.5.12&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8.4.x&lt;/td&gt;
&lt;td&gt;Fixed in &lt;strong&gt;8.4.27&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8.3.x&lt;/td&gt;
&lt;td&gt;Fixed in &lt;strong&gt;8.3.36&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8.2.x&lt;/td&gt;
&lt;td&gt;Fixed in &lt;strong&gt;8.2.35&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;8.1 / 8.0 / 7.4&lt;/td&gt;
&lt;td&gt;No official release. Minimal source backports were given to distributors, so the fix depends on your vendor's package&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5.4 – 7.3&lt;/td&gt;
&lt;td&gt;End of life. No fix from the PHP project. Some third-party repositories ship their own patches&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The affected code path goes back to PHP 5.4.0 (2012).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Distribution packages backport.&lt;/strong&gt; The version string often stays old. Ubuntu 24.04's fixed package is &lt;code&gt;8.3.6-0ubuntu0.24.04.13&lt;/code&gt;, not 8.3.36. Compare package revisions against your vendor's advisory, not upstream version numbers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Which SAPI is exposed
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;SAPI&lt;/th&gt;
&lt;th&gt;Risk&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Apache mod_php&lt;/td&gt;
&lt;td&gt;Highest. Full RCE over HTTP confirmed with default settings&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PHP-FPM&lt;/td&gt;
&lt;td&gt;High. Upload-progress use-after-free confirmed. The advisory says full OOM exploitation was not shown, while the Foundation blog says RCE via application code paths was confirmed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LiteSpeed LSAPI (lsphp), FrankenPHP&lt;/td&gt;
&lt;td&gt;Possibly affected based on source analysis, not confirmed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;php-cgi&lt;/td&gt;
&lt;td&gt;Affected (anything that serves web requests)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CLI&lt;/td&gt;
&lt;td&gt;No web exposure described if it does not serve HTTP&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Updating, per platform
&lt;/h2&gt;

&lt;p&gt;The general rule is the same everywhere: &lt;strong&gt;update the package, restart every PHP worker, verify over HTTP.&lt;/strong&gt; Installing the package alone proves nothing. Control-panel PHP builds are usually separate from the OS packages, so &lt;code&gt;apt&lt;/code&gt; or &lt;code&gt;dnf&lt;/code&gt; will not touch them.&lt;/p&gt;

&lt;h3&gt;
  
  
  Debian
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Debian 13 (trixie): &lt;code&gt;php8.4&lt;/code&gt; fixed in &lt;code&gt;8.4.27-1~deb13u1&lt;/code&gt; (DSA-6574-1).&lt;/li&gt;
&lt;li&gt;Debian 12 (bookworm): &lt;code&gt;php8.2&lt;/code&gt; reported fixed in &lt;code&gt;8.2.35-1~deb12u1&lt;/code&gt;. One source still showed bookworm as vulnerable at the time, so check the Debian security tracker before you trust it.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;apt-get update
&lt;span class="nb"&gt;sudo &lt;/span&gt;apt-get &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;--only-upgrade&lt;/span&gt; &lt;span class="s1"&gt;'php8.2*'&lt;/span&gt; libapache2-mod-php8.2
dpkg &lt;span class="nt"&gt;-l&lt;/span&gt; | &lt;span class="nb"&gt;grep &lt;/span&gt;php
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Debian changelog may only say "New upstream version", so check the package version itself.&lt;/p&gt;

&lt;h3&gt;
  
  
  Ubuntu
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;22.04: &lt;code&gt;8.1.2-1ubuntu2.28&lt;/code&gt; · 24.04: &lt;code&gt;8.3.6-0ubuntu0.24.04.13&lt;/code&gt; · 26.04: &lt;code&gt;8.5.4-0ubuntu1.5&lt;/code&gt; (USN-8948-1)&lt;/li&gt;
&lt;li&gt;18.04 / 20.04: via Ubuntu Pro ESM (USN-8948-2)
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;zgrep &lt;span class="nt"&gt;-E&lt;/span&gt; &lt;span class="s1"&gt;'CVE-2026-10961[89]'&lt;/span&gt; /usr/share/doc/php8.3-fpm/changelog.Debian.gz
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Third-party Debian/Ubuntu repositories
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;packages.sury.org&lt;/code&gt;: all four supported branches were fixed on amd64 first; arm64 lagged at the last check. Updates for branches back to 5.6 were reported.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;ppa:ondrej/php&lt;/code&gt;: all four branches fixed on amd64 and arm64 by 21:22 UTC on 21 October. Some mirrors served 8.3.35 for a while that day. If yours lags, use &lt;code&gt;ppa.launchpadcontent.net&lt;/code&gt; directly.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  RHEL, AlmaLinux, Rocky
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;AlmaLinux 8/9/10: all PHP streams in production repositories from 21:15 UTC on 21 October.&lt;/li&gt;
&lt;li&gt;Red Hat: check the CVE page on access.redhat.com for your stream. I could not verify an erratum number from a primary source.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf upgrade &lt;span class="s1"&gt;'php*'&lt;/span&gt;
dnf updateinfo list &lt;span class="nt"&gt;--security&lt;/span&gt; | &lt;span class="nb"&gt;grep &lt;/span&gt;php
rpm &lt;span class="nt"&gt;-q&lt;/span&gt; php-common
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Remi repository
&lt;/h3&gt;

&lt;p&gt;Sources conflict here. The PHP announcement and some vendors said Remi would not ship on 21 October. A later status page listed fixed 8.2–8.5 builds for RHEL 8/9/10 and best-effort patches for 7.4/8.0/8.1. Run &lt;code&gt;sudo dnf clean metadata&lt;/code&gt;, update (&lt;code&gt;dnf upgrade 'php84-*'&lt;/code&gt; for SCL packages), and check the installed version.&lt;/p&gt;

&lt;h3&gt;
  
  
  cPanel (EasyApache 4)
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;ea-php82&lt;/code&gt; to &lt;code&gt;ea-php85&lt;/code&gt; are fixed, with backports for &lt;code&gt;ea-php74/80/81&lt;/code&gt;. Update via WHM &amp;gt; EasyApache 4 or:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dnf upgrade &lt;span class="s1"&gt;'ea-php*'&lt;/span&gt;     &lt;span class="c"&gt;# or: yum update 'ea-php*'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then restart Apache and PHP-FPM.&lt;/p&gt;

&lt;h3&gt;
  
  
  Plesk
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;plesk-php&lt;/code&gt; packages are fixed on all four supported branches. EOL branches depend on builds dated 21 October or later. The panel's own PHP has a separate hotfix. Windows fixes were behind Linux.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;plesk installer update
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  DirectAdmin (CustomBuild)
&lt;/h3&gt;

&lt;p&gt;Per the vendor forum, the new build moves 8.2–8.5 to the fixed releases and adds patches for 8.1/8.0/7.4. A "failed hunk" on CentOS 7.9 with the 8.1 patch was reported.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;da build update &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; da build php
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  LiteSpeed lsphp packages
&lt;/h3&gt;

&lt;p&gt;Update &lt;code&gt;lsphpXY&lt;/code&gt; from the LiteSpeed repository, then kill the PHP processes. &lt;strong&gt;Restarting the web server does not restart detached lsphp workers.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;apt update &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;sudo &lt;/span&gt;apt &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;--only-upgrade&lt;/span&gt; &lt;span class="s1"&gt;'lsphp8*'&lt;/span&gt;   &lt;span class="c"&gt;# Debian/Ubuntu&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;dnf upgrade &lt;span class="s1"&gt;'lsphp*'&lt;/span&gt;                                      &lt;span class="c"&gt;# RHEL family&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;pkill lsphp
apt-cache policy lsphp83    &lt;span class="c"&gt;# check the candidate version first&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the day, some LiteSpeed repositories were still serving old builds for hours. Check the candidate version before you assume the update did anything. Update OPcache together with the core; a new core with an old OPcache module is an incomplete fix. If your control panel runs its own lsphp binary for the panel UI, that one needs updating too.&lt;/p&gt;

&lt;h3&gt;
  
  
  Docker official images
&lt;/h3&gt;

&lt;p&gt;All &lt;code&gt;php:8.x.y&lt;/code&gt; official images are fixed on amd64 and arm64. The &lt;code&gt;8.5.12RC1&lt;/code&gt; and &lt;code&gt;8.4.27RC1&lt;/code&gt; tags are &lt;strong&gt;not&lt;/strong&gt; the fix. Pull the new tag, rebuild your images with &lt;code&gt;docker build --pull&lt;/code&gt;, and recreate the containers. On Alpine, 3.24 has fixed 8.3/8.4/8.5 (and edge), 3.24 has no 8.2, and older Alpine releases will not get the fix.&lt;/p&gt;

&lt;h3&gt;
  
  
  Windows and macOS
&lt;/h3&gt;

&lt;p&gt;windows.php.net lists fixed builds for all four branches; restart whatever serves PHP (IIS FastCGI, Apache, php-cgi). Homebrew has all four: &lt;code&gt;brew upgrade php&lt;/code&gt;, then restart the services.&lt;/p&gt;

&lt;h3&gt;
  
  
  FrankenPHP, long-running workers, queues
&lt;/h3&gt;

&lt;p&gt;FrankenPHP embeds PHP, so &lt;code&gt;apt&lt;/code&gt;/&lt;code&gt;dnf&lt;/code&gt; will not update it; you need a new FrankenPHP release or image. Restart queue workers and any long-running PHP server process. A "reload" command inside a framework does not reload the PHP binary.&lt;/p&gt;

&lt;h2&gt;
  
  
  Restart and verify
&lt;/h2&gt;

&lt;p&gt;This is where most "patched" servers are still vulnerable.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# PHP-FPM&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart php8.3-fpm        &lt;span class="c"&gt;# RHEL: php-fpm&lt;/span&gt;

&lt;span class="c"&gt;# Apache mod_php: full restart. A graceful reload kept the old libphp running.&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart apache2           &lt;span class="c"&gt;# or httpd&lt;/span&gt;

&lt;span class="c"&gt;# Any process still mapping deleted (old) libraries?&lt;/span&gt;
&lt;span class="nb"&gt;sudo grep&lt;/span&gt; &lt;span class="nt"&gt;-lE&lt;/span&gt; &lt;span class="s1"&gt;' /(usr|lib|opt).*\(deleted\)$'&lt;/span&gt; /proc/[0-9]&lt;span class="k"&gt;*&lt;/span&gt;/maps
&lt;span class="c"&gt;# Empty output = no process running the old binary.&lt;/span&gt;

&lt;span class="c"&gt;# Debian/Ubuntu&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;needrestart &lt;span class="nt"&gt;-b&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; l
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;php -v&lt;/code&gt; only shows the CLI. For the web side, call a temporary &lt;code&gt;&amp;lt;?php echo PHP_VERSION;&lt;/code&gt; page over HTTP for every PHP branch you run and &lt;strong&gt;delete it immediately&lt;/strong&gt;, or read &lt;code&gt;X-Powered-By&lt;/code&gt; if you have it enabled. On distro packages, the package revision is what counts.&lt;/p&gt;

&lt;h2&gt;
  
  
  Temporary mitigations and their limits
&lt;/h2&gt;

&lt;p&gt;The PHP project is clear: updating is the only complete fix, and the 109619 advisory says there is no complete general configuration mitigation. If you cannot patch for a few hours, these reduce exposure. Set them in &lt;code&gt;php.ini&lt;/code&gt; or the FPM pool with &lt;code&gt;php_admin_value&lt;/code&gt; / &lt;code&gt;php_admin_flag&lt;/code&gt;. &lt;code&gt;ini_set()&lt;/code&gt; and &lt;code&gt;.user.ini&lt;/code&gt; run too late. Restart workers afterwards and check the effective value with a real request.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mitigation&lt;/th&gt;
&lt;th&gt;What it covers&lt;/th&gt;
&lt;th&gt;Limits&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;session.upload_progress.enabled = 0&lt;/code&gt; (default is On)&lt;/td&gt;
&lt;td&gt;The upload-progress path of 109619&lt;/td&gt;
&lt;td&gt;Does not fix the OOM bug (109618). The 8.5 backtrace path remains&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;fatal_error_backtraces = 0&lt;/code&gt; on 8.5&lt;/td&gt;
&lt;td&gt;The 8.5 path of 109619&lt;/td&gt;
&lt;td&gt;8.5 only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;memory_limit = -1&lt;/code&gt;, not overridable by the app&lt;/td&gt;
&lt;td&gt;Removes the predictable limit 109618 relies on&lt;/td&gt;
&lt;td&gt;Partial. One request can eat all server memory. A container/cgroup limit is an availability guard, not a fix&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Do &lt;strong&gt;not&lt;/strong&gt; use &lt;code&gt;RLIMIT_AS&lt;/code&gt; / &lt;code&gt;RLIMIT_DATA&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;These trigger the vulnerable &lt;code&gt;longjmp()&lt;/code&gt; path&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Low &lt;code&gt;post_max_size&lt;/code&gt;, front-end request size limits, limits on parts, headers, nesting, &lt;code&gt;max_input_vars&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;109618 triggers&lt;/td&gt;
&lt;td&gt;Exploits can stay inside the limits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;pm.max_requests = 1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Breaks the producer/consumer chain on one worker&lt;/td&gt;
&lt;td&gt;Large performance cost. Not a fix&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Block &lt;code&gt;multipart/form-data&lt;/code&gt; if the app does not need uploads&lt;/td&gt;
&lt;td&gt;The known trigger path&lt;/td&gt;
&lt;td&gt;Breaks uploads, some payment and contact forms, admin image upload. Test&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Take the site offline / static page / GET and HEAD only&lt;/td&gt;
&lt;td&gt;Safest stopgap (CISA, NSA, NCSC, BSI, CERT-EU)&lt;/td&gt;
&lt;td&gt;An application "maintenance mode" does not help, PHP still parses the request&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;WAF rules&lt;/td&gt;
&lt;td&gt;Known exploit patterns&lt;/td&gt;
&lt;td&gt;Only covers traffic that passes through the WAF. If the origin is reachable directly, it does nothing. Attackers are changing payloads&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;; php.ini
&lt;/span&gt;&lt;span class="py"&gt;session.upload_progress.enabled&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;0&lt;/span&gt;
&lt;span class="c"&gt;; PHP 8.5 only
&lt;/span&gt;&lt;span class="py"&gt;fatal_error_backtraces&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;0&lt;/span&gt;

&lt;span class="c"&gt;; FPM pool
&lt;/span&gt;&lt;span class="err"&gt;php_admin_flag&lt;/span&gt;&lt;span class="nn"&gt;[session.upload_progress.enabled]&lt;/span&gt; &lt;span class="err"&gt;=&lt;/span&gt; &lt;span class="err"&gt;off&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Blocking multipart at the edge:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$http_content_type&lt;/span&gt; &lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="sr"&gt;*&lt;/span&gt; &lt;span class="s"&gt;"multipart/form-data")&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="kn"&gt;return&lt;/span&gt; &lt;span class="mi"&gt;403&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight apache"&gt;&lt;code&gt;&lt;span class="nc"&gt;RewriteEngine&lt;/span&gt; &lt;span class="ss"&gt;On&lt;/span&gt;
&lt;span class="nc"&gt;RewriteCond&lt;/span&gt; %{HTTP:Content-Type} multipart/form-data [NC]
&lt;span class="nc"&gt;RewriteRule&lt;/span&gt; ^ - [F]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cloudflare turned on managed WAF rules for all plans, including free, from 03:02 UTC on 21 October. Paid customers should confirm the rules are in &lt;strong&gt;block&lt;/strong&gt; mode. Some hosting-side rule sets only log on certain web servers instead of blocking, so read your vendor's note for your stack.&lt;/p&gt;

&lt;p&gt;Avoid unofficial "fix" plugins and one-line scripts. They are not reviewed, and vendors advise against them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Was I hit? A checklist
&lt;/h2&gt;

&lt;p&gt;There is no official IOC list (file hashes, IP ranges) yet. These are behavioural signals. Finding nothing does not prove nothing happened, because older logs may not have recorded the relevant error levels. &lt;strong&gt;Copy your logs before you touch anything.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Logs&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Unusual &lt;code&gt;POST&lt;/code&gt; requests with &lt;code&gt;multipart/form-data&lt;/code&gt; since 20 October, especially to URLs that have no upload form&lt;/li&gt;
&lt;li&gt;[ ] "Allowed memory size exhausted" errors and bursts of HTTP 500 right after them&lt;/li&gt;
&lt;li&gt;[ ] Unusually large or garbled &lt;code&gt;PHPSESSID&lt;/code&gt; cookies in responses (the one documented observable sign of the memory leak)&lt;/li&gt;
&lt;li&gt;[ ] Unexplained worker crashes (&lt;code&gt;child exited on signal 11&lt;/code&gt;, segfaults)&lt;/li&gt;
&lt;li&gt;[ ] WAF events matching the vendor's rule IDs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Processes (strongest signal)&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] &lt;code&gt;sh&lt;/code&gt;, &lt;code&gt;bash&lt;/code&gt;, &lt;code&gt;python&lt;/code&gt;, &lt;code&gt;perl&lt;/code&gt;, &lt;code&gt;curl&lt;/code&gt;, &lt;code&gt;wget&lt;/code&gt;, &lt;code&gt;nc&lt;/code&gt; started by &lt;code&gt;php-fpm&lt;/code&gt;, &lt;code&gt;php-cgi&lt;/code&gt;, &lt;code&gt;lsphp&lt;/code&gt;, &lt;code&gt;httpd&lt;/code&gt; or &lt;code&gt;apache2&lt;/code&gt; (on Windows: &lt;code&gt;cmd.exe&lt;/code&gt;, &lt;code&gt;powershell.exe&lt;/code&gt;, &lt;code&gt;certutil.exe&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;[ ] Command lines with &lt;code&gt;/dev/tcp/&lt;/code&gt;, &lt;code&gt;mkfifo&lt;/code&gt;, &lt;code&gt;fsockopen&lt;/code&gt;, or &lt;code&gt;whoami&lt;/code&gt;, &lt;code&gt;id&lt;/code&gt;, &lt;code&gt;uname -a&lt;/code&gt; from the web user&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Filesystem&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# PHP files created or modified since the leak (adjust the web root)&lt;/span&gt;
find /var/www &lt;span class="nt"&gt;-name&lt;/span&gt; &lt;span class="s1"&gt;'*.php'&lt;/span&gt; &lt;span class="nt"&gt;-newermt&lt;/span&gt; &lt;span class="s1"&gt;'2026-10-20'&lt;/span&gt; &lt;span class="nt"&gt;-type&lt;/span&gt; f
&lt;span class="c"&gt;# PHP files inside upload directories&lt;/span&gt;
find /var/www &lt;span class="nt"&gt;-path&lt;/span&gt; &lt;span class="s1"&gt;'*upload*'&lt;/span&gt; &lt;span class="nt"&gt;-name&lt;/span&gt; &lt;span class="s1"&gt;'*.ph*'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;[ ] Short or obfuscated files with &lt;code&gt;eval(&lt;/code&gt;, &lt;code&gt;base64_decode(&lt;/code&gt;, &lt;code&gt;gzinflate(&lt;/code&gt;, &lt;code&gt;assert(&lt;/code&gt;, &lt;code&gt;system(&lt;/code&gt;, &lt;code&gt;passthru(&lt;/code&gt;, &lt;code&gt;shell_exec(&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;[ ] Executables or PHP files in &lt;code&gt;/tmp&lt;/code&gt; and &lt;code&gt;/dev/shm&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;[ ] New cron jobs, new admin accounts (system, panel, CMS), changed &lt;code&gt;authorized_keys&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;[ ] New &lt;code&gt;.htaccess&lt;/code&gt;, &lt;code&gt;.user.ini&lt;/code&gt; or &lt;code&gt;auto_prepend_file&lt;/code&gt; entries&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;If anything looks wrong, assume compromise&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Isolate the server, keep web and PHP logs&lt;/li&gt;
&lt;li&gt;[ ] Rotate secrets, because of the memory leak: database passwords, API keys in &lt;code&gt;.env&lt;/code&gt;, payment provider keys, SMTP passwords, JWT/app secrets, OAuth tokens, TLS private keys (reissue and revoke)&lt;/li&gt;
&lt;li&gt;[ ] Invalidate all user sessions, change admin passwords, enforce 2FA&lt;/li&gt;
&lt;li&gt;[ ] Rebuild from a clean backup if needed&lt;/li&gt;
&lt;li&gt;[ ] Check your legal breach-notification deadlines if personal data was involved (72 hours under GDPR, and the same under Turkey's KVKK)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Still on PHP 8.1 or older?
&lt;/h2&gt;

&lt;p&gt;CISA's guidance is blunt: if you cannot patch now, &lt;strong&gt;or you run PHP 8.1 or older&lt;/strong&gt;, take those systems offline until they are patched or upgraded. A practical order:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Today:&lt;/strong&gt; inventory every PHP runtime: servers, containers, staging, the control panel's own PHP, appliances with embedded PHP.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Within hours:&lt;/strong&gt; apply a vendor backport if one exists (Ubuntu 22.04's 8.1, Ubuntu Pro ESM, cPanel &lt;code&gt;ea-php74/80/81&lt;/code&gt;, DirectAdmin 7.4/8.0/8.1, the Sury repository, Plesk builds from 21 October). If none exists, take it offline or put it behind a VPN. At minimum: disable upload progress, block multipart, and use a WAF, knowing all three are partial.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Days:&lt;/strong&gt; move the application to a supported branch, ideally 8.3 or 8.4. Test on staging first: removed functions, plugin compatibility, and whether any code encoder you rely on supports the new branch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Software that cannot move:&lt;/strong&gt; turn it into a static archive or make it reachable only from the internal network.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long term:&lt;/strong&gt; a supported-branch policy (each PHP branch gets two years of active support plus two years of security fixes), automatic security updates, a patch procedure that includes restarting workers, and tracking vendors who embed PHP in their products.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Two scams riding on this
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Fake "$PHP2SHELL" token.&lt;/strong&gt; Fake memecoin promotions using the PHP2Shell name are circulating on X. Pushing a fake token through compromised or impersonating accounts is a known pattern. There is no token. Do not connect a wallet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fake proof-of-concept repositories.&lt;/strong&gt; GitHub and exploit aggregators have many "PHP2Shell PoC" repos. None has been confirmed as working by the PHP security team, Huntress or The Hacker News. Repos like this are a known way to deliver malware to researchers and admins. If you must look, do it in an isolated VM, never on a server.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Exploitation status (as of 22 October)
&lt;/h2&gt;

&lt;p&gt;Reports disagree. Cloudflare and watchTowr reported exploitation attempts against their sensors and attackers modifying payloads to get around rules. Patchstack's own telemetry showed probing, not breaches. CISA, BSI and Huntress had not confirmed exploitation as of the evening of 21 October, and the PHP security team found no working public exploit at release time. Neither CVE was in the KEV catalogue at that point. Patch details and the diff are public, so this can change quickly. Treat unpatched servers as exposed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;PHP advisory CVE-2026-109618: &lt;a href="https://github.com/php/php-src/security/advisories/GHSA-w8hw-w9fx-j79w" rel="noopener noreferrer"&gt;https://github.com/php/php-src/security/advisories/GHSA-w8hw-w9fx-j79w&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;PHP advisory CVE-2026-109619: &lt;a href="https://github.com/php/php-src/security/advisories/GHSA-6wmf-4rv5-p3rp" rel="noopener noreferrer"&gt;https://github.com/php/php-src/security/advisories/GHSA-6wmf-4rv5-p3rp&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;PHP Foundation: &lt;a href="https://thephp.foundation/blog/2026/10/21/todays-php-security-release-and-how-were-strengthening-php-security/" rel="noopener noreferrer"&gt;https://thephp.foundation/blog/2026/10/21/todays-php-security-release-and-how-were-strengthening-php-security/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;php.net release notes: &lt;a href="https://www.php.net/archive/2026.php#2026-10-21-5" rel="noopener noreferrer"&gt;https://www.php.net/archive/2026.php#2026-10-21-5&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CISA: &lt;a href="https://www.cisa.gov/news-events/alerts/2026/10/21/critical-vulnerabilities-php-require-immediate-patching" rel="noopener noreferrer"&gt;https://www.cisa.gov/news-events/alerts/2026/10/21/critical-vulnerabilities-php-require-immediate-patching&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;UK NCSC: &lt;a href="https://www.ncsc.gov.uk/news/critical-security-issue-affecting-php" rel="noopener noreferrer"&gt;https://www.ncsc.gov.uk/news/critical-security-issue-affecting-php&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CERT-EU 2026-018: &lt;a href="https://cert.europa.eu/publications/security-advisories/2026-018/" rel="noopener noreferrer"&gt;https://cert.europa.eu/publications/security-advisories/2026-018/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;BSI: &lt;a href="https://www.bsi.bund.de/SharedDocs/Cybersicherheitswarnungen/DE/2026/2026-297981-1032.html" rel="noopener noreferrer"&gt;https://www.bsi.bund.de/SharedDocs/Cybersicherheitswarnungen/DE/2026/2026-297981-1032.html&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Red Hat CVE page: &lt;a href="https://access.redhat.com/security/cve/cve-2026-109618" rel="noopener noreferrer"&gt;https://access.redhat.com/security/cve/cve-2026-109618&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;AlmaLinux: &lt;a href="https://almalinux.org/blog/2026-10-21-php2shell-cve-2026-109618-cve-2026-109619/" rel="noopener noreferrer"&gt;https://almalinux.org/blog/2026-10-21-php2shell-cve-2026-109618-cve-2026-109619/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Plesk: &lt;a href="https://support.plesk.com/hc/en-us/articles/44397729587351-Vulnerability-CVE-2026-109618-and-CVE-2026-109619-Remote-code-execution-in-PHP-shipped-with-Plesk" rel="noopener noreferrer"&gt;https://support.plesk.com/hc/en-us/articles/44397729587351-Vulnerability-CVE-2026-109618-and-CVE-2026-109619-Remote-code-execution-in-PHP-shipped-with-Plesk&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;DirectAdmin forum: &lt;a href="https://forum.directadmin.com/threads/critical-php-updates.82726/" rel="noopener noreferrer"&gt;https://forum.directadmin.com/threads/critical-php-updates.82726/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Cloudflare: &lt;a href="https://blog.cloudflare.com/php-vulnerabilities-waf-protection" rel="noopener noreferrer"&gt;https://blog.cloudflare.com/php-vulnerabilities-waf-protection&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Wiz: &lt;a href="https://www.wiz.io/blog/php2shell-php-rce" rel="noopener noreferrer"&gt;https://www.wiz.io/blog/php2shell-php-rce&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Sansec: &lt;a href="https://sansec.io/research/php2shell" rel="noopener noreferrer"&gt;https://sansec.io/research/php2shell&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Huntress: &lt;a href="https://www.huntress.com/blog/php2shell-leaked-critical-php-vulnerabilities" rel="noopener noreferrer"&gt;https://www.huntress.com/blog/php2shell-leaked-critical-php-vulnerabilities&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Patchstack: &lt;a href="https://patchstack.com/articles/php-major-vulnerability-alert-10-2026/" rel="noopener noreferrer"&gt;https://patchstack.com/articles/php-major-vulnerability-alert-10-2026/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Per-distro patch status: &lt;a href="https://ma.ttias.be/php-cve-2026-109618-patch/" rel="noopener noreferrer"&gt;https://ma.ttias.be/php-cve-2026-109618-patch/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Tenable FAQ: &lt;a href="https://www.tenable.com/blog/php2shell-cve-2026-109618-cve-2026-109619-frequently-asked-questions-about-critical-php-remote" rel="noopener noreferrer"&gt;https://www.tenable.com/blog/php2shell-cve-2026-109618-cve-2026-109619-frequently-asked-questions-about-critical-php-remote&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A Turkish summary of this incident is on our company blog at Alesta WEB (&lt;a href="https://alestaweb.com" rel="noopener noreferrer"&gt;alestaweb.com&lt;/a&gt;). If you spot an error or a vendor update I missed, leave a comment and I will correct the post.&lt;/p&gt;

</description>
      <category>php</category>
      <category>security</category>
      <category>webdev</category>
      <category>devops</category>
    </item>
    <item>
      <title>My photo.jpg Was a 29-Byte HTML Page: Validating Downloaded Images Before They Reach Production</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Fri, 16 Oct 2026 07:26:24 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/my-photojpg-was-a-29-byte-html-page-validating-downloaded-images-before-they-reach-production-416p</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/my-photojpg-was-a-29-byte-html-page-validating-downloaded-images-before-they-reach-production-416p</guid>
      <description>&lt;p&gt;We publish a handful of technical articles every day on our company blog, and every article gets two real photographs: a cover and one inline image. The pipeline is boring on purpose. Pick photos, download them with &lt;code&gt;curl&lt;/code&gt;, copy them to the server, insert the article, check the page.&lt;/p&gt;

&lt;p&gt;One morning every file in the batch downloaded "successfully". &lt;code&gt;curl&lt;/code&gt; exited with 0 and the files had the right names and the &lt;code&gt;.jpg&lt;/code&gt; extension. Then a quick look at the folder showed something was off.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; foto/
&lt;span class="go"&gt;-rw-r--r-- 1 user user 29  cover-1.jpg
-rw-r--r-- 1 user user 29  cover-2.jpg
&lt;/span&gt;&lt;span class="c"&gt;...
&lt;/span&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;file foto/&lt;span class="k"&gt;*&lt;/span&gt;
&lt;span class="go"&gt;foto/cover-1.jpg: HTML document, ASCII text
foto/cover-2.jpg: HTML document, ASCII text
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Twenty-nine bytes. Not a photo, an error page with a &lt;code&gt;.jpg&lt;/code&gt; name. This post covers why it happened and the checks we added so that a file like that can't reach a live page again.&lt;/p&gt;

&lt;h2&gt;
  
  
  What went wrong: an ID that looks like a file name
&lt;/h2&gt;

&lt;p&gt;We take photos from Unsplash, whose license allows free commercial use. Each photo has a short public ID that appears in its page URL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://unsplash.com/photos/4Ya_nyKge5k
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The image files themselves live on a CDN, with URLs like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https://images.unsplash.com/photo-1542831371-29b0f74f9713?w=1600&amp;amp;h=900&amp;amp;fit=crop
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Our script built the CDN URL from the short ID:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-s&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; cover.jpg &lt;span class="s2"&gt;"https://images.unsplash.com/photo-&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;ID&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;?w=1600&amp;amp;h=900&amp;amp;fit=crop&amp;amp;fm=jpg"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It looked right, and for a while we never noticed, because we had been copying CDN URLs by hand. The first day the script used the short IDs directly, every request returned a 404 with a tiny HTML body. The short public ID (&lt;code&gt;4Ya_nyKge5k&lt;/code&gt;) and the storage key (&lt;code&gt;1542831371-29b0f74f9713&lt;/code&gt;) are two different identifiers. One is for people and URLs, the other is for the file store. Nothing in the format tells you that.&lt;/p&gt;

&lt;p&gt;The fix for the URL itself is to get the real file URL from a source that knows it. With Unsplash's official API, every photo object has a &lt;code&gt;urls&lt;/code&gt; block (&lt;code&gt;raw&lt;/code&gt;, &lt;code&gt;full&lt;/code&gt;, &lt;code&gt;regular&lt;/code&gt;...) pointing at the CDN, so you never construct it yourself. (If you use the API, its guidelines also ask you to call the photo's &lt;code&gt;download_location&lt;/code&gt; endpoint when someone downloads it.) For one-off manual work, the photo's download link redirects to the real file, and the &lt;code&gt;Location&lt;/code&gt; header shows the storage key:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-sI&lt;/span&gt; &lt;span class="s2"&gt;"https://unsplash.com/photos/&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;ID&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/download"&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-i&lt;/span&gt; &lt;span class="s1"&gt;'^location:'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;But the URL was only half the bug. The other half was that a 404 turned into a "successful" download. That part has nothing to do with Unsplash.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why curl said everything was fine
&lt;/h2&gt;

&lt;p&gt;By default &lt;code&gt;curl&lt;/code&gt; doesn't treat an HTTP error status as a failure. A 404 or 500 is a complete, valid HTTP response. &lt;code&gt;curl&lt;/code&gt; saves the body and exits with 0. With &lt;code&gt;-s&lt;/code&gt; there's no progress output either, so there's nothing on screen to look at.&lt;/p&gt;

&lt;p&gt;There are three things worth adding to almost every download in a script.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Make HTTP errors fail.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-fsS&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; cover.jpg &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$URL&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;          &lt;span class="c"&gt;# -f: exit 22 on HTTP &amp;gt;= 400, keep no body&lt;/span&gt;
curl &lt;span class="nt"&gt;--fail-with-body&lt;/span&gt; &lt;span class="nt"&gt;-sS&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; cover.jpg &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$URL&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;   &lt;span class="c"&gt;# curl 7.76+: fail, but keep the body for debugging&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;-S&lt;/code&gt; brings back error messages that &lt;code&gt;-s&lt;/code&gt; would hide. With &lt;code&gt;set -e&lt;/code&gt; in the script, a 404 now stops the batch.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Look at what the server says it sent.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-r&lt;/span&gt; code &lt;span class="nb"&gt;type&lt;/span&gt; &amp;lt; &amp;lt;&lt;span class="o"&gt;(&lt;/span&gt;curl &lt;span class="nt"&gt;-sS&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; cover.jpg &lt;span class="nt"&gt;-w&lt;/span&gt; &lt;span class="s1"&gt;'%{http_code} %{content_type}\n'&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$URL&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$code&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; 200 &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;[[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$type&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; image/&lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
  &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"bad download: &lt;/span&gt;&lt;span class="nv"&gt;$code&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$type&lt;/span&gt;&lt;span class="s2"&gt; (&lt;/span&gt;&lt;span class="nv"&gt;$URL&lt;/span&gt;&lt;span class="s2"&gt;)"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2
  &lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;-w&lt;/code&gt; prints transfer details after the download. Checking the content type catches the case where the status is 200 but the body is a login page, a consent page or a bot check. That last one is becoming common: a CDN or WAF answers an automated client with a "checking your browser" page and a 200 or 403.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Look at the bytes, not the name.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The extension is a label you chose. The first bytes of the file are what it actually is. A JPEG starts with &lt;code&gt;FF D8 FF&lt;/code&gt;, a PNG with &lt;code&gt;89 50 4E 47&lt;/code&gt;, WebP with &lt;code&gt;RIFF....WEBP&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;file &lt;span class="nt"&gt;--mime-type&lt;/span&gt; &lt;span class="nt"&gt;-b&lt;/span&gt; cover.jpg     &lt;span class="c"&gt;# image/jpeg ... or text/html&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  A small validator we now run on every image
&lt;/h2&gt;

&lt;p&gt;The shell checks catch failed downloads. Before publishing we also want to know the image is complete and has the size the page expects. This is the Python version we run over the folder before anything is uploaded:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;pathlib&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Path&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;PIL&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Image&lt;/span&gt;  &lt;span class="c1"&gt;# pip install pillow
&lt;/span&gt;
&lt;span class="n"&gt;EXPECTED&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-kapak.jpg&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1600&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;900&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;-1.jpg&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;675&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
&lt;span class="n"&gt;MIN_BYTES&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10_000&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;
    &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="n"&gt;data&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_bytes&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;MIN_BYTES&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;too small (&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; bytes)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;b&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\xff\xd8\xff&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;not a JPEG (starts with &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;!r}&lt;/span&gt;&lt;span class="s"&gt;)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;Image&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;im&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;im&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;verify&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;              &lt;span class="c1"&gt;# structure check, catches truncated files
&lt;/span&gt;        &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;Image&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;im&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;im&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;load&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;                &lt;span class="c1"&gt;# full decode
&lt;/span&gt;            &lt;span class="n"&gt;size&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;im&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cannot decode: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;suffix&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;want&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;EXPECTED&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;endswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;suffix&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;size&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="n"&gt;want&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;size &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;size&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;, expected &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;want&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt;

&lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]).&lt;/span&gt;&lt;span class="nf"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;*.jpg&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)):&lt;/span&gt;
    &lt;span class="n"&gt;issues&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;issues&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;FAIL &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;; &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;issues&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;bad&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; bad file(s)&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;bad&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few notes from using it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;verify()&lt;/code&gt; has to be followed by a fresh &lt;code&gt;open()&lt;/code&gt;. After &lt;code&gt;verify()&lt;/code&gt; the image object can't be used for anything else.&lt;/li&gt;
&lt;li&gt;The size threshold is crude, but useful. A 1600x900 photo at quality 80 is usually well above 100 KB. Anything under 10 KB is almost always an error page, a placeholder, or a blank image.&lt;/li&gt;
&lt;li&gt;Checking dimensions catches CDN parameters that were ignored. If &lt;code&gt;fit=crop&lt;/code&gt; doesn't apply, you get the original aspect ratio, and the layout jumps.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Then check the live URL, not the local file
&lt;/h2&gt;

&lt;p&gt;A correct file on disk can still be broken on the page. Since that morning we have hit two more ways for a good image to disappear after upload.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Line endings in file names.&lt;/strong&gt; A file list produced by a Windows script and consumed by a bash loop carried a trailing &lt;code&gt;\r&lt;/code&gt; into every file name. The files were on the server, &lt;code&gt;ls&lt;/code&gt; showed them, and every URL returned 404, because the real name was &lt;code&gt;cover.jpg\r&lt;/code&gt;. This finds them:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;ls&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;$'&lt;/span&gt;&lt;span class="se"&gt;\r&lt;/span&gt;&lt;span class="s1"&gt;'&lt;/span&gt;      &lt;span class="c"&gt;# should print 0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;The image goes through more than one URL.&lt;/strong&gt; Our site serves images through an optimizing loader endpoint that resizes on the fly, not the raw file. The raw file URL can be 200 while the resized version fails. So the post-publish check requests both, and a headless browser loads the article and counts images that didn't decode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Playwright, after page.goto(...)&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;broken&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;page&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;evaluate&lt;/span&gt;&lt;span class="p"&gt;(()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
  &lt;span class="p"&gt;[...&lt;/span&gt;&lt;span class="nb"&gt;document&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;querySelectorAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;main img&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;img&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;img&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;complete&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;img&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;naturalWidth&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;naturalWidth === 0&lt;/code&gt; on a completed image means the browser got something it couldn't decode. That's exactly what a 29-byte HTML file served as &lt;code&gt;image/jpeg&lt;/code&gt; looks like.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;An HTTP error is not a curl error. Use &lt;code&gt;-f&lt;/code&gt; or &lt;code&gt;--fail-with-body&lt;/code&gt;, and &lt;code&gt;-S&lt;/code&gt; with &lt;code&gt;-s&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Check &lt;code&gt;%{http_code}&lt;/code&gt; and &lt;code&gt;%{content_type}&lt;/code&gt; for every download in a script. Bot-check pages often come back with a 200.&lt;/li&gt;
&lt;li&gt;Trust magic bytes, not extensions: &lt;code&gt;file --mime-type&lt;/code&gt;, or the first three bytes of a JPEG.&lt;/li&gt;
&lt;li&gt;Decode the image before publishing it. Pillow's &lt;code&gt;verify()&lt;/code&gt; plus &lt;code&gt;load()&lt;/code&gt; catches truncated and fake files.&lt;/li&gt;
&lt;li&gt;A public ID in a URL isn't necessarily the storage key. Take file URLs from the API response instead of building them.&lt;/li&gt;
&lt;li&gt;After upload, test the URL the visitor actually loads, including any resizing endpoint, and count undecodable images in a real browser.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this is clever. It is the set of checks we wish we had from the first day. We run this pipeline for the blog at Alesta WEB (&lt;a href="https://alestaweb.com" rel="noopener noreferrer"&gt;alestaweb.com&lt;/a&gt;). The morning it failed, nothing reached a reader, mostly because a check looked at the bytes and not the exit code.&lt;/p&gt;

</description>
      <category>webdev</category>
      <category>bash</category>
      <category>python</category>
      <category>automation</category>
    </item>
    <item>
      <title>Instant ADD COLUMN, Until a FULLTEXT Index Shows Up: A MariaDB Migration That Froze Writes</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Mon, 12 Oct 2026 20:21:16 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/instant-add-column-until-a-fulltext-index-shows-up-a-mariadb-migration-that-froze-writes-3ni3</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/instant-add-column-until-a-fulltext-index-shows-up-a-mariadb-migration-that-froze-writes-3ni3</guid>
      <description>&lt;p&gt;Adding a nullable column to an InnoDB table is supposed to be a non-event. On MariaDB 10.3+ and MySQL 8.0+ it is usually an &lt;em&gt;instant&lt;/em&gt; operation: a metadata change, no data copied, done in milliseconds no matter how big the table is.&lt;/p&gt;

&lt;p&gt;Last week we shipped a release of a news CMS that added a few columns to its articles table. On most installations the migration finished before anyone noticed. On the larger ones, the article table (a few hundred MB) was rebuilt from scratch, the release took minutes instead of seconds, and every page view that tried to bump a view counter sat waiting until the rebuild finished.&lt;/p&gt;

&lt;p&gt;The difference between the two groups was one line in the table definition:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="n"&gt;FULLTEXT&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="n"&gt;ft_search&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This post is the short version of what happened, a reproduction you can run yourself, and the two habits that would have made it a non-event.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The schema looked roughly like this (names simplified):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;articles&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt;     &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="n"&gt;AUTO_INCREMENT&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;title&lt;/span&gt;  &lt;span class="nb"&gt;VARCHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="n"&gt;body&lt;/span&gt;   &lt;span class="nb"&gt;TEXT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;hits&lt;/span&gt;   &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="c1"&gt;-- ...&lt;/span&gt;
  &lt;span class="n"&gt;FULLTEXT&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="n"&gt;ft_search&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;ENGINE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;InnoDB&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The migration was written the way most migrations are written, one statement per change:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;articles&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;source_url&lt;/span&gt; &lt;span class="nb"&gt;VARCHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;255&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;articles&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;is_sponsored&lt;/span&gt; &lt;span class="nb"&gt;TINYINT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;articles&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;updated_by&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every article page runs something like &lt;code&gt;UPDATE articles SET hits = hits + 1 WHERE id = ?&lt;/code&gt;. That detail matters later.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproducing it
&lt;/h2&gt;

&lt;p&gt;I rebuilt the situation on a local MariaDB instance with two identical tables of 200,000 rows each (about 320 MB). The only difference: &lt;code&gt;a1&lt;/code&gt; has a FULLTEXT index, &lt;code&gt;a2&lt;/code&gt; does not.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;a1&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="n"&gt;AUTO_INCREMENT&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="nb"&gt;VARCHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="nb"&gt;TEXT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hits&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;FULLTEXT&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="n"&gt;ft&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;ENGINE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;InnoDB&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;a2&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="n"&gt;AUTO_INCREMENT&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;title&lt;/span&gt; &lt;span class="nb"&gt;VARCHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="nb"&gt;TEXT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hits&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;ENGINE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;InnoDB&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;First, ask the server what it is willing to do instead of letting it choose:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;a2&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;c1&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ALGORITHM&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;INSTANT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- Query OK&lt;/span&gt;

&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;a1&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;c1&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ALGORITHM&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;INSTANT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation.&lt;/span&gt;
&lt;span class="c1"&gt;-- Try ALGORITHM=INPLACE&lt;/span&gt;

&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;a1&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;c1&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ALGORITHM&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;INPLACE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;LOCK&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;NONE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- ERROR 1846 (0A000): LOCK=NONE is not supported.&lt;/span&gt;
&lt;span class="c1"&gt;-- Reason: Fulltext index creation requires a lock. Try LOCK=SHARED&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That second error is the whole story. With a FULLTEXT index on the table, adding a column is not instant. The table gets rebuilt, and because the full-text index has to be rebuilt along with it, the server needs at least a &lt;strong&gt;shared lock&lt;/strong&gt;: reads keep working, &lt;strong&gt;writes wait&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Then I timed it, with a second connection firing a single counter update one second into each ALTER:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Operation&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Concurrent &lt;code&gt;UPDATE ... hits = hits + 1&lt;/code&gt; waited&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;a1&lt;/code&gt; (FULLTEXT), two columns in &lt;strong&gt;one&lt;/strong&gt; &lt;code&gt;ALTER&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;34.3 s&lt;/td&gt;
&lt;td&gt;33.3 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;a1&lt;/code&gt; (FULLTEXT), two columns in &lt;strong&gt;two&lt;/strong&gt; &lt;code&gt;ALTER&lt;/code&gt;s&lt;/td&gt;
&lt;td&gt;177.1 s total&lt;/td&gt;
&lt;td&gt;44.3 s (during the first one)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;a2&lt;/code&gt; (no FULLTEXT), two columns in one &lt;code&gt;ALTER&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;0.06 s&lt;/td&gt;
&lt;td&gt;–&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Same table size, same columns. Without the FULLTEXT index the change took 60 milliseconds. With it, a single counter update was blocked for more than half a minute, on a laptop with nothing else running.&lt;/p&gt;

&lt;p&gt;On a live site that is not one blocked update. It is every page view during that window, each holding a connection while it waits. Depending on your connection limits and timeouts, that is the difference between "the release was a bit slow" and "the site returned errors for a minute".&lt;/p&gt;

&lt;h2&gt;
  
  
  Why FULLTEXT changes the rules
&lt;/h2&gt;

&lt;p&gt;InnoDB implements full-text search with a set of auxiliary tables and a hidden &lt;code&gt;FTS_DOC_ID&lt;/code&gt; column that ties each row to its entries in the full-text index. The instant &lt;code&gt;ADD COLUMN&lt;/code&gt; path works by changing only metadata and leaving existing rows physically untouched. Tables with a FULLTEXT index are excluded from that path, so the server falls back to rebuilding the table. During the rebuild it also rebuilds the full-text index, and that step can't run with concurrent writes, hence &lt;code&gt;LOCK=SHARED&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;None of this is a bug. It is documented behaviour in both MariaDB and MySQL. The problem is that it is easy to never hit it in development, where the table has a few hundred rows and every ALTER finishes instantly regardless of the algorithm.&lt;/p&gt;

&lt;h2&gt;
  
  
  Mistake 1: one statement per column
&lt;/h2&gt;

&lt;p&gt;Our migration had three &lt;code&gt;ALTER TABLE&lt;/code&gt; statements, which meant three full rebuilds, three times the write freeze. The fix is boring:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;articles&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;source_url&lt;/span&gt;   &lt;span class="nb"&gt;VARCHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;255&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;is_sponsored&lt;/span&gt; &lt;span class="nb"&gt;TINYINT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;updated_by&lt;/span&gt;   &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One statement, one rebuild. In the reproduction, two separate statements took 177 seconds total versus 34 for the combined one. You will see different numbers on your hardware, but the shape holds: rebuild cost scales with the number of statements, not the number of columns.&lt;/p&gt;

&lt;p&gt;This is worth making a rule even for tables without FULLTEXT today. Indexes get added later, by someone else, for a search feature, and your migration habits don't get re-reviewed when that happens.&lt;/p&gt;

&lt;h2&gt;
  
  
  Mistake 2: letting the server pick the algorithm
&lt;/h2&gt;

&lt;p&gt;When you write a bare &lt;code&gt;ALTER TABLE ... ADD COLUMN&lt;/code&gt;, the server silently picks the cheapest algorithm &lt;em&gt;it can use&lt;/em&gt;. If instant isn't available, it quietly does a rebuild. Nothing in the output tells you which one happened, except the time it took.&lt;/p&gt;

&lt;p&gt;Make the expectation explicit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;articles&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;source_url&lt;/span&gt; &lt;span class="nb"&gt;VARCHAR&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;255&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;updated_by&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;ALGORITHM&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;INSTANT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If instant is possible, nothing changes. If it isn't, the statement &lt;strong&gt;fails immediately&lt;/strong&gt; with error 1845 instead of starting a multi-minute rebuild. A failed migration in a release script is annoying; a silent table rebuild on a busy table is an incident. You want the annoying one.&lt;/p&gt;

&lt;p&gt;For a migration that is meant to be online but can't be instant, ask for the lock level you can actually tolerate:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;articles&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ALGORITHM&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;INPLACE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;LOCK&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;NONE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same idea: fail fast if the server can't keep writes flowing, then decide consciously.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do when the rebuild is unavoidable
&lt;/h2&gt;

&lt;p&gt;Once &lt;code&gt;ALGORITHM=INSTANT&lt;/code&gt; has told you "no", you have a few options. None of them is free.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Schedule it.&lt;/strong&gt; Run the single combined ALTER at the quietest hour. For many sites, a 30-second write pause at 4 a.m. is fine, as long as it is expected and short. Put a short &lt;code&gt;lock_wait_timeout&lt;/code&gt; on the session so the ALTER gives up instead of queueing behind a long-running transaction:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;SESSION&lt;/span&gt; &lt;span class="n"&gt;lock_wait_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;articles&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="p"&gt;...,&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="p"&gt;...;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This protects against the other classic: the ALTER waits for a metadata lock held by some long query, and every new query on the table piles up &lt;em&gt;behind the waiting ALTER&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Make the hot writes tolerate it.&lt;/strong&gt; In our case the blocked writes were view counters. They don't need to hit the main table synchronously. Buffering them (a small counter table, a cache, or a queue that is flushed periodically) removes the most frequent write from the critical table. That helps far beyond migrations.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Drop and re-create the FULLTEXT index around the change.&lt;/strong&gt; Drop the index, add the columns (now instant), add the index back. The column change becomes cheap, but re-creating the full-text index is itself a long operation, and search returns nothing until it completes. This only makes sense if search being unavailable for a while is acceptable and writes are not.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Use an online schema change tool.&lt;/strong&gt; &lt;code&gt;pt-online-schema-change&lt;/code&gt; and &lt;code&gt;gh-ost&lt;/code&gt; copy the table in the background and swap it in at the end. They are built for exactly this, but they add operational complexity, need care with triggers and foreign keys, and are overkill for a 300 MB table.&lt;/p&gt;

&lt;h2&gt;
  
  
  A pre-flight check you can add to your release script
&lt;/h2&gt;

&lt;p&gt;Before running migrations against a large installation, list the tables that can't get instant column changes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;DISTINCT&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;data_length&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index_length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;1048576&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;size_mb&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;information_schema&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;statistics&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;information_schema&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tables&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
  &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table_schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table_schema&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;table_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;table_name&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;table_schema&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;index_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'FULLTEXT'&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;size_mb&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a migration touches one of those tables and the table is larger than a few tens of MB, the migration needs a plan, not just a statement.&lt;/p&gt;

&lt;p&gt;And the cheapest test of all: run the migration against a copy of your largest real dataset, not the development seed. Our development database had a few hundred articles. The rebuild there took less time than the HTTP request that triggered it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Instant &lt;code&gt;ADD COLUMN&lt;/code&gt; is the default you get used to, not a guarantee. A FULLTEXT index on the table switches it off.&lt;/li&gt;
&lt;li&gt;Without instant, the table is rebuilt and, because of the full-text index, &lt;strong&gt;writes are blocked&lt;/strong&gt; for the duration (&lt;code&gt;LOCK=SHARED&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Put all column changes for a table into &lt;strong&gt;one&lt;/strong&gt; &lt;code&gt;ALTER TABLE&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Write &lt;code&gt;ALGORITHM=INSTANT&lt;/code&gt; (or &lt;code&gt;LOCK=NONE&lt;/code&gt;) explicitly so impossible expectations fail fast instead of turning into a silent rebuild.&lt;/li&gt;
&lt;li&gt;Test migrations against production-sized data and keep frequent, low-value writes like counters off your largest tables.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We hit this while rolling out a release of the news CMS we build at Alesta WEB (&lt;a href="https://alestaweb.com" rel="noopener noreferrer"&gt;alestaweb.com&lt;/a&gt;). The fix in the end was two lines in the migration. Finding out why the release had stalled on the big sites took considerably longer.&lt;/p&gt;

</description>
      <category>mariadb</category>
      <category>mysql</category>
      <category>database</category>
      <category>devops</category>
    </item>
    <item>
      <title>Green Build, Broken Site: Three Next.js 16 Failures You Only See in Production</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Thu, 01 Oct 2026 12:20:21 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/green-build-broken-site-three-nextjs-16-failures-you-only-see-in-production-134p</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/green-build-broken-site-three-nextjs-16-failures-you-only-see-in-production-134p</guid>
      <description>&lt;p&gt;One Next.js 16 site, one working day, three failures. None of them showed up on the development machine. All three passed the build. Each one broke a single feature quietly while the rest of the site looked fine.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Image uploads in the admin panel returned 422 for every file.&lt;/li&gt;
&lt;li&gt;Files uploaded after the server started returned 404 until the next restart.&lt;/li&gt;
&lt;li&gt;Scheduled posts went live in the database but never appeared on the site.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;They share one root cause: &lt;strong&gt;the environment you test in is not the one you ship.&lt;/strong&gt; Below is each failure, how it showed up, the fix, and the check that would have caught it before a user did.&lt;/p&gt;

&lt;h2&gt;
  
  
  1. The image library that compiled to nothing
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Symptom
&lt;/h3&gt;

&lt;p&gt;The admin upload endpoint re-encodes every image with &lt;code&gt;sharp&lt;/code&gt; (strip metadata, resize, convert to WebP). In production it answered every request with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"hata"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"Görsel işlenemedi"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's "image could not be processed", the message from our own &lt;code&gt;catch&lt;/code&gt; block. The interesting part was what the &lt;code&gt;catch&lt;/code&gt; block was hiding.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why it only happened in production
&lt;/h3&gt;

&lt;p&gt;Locally we built with &lt;code&gt;next build --webpack&lt;/code&gt;, because a font download failed under Turbopack on that Windows machine. The server ran a plain &lt;code&gt;next build&lt;/code&gt;, which in Next.js 16 means &lt;strong&gt;Turbopack&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;In the Turbopack output, the static import&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;sharp&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;sharp&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;had been replaced with &lt;code&gt;(void 0)&lt;/code&gt;. The handler compiled, the route existed, and then the first call failed at runtime.&lt;/p&gt;

&lt;p&gt;The usual advice is to list the package in &lt;code&gt;serverExternalPackages&lt;/code&gt;. &lt;code&gt;sharp&lt;/code&gt; is already on Next's built-in list, and adding it explicitly made no difference in that build.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fix
&lt;/h3&gt;

&lt;p&gt;Resolve the module at runtime, from the app's own &lt;code&gt;node_modules&lt;/code&gt;, so that the bundler has nothing to analyse statically:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;createRequire&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:module&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:path&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="nx"&gt;Sharp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;sharp&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;MODULE_NAME&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;sharp&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// a variable, not a literal: nothing to resolve at build time&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;cached&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Sharp&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;loadSharp&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt; &lt;span class="nx"&gt;Sharp&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;cached&lt;/span&gt; &lt;span class="o"&gt;??=&lt;/span&gt; &lt;span class="nf"&gt;createRequire&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cwd&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;package.json&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))(&lt;/span&gt;&lt;span class="nx"&gt;MODULE_NAME&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nx"&gt;Sharp&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;cached&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The type import keeps full typings, and &lt;code&gt;loadSharp()(buffer).webp().toBuffer()&lt;/code&gt; behaves exactly as before.&lt;/p&gt;

&lt;h3&gt;
  
  
  The check that would have caught it
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Build in CI with the same bundler you deploy with.&lt;/strong&gt; If your local build needs &lt;code&gt;--webpack&lt;/code&gt; to work around something, that is a second build pipeline, and only one of the two runs in production. The upload was tested, but against the webpack build.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Uploads that 404 until the next restart
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Symptom
&lt;/h3&gt;

&lt;p&gt;Once uploads worked, a new one appeared: the API returned a URL like &lt;code&gt;/uploads/logos/abc123.webp&lt;/code&gt;, the file was on disk with correct permissions, and the URL returned 404. After a process restart the same URL worked.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;next start&lt;/code&gt; serves the &lt;code&gt;public/&lt;/code&gt; files it found when the process started. Anything written there afterwards is invisible to the static file handler: the same request goes from 404 to 200 after nothing but a restart. The dev server is more forgiving, which is why nobody noticed.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;public/&lt;/code&gt; is meant for assets that ship with the build, not for user content. Most guides tell you to move uploads to object storage, and that is the right long-term answer. But if you self-host on a single server and want to keep files on local disk, there is a small fix.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fix
&lt;/h3&gt;

&lt;p&gt;Keep writing to &lt;code&gt;public/uploads/&lt;/code&gt;, and add a catch-all route handler that serves from that directory only when the static handler misses:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// app/uploads/[...path]/route.ts&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:path&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;promises&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nx"&gt;fs&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:fs&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;ROOT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;cwd&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;public&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;uploads&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;TYPES&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Record&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;.webp&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;image/webp&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;.png&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;image/png&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;.jpg&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;image/jpeg&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;.jpeg&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;image/jpeg&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;.gif&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;image/gif&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;.avif&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;image/avif&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;

&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;GET&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;_req&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;params&lt;/span&gt; &lt;span class="p"&gt;}:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;params&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;})&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;parts&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;params&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;some&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;p&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;.&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;..&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\\&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;includes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;&lt;span class="s2"&gt;0&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Not found&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;404&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;file&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ROOT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;TYPES&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nx"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;extname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;file&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;toLowerCase&lt;/span&gt;&lt;span class="p"&gt;()];&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="kd"&gt;type&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;file&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startsWith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ROOT&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nx"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;sep&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Not found&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;404&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;

  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;st&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;fs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;stat&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;file&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isFile&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Not found&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;404&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Uint8Array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;fs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;readFile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;file&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="na"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Content-Type&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kd"&gt;type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Content-Length&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;st&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;size&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
        &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Cache-Control&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;public, max-age=31536000, immutable&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;// names are random and never reused&lt;/span&gt;
        &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;X-Content-Type-Options&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;nosniff&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Response&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Not found&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;404&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details matter more than the rest:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Path traversal.&lt;/strong&gt; Reject &lt;code&gt;..&lt;/code&gt; segments &lt;em&gt;and&lt;/em&gt; confirm that the resolved path is still under the root. Either check alone has gaps.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No SVG.&lt;/strong&gt; An SVG can carry script. If you need SVG uploads, serve them as &lt;code&gt;Content-Disposition: attachment&lt;/code&gt; or from another origin.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Immutable caching is only safe because file names are random.&lt;/strong&gt; If your uploads keep their original names and can be overwritten, drop &lt;code&gt;immutable&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Files present at startup are still served by the static handler. The route only handles the rest.&lt;/p&gt;

&lt;h3&gt;
  
  
  The check that would have caught it
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Test upload-then-fetch against a production build, without restarting in between.&lt;/strong&gt; "Upload returns 200" is not the test. "The returned URL serves the bytes" is.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Scheduled posts that published but never appeared
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Symptom
&lt;/h3&gt;

&lt;p&gt;A cron job calls an endpoint every ten minutes. The endpoint flips &lt;code&gt;scheduled&lt;/code&gt; posts whose time has come to &lt;code&gt;published&lt;/code&gt;, then invalidates the &lt;code&gt;posts&lt;/code&gt; cache tag. The database showed the posts as published. The home page, the sidebar and the "latest posts" blocks still showed the old list, indefinitely.&lt;/p&gt;

&lt;p&gt;The cron used &lt;code&gt;curl -s ... &amp;gt; /dev/null&lt;/code&gt;, so its failures went nowhere. The process log had the answer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Error: updateTag can only be called from within a Server Action. To invalidate
cache tags in Route Handlers or other contexts, use revalidateTag instead.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Why
&lt;/h3&gt;

&lt;p&gt;Next.js 16 split tag invalidation in two:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;updateTag(tag)&lt;/code&gt;: &lt;strong&gt;Server Actions only.&lt;/strong&gt; Read-your-own-writes semantics, so the user who submitted the form sees the change immediately.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;revalidateTag(tag, profile)&lt;/code&gt;: works anywhere on the server, &lt;strong&gt;including Route Handlers&lt;/strong&gt;, and now takes a second argument.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The endpoint called &lt;code&gt;updateTag&lt;/code&gt;. The database write happened first, then the call threw, and the cache was never touched. Because the post was already &lt;code&gt;published&lt;/code&gt;, the next cron run found nothing to do, so it never retried. And because data cached with &lt;code&gt;unstable_cache&lt;/code&gt; and no &lt;code&gt;revalidate&lt;/code&gt; time stays until its tag is invalidated, "eventually" never came.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fix
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;revalidateTag&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;next/cache&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;// updateTag only works inside Server Actions; route handlers use revalidateTag.&lt;/span&gt;
&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;count&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nf"&gt;revalidateTag&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;posts&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;expire&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;{ expire: 0 }&lt;/code&gt; expires the entries immediately. The &lt;code&gt;"max"&lt;/code&gt; profile would serve stale data once and refresh in the background. For "the post should be visible now", expire immediately.&lt;/p&gt;

&lt;p&gt;We verified it end to end: inserted a scheduled test post, called the endpoint (a JSON body reporting one published post, instead of a 500), requested a page and saw the post in the sidebar, then deleted the test row.&lt;/p&gt;

&lt;p&gt;One more trap: &lt;strong&gt;deleting a row with raw SQL invalidates nothing.&lt;/strong&gt; After removing the test post, it stayed in the cached sidebar until the tag was invalidated again. Any out-of-band write (a migration, a manual fix, an import script) needs a tag invalidation afterwards, or a way to trigger one.&lt;/p&gt;

&lt;h3&gt;
  
  
  The check that would have caught it
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Don't send cron output to &lt;code&gt;/dev/null&lt;/code&gt;.&lt;/strong&gt; Log the response body, or at least fail loudly on a non-2xx status (&lt;code&gt;curl -fsS&lt;/code&gt;). The endpoint had been returning 500 for every run that had work to do.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the three have in common
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;Dev&lt;/th&gt;
&lt;th&gt;Prod&lt;/th&gt;
&lt;th&gt;What actually differed&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;sharp&lt;/code&gt; was &lt;code&gt;undefined&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;worked&lt;/td&gt;
&lt;td&gt;422&lt;/td&gt;
&lt;td&gt;webpack locally, Turbopack on the server&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;new uploads 404&lt;/td&gt;
&lt;td&gt;worked&lt;/td&gt;
&lt;td&gt;404 until restart&lt;/td&gt;
&lt;td&gt;dev server re-scans &lt;code&gt;public/&lt;/code&gt;, prod doesn't&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;cache never refreshed&lt;/td&gt;
&lt;td&gt;not exercised&lt;/td&gt;
&lt;td&gt;stale forever&lt;/td&gt;
&lt;td&gt;the code path only ran from cron&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;None of these is a Next.js bug in the sense of "the framework is broken". Two are documented behaviours and one is a bundler edge case with a clean workaround. All three survived because the only environment that ran the code path was production, and production was set up to stay quiet about failures.&lt;/p&gt;

&lt;p&gt;The checklist we now run before calling a deploy done:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Build with the production bundler, no flags that the server doesn't use.&lt;/li&gt;
&lt;li&gt;Upload a file, then fetch the returned URL &lt;strong&gt;without restarting&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Run every cron endpoint once by hand and read the response body.&lt;/li&gt;
&lt;li&gt;After any manual database change, invalidate the affected cache tags.&lt;/li&gt;
&lt;/ol&gt;




&lt;p&gt;&lt;em&gt;We build news, e-commerce and restaurant POS software at &lt;a href="https://alestaweb.com" rel="noopener noreferrer"&gt;Alesta WEB&lt;/a&gt;. These notes come from moving our own site to a new stack this week.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>nextjs</category>
      <category>webdev</category>
      <category>javascript</category>
      <category>devops</category>
    </item>
    <item>
      <title>Two Proxies, One Comma, Every Page a 500</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Mon, 28 Sep 2026 06:29:57 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/two-proxies-one-comma-every-page-a-500-1h69</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/two-proxies-one-comma-every-page-a-500-1h69</guid>
      <description>&lt;p&gt;A Next.js site went live behind a CDN and every single page returned 500. Not the admin area, not the login route — the home page, the about page, static-looking marketing pages. The build was green. The same build ran without a single error on the development machine.&lt;/p&gt;

&lt;p&gt;The cause was one header with one extra value in it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;x-forwarded-proto: https, https
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This post walks through why that header looks the way it does, exactly where it breaks, why the obvious fix causes a second, different failure, and the two neighbouring headers that go wrong for the same reason.&lt;/p&gt;

&lt;p&gt;All behaviour below was re-measured afterwards in isolation, against the same library build that was deployed. Nothing is from memory.&lt;/p&gt;

&lt;h2&gt;
  
  
  The shape of the deployment
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;browser ──https──▶ CDN ──https──▶ origin web server ──http──▶ Next.js (node)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two reverse proxies sit in front of the application: the CDN at the edge, and the web server on the origin machine that terminates TLS and forwards to the Node process on a local port.&lt;/p&gt;

&lt;p&gt;Both of them are doing their job correctly. Each one records what &lt;em&gt;it&lt;/em&gt; saw on the way in. The CDN saw HTTPS, so it sets &lt;code&gt;X-Forwarded-Proto: https&lt;/code&gt;. The origin server also received HTTPS — from the CDN — so it adds its own &lt;code&gt;https&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Depending on the proxy, "adds" means either a second header line or a comma-appended value. From the application's point of view it makes no difference, and that is the first thing worth measuring.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two header lines are one string
&lt;/h2&gt;

&lt;p&gt;The Fetch &lt;code&gt;Headers&lt;/code&gt; object, which is what a Next.js middleware receives, combines repeated header lines into a single comma-separated value:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;h&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Headers&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="nx"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;x-forwarded-proto&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;https&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="nx"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;x-forwarded-proto&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;https&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="nx"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;x-forwarded-proto&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="c1"&gt;// "https, https"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So there is no way to "just read the first one" through &lt;code&gt;.get()&lt;/code&gt;. Whatever sits between you and the client, you get the list.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it breaks
&lt;/h2&gt;

&lt;p&gt;The site used Auth.js v5 (&lt;code&gt;next-auth&lt;/code&gt; beta) with its &lt;code&gt;auth()&lt;/code&gt; wrapper around the middleware. The wrapper is convenient: it resolves the session for every request and hands it to your middleware as &lt;code&gt;req.auth&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;To resolve the session, it builds an internal URL for its own session endpoint. When no &lt;code&gt;AUTH_URL&lt;/code&gt; environment variable is set, that URL is assembled from request headers. The relevant part of &lt;code&gt;createActionURL&lt;/code&gt; in &lt;code&gt;@auth/core&lt;/code&gt; reads:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;detectedHost&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;x-forwarded-host&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;host&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;detectedProtocol&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;x-forwarded-proto&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="nx"&gt;protocol&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;https&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;_protocol&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;detectedProtocol&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;endsWith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;detectedProtocol&lt;/span&gt;
  &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;detectedProtocol&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nx"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;_protocol&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;//&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;detectedHost&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With one proxy, that produces &lt;code&gt;https://example.org&lt;/code&gt;. With two, it produces this string:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;https, https://example.org
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;which is not a URL. Calling it directly, against the same library build:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Input&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;single &lt;code&gt;https&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;https://example.org/api/auth/session&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;https, https&lt;/code&gt; (two header lines)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;TypeError: Invalid URL&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;https, https&lt;/code&gt; (one comma list)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;TypeError: Invalid URL&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;http, https&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;TypeError: Invalid URL&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The error is thrown inside the middleware, and the middleware runs on every matched path. So it is not "auth is broken" — it is &lt;em&gt;the site&lt;/em&gt; that is broken. Every page, including ones that never look at the session, returns 500.&lt;/p&gt;

&lt;p&gt;The development machine never showed it because it had no proxy in front at all, let alone two.&lt;/p&gt;

&lt;h2&gt;
  
  
  The obvious fix, and the second failure
&lt;/h2&gt;

&lt;p&gt;Reading the code above, the escape hatch is clear: if &lt;code&gt;AUTH_URL&lt;/code&gt; is set, the header branch is never taken. Measured:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Input&lt;/th&gt;
&lt;th&gt;&lt;code&gt;AUTH_URL&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;https, https&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;unset&lt;/td&gt;
&lt;td&gt;&lt;code&gt;TypeError: Invalid URL&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;https, https&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;https://example.org&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;https://example.org/api/auth/session&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So setting &lt;code&gt;AUTH_URL&lt;/code&gt; removes the crash. But in the wrapper, &lt;code&gt;AUTH_URL&lt;/code&gt; does a second thing. Before your middleware runs, the request itself is rebuilt with the configured origin:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;reqWithEnvURL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;url&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;AUTH_URL&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;NEXTAUTH_URL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;origin&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;envOrigin&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;href&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;origin&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;nextUrl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;NextRequest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;href&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;origin&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;envOrigin&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Inside the application, the request used to say &lt;code&gt;http://127.0.0.1:3000/...&lt;/code&gt;. Now it says &lt;code&gt;https://example.org/...&lt;/code&gt;. Any later middleware that builds a rewrite from &lt;code&gt;req.url&lt;/code&gt; — the i18n locale rewrite, in this case — now rewrites to the &lt;strong&gt;public&lt;/strong&gt; address instead of the local one. A rewrite to an external origin is a proxy hop back out through the CDN, into the same middleware, which rewrites again.&lt;/p&gt;

&lt;p&gt;The observed result on the live site was a redirect loop on &lt;code&gt;/&lt;/code&gt;. The crash was gone; the site was still down.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually fixed it
&lt;/h2&gt;

&lt;p&gt;The middleware never needed the full session. It needed one question answered — &lt;em&gt;is there a valid admin token on this request?&lt;/em&gt; — to guard a handful of routes.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;getToken&lt;/code&gt; from &lt;code&gt;next-auth/jwt&lt;/code&gt; answers exactly that, and its implementation touches only two headers: &lt;code&gt;cookie&lt;/code&gt; and &lt;code&gt;authorization&lt;/code&gt;. It never parses the protocol, never parses the host, never builds a URL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;getToken&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;next-auth/jwt&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;getToken&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;secret&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;AUTH_SECRET&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="na"&gt;secureCookie&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Replacing the &lt;code&gt;auth()&lt;/code&gt; wrapper with &lt;code&gt;getToken&lt;/code&gt; in the middleware fixed both failures at once. &lt;code&gt;AUTH_URL&lt;/code&gt; stays in the environment, because the route handlers under &lt;code&gt;/api/auth/*&lt;/code&gt; still use it — it just no longer rewrites every request passing through middleware.&lt;/p&gt;

&lt;h3&gt;
  
  
  The &lt;code&gt;secureCookie&lt;/code&gt; line is not optional
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;getToken&lt;/code&gt; has to know the cookie's &lt;em&gt;name&lt;/em&gt;, and the name depends on whether the session was issued over HTTPS. Measured from the library's cookie defaults:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;code&gt;secureCookie&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Cookie name it looks for&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;false&lt;/code&gt; (default)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;authjs.session-token&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;true&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;__Secure-authjs.session-token&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Behind TLS, the login flow issues the &lt;code&gt;__Secure-&lt;/code&gt; cookie. Without &lt;code&gt;secureCookie: true&lt;/code&gt;, &lt;code&gt;getToken&lt;/code&gt; looks for the other name, finds nothing, and returns &lt;code&gt;null&lt;/code&gt; — no error, just "not logged in", forever. The flip side: with it hard-coded to &lt;code&gt;true&lt;/code&gt;, admin login on a plain-HTTP local dev server will not work. That is the right trade-off for production, but it needs to be written down somewhere a teammate will find it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The neighbouring headers have the same problem
&lt;/h2&gt;

&lt;p&gt;Once one forwarded header is known to carry a list, the others deserve the same suspicion.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;code&gt;X-Forwarded-For&lt;/code&gt;: the last value is the CDN
&lt;/h3&gt;

&lt;p&gt;Common advice for reading the client IP is "take the last value of &lt;code&gt;X-Forwarded-For&lt;/code&gt;, because the client can forge the earlier ones." That advice assumes exactly one trusted proxy. With a CDN in front, the chain looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;x-forwarded-for: 203.0.113.7, 198.51.100.20
                 └ client     └ CDN edge node
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The last value is the CDN's edge node. Every visitor routed through that node shares it. A per-IP rate limiter or login lockout keyed on it doesn't limit one abuser — it limits everyone at once, and one failed-password burst locks the whole region out of the login form.&lt;/p&gt;

&lt;p&gt;This was caught in review before it shipped, but it is the same class of bug. The fix is to prefer the CDN's own client-IP header — Cloudflare sends &lt;code&gt;CF-Connecting-IP&lt;/code&gt;, other CDNs have their own equivalent — and to fall back to &lt;code&gt;X-Forwarded-For&lt;/code&gt; only when that header is absent. Only trust the CDN header if the origin actually refuses traffic that did not come through the CDN; otherwise anyone can set it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Canonical redirects: read the first value, or the CDN's own signal
&lt;/h3&gt;

&lt;p&gt;The same middleware issued an HTTP → HTTPS canonical redirect. It now reads the CDN's visitor-scheme header where available, and otherwise takes the &lt;strong&gt;first&lt;/strong&gt; element of &lt;code&gt;X-Forwarded-Proto&lt;/code&gt; — the hop nearest the browser — after splitting on the comma:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;proto&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;x-forwarded-proto&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="dl"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;,&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reading the whole string and comparing it to &lt;code&gt;"https"&lt;/code&gt; would have been false for every request, and redirected every request to itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checklist for "works on staging, 500 behind the CDN"
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Count your proxies.&lt;/strong&gt; A setup with one proxy (or none) and production with two are different environments, even with identical code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dump the forwarded headers from production&lt;/strong&gt; once — &lt;code&gt;x-forwarded-proto&lt;/code&gt;, &lt;code&gt;x-forwarded-host&lt;/code&gt;, &lt;code&gt;x-forwarded-for&lt;/code&gt; — and look for commas.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grep dependencies for code that turns those headers into a URL.&lt;/strong&gt; Any &lt;code&gt;new URL(\&lt;/code&gt;${proto}://${host}&lt;code&gt;)&lt;/code&gt; is a crash waiting for a second proxy.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keep middleware small.&lt;/strong&gt; If it only needs "is there a token", read the token; don't resolve a whole session on every request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Know what your auth environment variables do besides the thing you set them for.&lt;/strong&gt; &lt;code&gt;AUTH_URL&lt;/code&gt; fixed one failure and caused another.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Never key rate limits on the last &lt;code&gt;X-Forwarded-For&lt;/code&gt; value behind a CDN.&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test through the full chain before cutover&lt;/strong&gt;, not against the origin port directly.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The general shape
&lt;/h2&gt;

&lt;p&gt;Nothing here is a bug in the usual sense. The CDN appended correctly. The origin server appended correctly. The Fetch API combined repeated headers exactly as specified. The auth library built a URL from the headers it was given. Each piece is correct on its own; the failure is in the assumption, three layers deep, that a header which is &lt;em&gt;allowed&lt;/em&gt; to be a list will only ever hold one item.&lt;/p&gt;

&lt;p&gt;Headers with &lt;code&gt;Forwarded&lt;/code&gt; in the name describe a chain. Code that reads them as a single value works until the day the chain gets one link longer — and that day is usually the day you put the site behind a CDN, in production, in front of everyone.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Written at &lt;a href="https://alestaweb.com" rel="noopener noreferrer"&gt;Alesta WEB&lt;/a&gt;, from a production cutover where every page returned 500 until one comma was accounted for.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>nextjs</category>
      <category>authjs</category>
      <category>webdev</category>
      <category>security</category>
    </item>
    <item>
      <title>Six Rows Deleted. Three Decrements Applied. No Error.</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Wed, 16 Sep 2026 06:52:54 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/six-rows-deleted-three-decrements-applied-no-error-14hp</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/six-rows-deleted-three-decrements-applied-no-error-14hp</guid>
      <description>&lt;p&gt;A cached counter on a listing page was wrong. Not wrong in an interesting way — just quietly, slightly too high. The kind of number nobody audits because nobody imagines it can drift.&lt;/p&gt;

&lt;p&gt;It had been drifting for months. Across thirteen installations of the same application, the reconciliation query eventually found roughly 83,000 rows where the stored counter disagreed with the actual count. No error log entry. No failed query. No exception. Every write had returned success, every day, for months.&lt;/p&gt;

&lt;p&gt;The cause is a single line of SQL that reads correctly, executes correctly, reports success correctly, and does not do what it says.&lt;/p&gt;

&lt;h2&gt;
  
  
  The query
&lt;/h2&gt;

&lt;p&gt;The cleanup job removes flagged replies and decrements the per-topic counter. One statement for the delete, one for the counter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;topics&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
  &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;replies&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;topic_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
  &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;spam&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;DELETE&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;replies&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;spam&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read it out loud and it sounds right. For every spam reply, subtract one from its topic's counter. Then delete the replies.&lt;/p&gt;

&lt;p&gt;It is not right. The &lt;code&gt;UPDATE&lt;/code&gt; subtracts one. Total. Per topic. No matter how many replies matched.&lt;/p&gt;

&lt;h2&gt;
  
  
  The measurement
&lt;/h2&gt;

&lt;p&gt;Three topics, nine replies, six of them flagged:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;topics&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;reply_count&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;ENGINE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;InnoDB&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;replies&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="n"&gt;AUTO_INCREMENT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;topic_id&lt;/span&gt; &lt;span class="nb"&gt;INT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;spam&lt;/span&gt; &lt;span class="nb"&gt;TINYINT&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;topic_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;ENGINE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;InnoDB&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;topics&lt;/span&gt; &lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;),(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;),(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;replies&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;topic_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="n"&gt;spam&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;VALUES&lt;/span&gt;
 &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
 &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
 &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Topic 1 has three flagged replies, topic 2 has two, topic 3 has one. Counters start at 5, 3, 1. After removing the flagged replies they should read 2, 1, 0.&lt;/p&gt;

&lt;p&gt;Run the &lt;code&gt;UPDATE&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;id  reply_count
1   4
2   2
3   0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Topic 1 lost one instead of three. Topic 2 lost one instead of two. Topic 3 is correct — and that is the whole problem, because topic 3 is what a hand test looks like.&lt;/p&gt;

&lt;p&gt;Identical numbers on two different MariaDB major versions. This is not a version quirk and not a bug. It is the documented behavior of a multi-table &lt;code&gt;UPDATE&lt;/code&gt;: each target row is visited &lt;strong&gt;at most once&lt;/strong&gt; per statement, regardless of how many joined rows satisfy the condition. The engine is not iterating the join result and applying your expression once per row. It is iterating the target table and asking "does any joined row match?"&lt;/p&gt;

&lt;p&gt;The answer is yes or no. Not three.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why nobody catches it in testing
&lt;/h2&gt;

&lt;p&gt;This is the part worth internalizing, because the trap is not the semantics — it's the shape of the evidence.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A constant assignment behaves perfectly.&lt;/strong&gt; Change the &lt;code&gt;SET&lt;/code&gt; from an expression to a literal:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;topics&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
  &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;replies&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;topic_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
  &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;spam&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;id  reply_count
1   0
2   0
3   0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Flawless. Every topic with at least one flagged reply gets zeroed. &lt;code&gt;SET status = 'archived'&lt;/code&gt;, &lt;code&gt;SET flagged = 1&lt;/code&gt;, &lt;code&gt;SET reviewed_at = NOW()&lt;/code&gt; — every one of these works exactly as written, because "visit the row once" and "set it to a constant" produce the same result whether you visit once or ten times.&lt;/p&gt;

&lt;p&gt;Multi-table &lt;code&gt;UPDATE ... JOIN&lt;/code&gt; is a pattern developers use dozens of times and it is correct in almost every one of them. It becomes wrong the moment the right-hand side of the assignment references the column being assigned. Nothing in the syntax marks that boundary.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The client reports identical success.&lt;/strong&gt; I ran the broken query and the correct query through the same connection and read back the driver's counters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;wrong query -&amp;gt; affected_rows = 3 | Rows matched: 3  Changed: 3  Warnings: 0
right query -&amp;gt; affected_rows = 3 | Rows matched: 3  Changed: 3  Warnings: 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three topics matched, three topics changed, zero warnings. Both times. The broken statement and the correct statement are indistinguishable from the application's point of view. If your job logs "updated 3 topics", that log line is true and useless.&lt;/p&gt;

&lt;p&gt;Now put the delete next to it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DELETE -&amp;gt; affected_rows = 6
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Six rows deleted, three decrements applied. That discrepancy is the only signal the database ever gives you, and it appears nowhere unless you deliberately compare the two numbers. Nobody compares them, because in a healthy job they are &lt;em&gt;supposed&lt;/em&gt; to differ — you delete N replies and touch M topics, and M is naturally smaller. The number that is wrong looks exactly like the number that is right.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The drift compounds silently.&lt;/strong&gt; Run the cleanup three times against a topic with three flagged replies:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;run 1 -&amp;gt; reply_count = 4
run 2 -&amp;gt; reply_count = 3
run 3 -&amp;gt; reply_count = 2
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One per run, forever, in the wrong direction from the truth. A nightly job doesn't produce one visible error; it produces a slow leak. By the time somebody notices the number looks odd, there is no incident to correlate it with — no deploy, no outage, no spike. Just a figure that has been wrong for a while.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fixes that don't work
&lt;/h2&gt;

&lt;p&gt;The instinct once you understand the cause is to make the &lt;code&gt;UPDATE&lt;/code&gt; iterate the join instead of the target. You can't. I tried the three obvious routes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;JOIN + GROUP BY            REJECTED (syntax error)
UPDATE DISTINCT            REJECTED (syntax error)
JOIN + ORDER BY + LIMIT    ACCEPTED
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first two are refused outright. The third is accepted on MariaDB and is worth a warning: it parses, it runs, it returns success, and it does not change the "once per target row" rule at all. You get a statement that looks like it addresses the problem and doesn't. (MySQL rejects &lt;code&gt;ORDER BY&lt;/code&gt;/&lt;code&gt;LIMIT&lt;/code&gt; on a multi-table &lt;code&gt;UPDATE&lt;/code&gt; — so this particular false trail is engine-dependent, which is worse than if it were universal.)&lt;/p&gt;

&lt;p&gt;There is no flag. The aggregation has to happen before the &lt;code&gt;UPDATE&lt;/code&gt; sees the rows.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;Aggregate in a derived table, join against that, and subtract the count:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;topics&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
  &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;topic_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt;
           &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;replies&lt;/span&gt;
          &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;spam&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
          &lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;topic_id&lt;/span&gt; &lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;g&lt;/span&gt;
    &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;g&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;topic_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
  &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="k"&gt;g&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;id  reply_count
1   2
2   1
3   0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the target table is still visited once per row — that never changed — but the value it needs is already a single number. The join produces exactly one row per topic by construction.&lt;/p&gt;

&lt;p&gt;If the counter is derivable at all, the sturdier option is to stop doing arithmetic on it and recompute it outright:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;topics&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
  &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt; &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                          &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;replies&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;
                         &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;topic_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
                           &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;spam&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;id  reply_count
1   2
2   1
3   0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same result, and with a property the incremental version can never have: it is &lt;strong&gt;idempotent&lt;/strong&gt;. Run it twice, run it a hundred times, the answer is the answer. An incremental counter is a running total of every code path that ever touched it, including the ones that were wrong and the ones that crashed halfway. An absolute recount has no memory of any of that. It costs more, and on a table where the counter is read far more often than it is written, that cost buys you a number that cannot drift.&lt;/p&gt;

&lt;p&gt;Use the derived-table form when a full recount is genuinely too expensive. Use the recount when it isn't. Do not use the plain &lt;code&gt;- 1&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Finding the damage you already have
&lt;/h2&gt;

&lt;p&gt;If this pattern is anywhere in your codebase, the counters are already wrong and have been for a while. The reconciliation query is cheap and worth running before you fix anything, so you know the size of the problem:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt;             &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;stored&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;               &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;actual&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;drift&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;topics&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;
&lt;span class="k"&gt;LEFT&lt;/span&gt; &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;replies&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;topic_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt;
&lt;span class="k"&gt;HAVING&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="k"&gt;ABS&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reply_count&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;COUNT&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;id  stored  actual  drift
1   4       2       2
2   2       1       1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One detail that will bite you when you write this from memory: the expression in &lt;code&gt;HAVING&lt;/code&gt; has to be repeated in full. Referencing the &lt;code&gt;drift&lt;/code&gt; alias fails on MariaDB with &lt;code&gt;Reference 'drift' not supported (reference to group function)&lt;/code&gt;. Same for the &lt;code&gt;ORDER BY&lt;/code&gt;. It is a thirty-second detour that reads like a typo when you hit it at the end of a long day.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;LEFT JOIN&lt;/code&gt; matters too. An inner join silently drops every topic whose replies were all deleted — and those are exactly the rows most likely to be wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  The general shape
&lt;/h2&gt;

&lt;p&gt;The defect here is not SQL trivia. It is a statement whose plain-English reading and whose actual semantics diverge, where every observable signal — return code, affected row count, warning count, error log — agrees with the reading and not with the semantics.&lt;/p&gt;

&lt;p&gt;You cannot test your way out of that with assertions on success. The only test that catches it asserts on the &lt;em&gt;value&lt;/em&gt;: delete three replies from one topic, then check the counter went down by three. That test takes four lines. It is the difference between finding this in an afternoon and finding it in 83,000 rows.&lt;/p&gt;

&lt;p&gt;If a counter in your schema has no test that pins its arithmetic to a real count, assume it is wrong and go measure it. It is a five-minute query and the answer is rarely zero.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Written at &lt;a href="https://alestaweb.com" rel="noopener noreferrer"&gt;Alesta WEB&lt;/a&gt;, from a real cleanup job that had been quietly subtracting one for months.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>mysql</category>
      <category>sql</category>
      <category>database</category>
      <category>webdev</category>
    </item>
    <item>
      <title>The Socket Was Open. The Site Was Dead for Fourteen Hours.</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Mon, 07 Sep 2026 06:59:17 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/the-socket-was-open-the-site-was-dead-for-fourteen-hours-aie</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/the-socket-was-open-the-site-was-dead-for-fourteen-hours-aie</guid>
      <description>&lt;p&gt;A site had been down since 19:54 the previous evening. Nobody reported it. The monitoring did not fire. When I finally looked, the cause was one process that had outlived its parent and was holding a socket nobody could use.&lt;/p&gt;

&lt;p&gt;Every individual check passed. DNS resolved. Port 443 accepted connections. The TLS handshake completed against a valid certificate with two months left on it. The web server was running and serving other sites on the same machine. And every request to that one site returned 503.&lt;/p&gt;

&lt;p&gt;The failure lived in the gap between "the process exists" and "the process works," which is a gap most health checks do not cover.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the checks reported
&lt;/h2&gt;

&lt;p&gt;I started from the outside, because the outside is cheap to measure:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;Check&lt;/th&gt;
&lt;th&gt;Result&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;DNS&lt;/td&gt;
&lt;td&gt;resolves to the expected address&lt;/td&gt;
&lt;td&gt;fine&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TCP&lt;/td&gt;
&lt;td&gt;ports 80 and 443 accept&lt;/td&gt;
&lt;td&gt;fine&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TLS&lt;/td&gt;
&lt;td&gt;handshake, chain, expiry&lt;/td&gt;
&lt;td&gt;fine, 68 days left&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTP (port 80)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;GET /&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;302&lt;/strong&gt; to the HTTPS URL&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;HTTPS&lt;/td&gt;
&lt;td&gt;&lt;code&gt;GET /&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;first a 20-second hang, then &lt;strong&gt;503&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Web server&lt;/td&gt;
&lt;td&gt;running, other sites on the box served fine&lt;/td&gt;
&lt;td&gt;fine&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two of those rows are the reason nobody noticed.&lt;/p&gt;

&lt;p&gt;The port-80 row returns 302 because the HTTP-to-HTTPS redirect is a server-level rewrite. It never reaches PHP. A monitor watching &lt;code&gt;http://&lt;/code&gt; sees a chain that starts with a perfectly healthy 302, and if it treats a redirect as up, it reports up. Forever. The application behind it can be completely dead.&lt;/p&gt;

&lt;p&gt;The TLS row is worse, because it looks like the most meaningful check in the list. A successful handshake proves the listener is alive and the certificate is valid. It proves nothing whatsoever about whether anything can generate a response body.&lt;/p&gt;

&lt;p&gt;The first HTTPS request I made hung for twenty seconds and timed out. The second returned 503 in 143 milliseconds. That inconsistency is itself a clue: the server was trying to hand the request to a backend, waiting, giving up, and eventually not bothering to wait at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  The log line that mattered
&lt;/h2&gt;

&lt;p&gt;The site's own error log was useless. Its last entry was seventeen days old, because the log level was &lt;code&gt;WARN&lt;/code&gt; and this condition never produced a warning at the virtual-host level.&lt;/p&gt;

&lt;p&gt;The server-wide log had it, at &lt;code&gt;INFO&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[INFO] [...#site] Connection idle time too long: 301 while in state: 6 ...
[INFO] [...#site] HttpExtConnector state: 0, request body sent: 0,
                  response body size: -2, response body sent: 0, attempts: 0.
[INFO] [...#site] External processor is not available.
[NOTICE] [...#site] oops! 503 Service Unavailable
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;External processor is not available&lt;/code&gt; is the whole diagnosis in five words. The web server had a request, knew which backend should handle it, and could not get a usable connection to that backend. In LSAPI terms the external processor is the PHP process pool. In PHP-FPM terms it is the pool behind the socket you point &lt;code&gt;fastcgi_pass&lt;/code&gt; at. Same shape, same failure mode.&lt;/p&gt;

&lt;p&gt;Note &lt;code&gt;attempts: 0&lt;/code&gt; and &lt;code&gt;response body size: -2&lt;/code&gt;. It did not fail partway through a response. It never got one started.&lt;/p&gt;

&lt;h2&gt;
  
  
  PPID 1
&lt;/h2&gt;

&lt;p&gt;The pool for this site is configured for ten children and ten concurrent connections. I expected either zero processes — pool dead, should have respawned — or ten busy ones.&lt;/p&gt;

&lt;p&gt;There was exactly one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;&lt;span class="go"&gt;  PID   PPID STAT     ELAPSED     TIME %CPU   RSS CMD
793380      1 SN      13:52:58 00:07:31  0.9  7036 lsphp
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read that line carefully, because it contains the entire bug.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;PPID 1&lt;/code&gt;.&lt;/strong&gt; The parent is gone. This process was adopted by init. A pool child whose master has died is an orphan, and an orphan is supervised by nothing. Nothing will restart it, nothing will kill it, nothing will notice it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;ELAPSED 13:52:58&lt;/code&gt; against &lt;code&gt;TIME 00:07:31&lt;/code&gt;.&lt;/strong&gt; Fourteen hours of wall clock, seven and a half minutes of CPU. It was not spinning. No load-average alert was ever going to catch this.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;STAT SN&lt;/code&gt;&lt;/strong&gt; — interruptible sleep, low priority. The kernel confirms it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;grep &lt;/span&gt;State /proc/793380/status
&lt;span class="go"&gt;State:  S (sleeping)

&lt;/span&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;cat&lt;/span&gt; /proc/793380/wchan
&lt;span class="go"&gt;hrtimer_nanosleep
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;hrtimer_nanosleep&lt;/code&gt; means the process is inside a timed sleep. Some PHP path called &lt;code&gt;sleep()&lt;/code&gt; and the value it was handed put the process to bed for what may as well be forever. An API retry backoff, a lock-wait loop, a scraper delay — the specific call matters less than the shape: a wait nobody bounded.&lt;/p&gt;

&lt;p&gt;Its open file descriptors closed the case:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; /proc/793380/fd
&lt;span class="gp"&gt;0 -&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;socket:[598208598]
&lt;span class="gp"&gt;1 -&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/local/.../stderr.log &lt;span class="o"&gt;(&lt;/span&gt;deleted&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="gp"&gt;2 -&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/local/.../stderr.log &lt;span class="o"&gt;(&lt;/span&gt;deleted&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="gp"&gt;3 -&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;socket:[598208598]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two descriptors on the listening socket. And stdout and stderr pointing at a log file marked &lt;strong&gt;(deleted)&lt;/strong&gt; — something removed that file while the process held it open, so every byte it wrote for fourteen hours went to an inode with no name. If it printed a fatal error, a stack trace, a reason, that is gone. Never let cleanup delete a log a live process still holds open; rotate it and signal the writer instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why nothing restarted it
&lt;/h2&gt;

&lt;p&gt;Here is the part that generalizes past any one server.&lt;/p&gt;

&lt;p&gt;A process supervisor decides whether to start a pool by asking two questions: does the socket file exist, and is the PID in the pid file alive?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;&lt;span class="go"&gt;srwxr-xr-x 1 ... /tmp/.../siteuser.sock
-rw-r--r-- 1 ... /tmp/.../siteuser.sock.pid
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both answers were yes. The socket file was on disk with the right permissions. The pid file named a process that was alive — sleeping, low priority, adopted by init, but very much running.&lt;/p&gt;

&lt;p&gt;So the supervisor concluded the pool was up and never spawned a replacement. Meanwhile the orphan, asleep inside a timer, accepted nothing. Every incoming connection landed on a socket with no reader, waited out the connect timeout, and became a 503.&lt;/p&gt;

&lt;p&gt;The pool was neither alive nor dead. It was &lt;strong&gt;occupying the identity of a working pool&lt;/strong&gt; — the socket path and the pid — without doing any of the work. A liveness check that asks "is the PID alive" cannot see this, and neither can anything watching for a crash, because nothing crashed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fourteen hours
&lt;/h2&gt;

&lt;p&gt;I went back to the access log and counted requests per hour:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;19:00  306
20:00   19
21:00    9
22:00   13
23:00    4
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the moment of death, and it matches the process start time to within a few minutes. From 20:00 onward the only requests left are bots, retries, and the handful of people who tried, got nothing, and left.&lt;/p&gt;

&lt;p&gt;Nineteen requests an hour is not zero, and that is the trap. A "traffic dropped to zero" alert never fires. A daily-total alert compares a bad day against a normal one and shrugs at a 90% drop if the threshold is 95%. The site served errors to every visitor for fourteen hours while the graph showed a line that was low but not flat.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix, and it is boring
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;kill&lt;/span&gt; &lt;span class="nt"&gt;-TERM&lt;/span&gt; 793380
&lt;span class="nb"&gt;rm&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; /tmp/.../siteuser.sock /tmp/.../siteuser.sock.pid
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Kill the orphan, remove the stale socket and pid file so the supervisor stops believing a pool exists, then send one request. The next request rebuilt the pool by itself. Local response: 200 in 0.171 seconds. From outside: 200 in 0.29 seconds, full page, no error signatures in the HTML. Four healthy children within seconds.&lt;/p&gt;

&lt;p&gt;Under ten seconds of repair after fourteen hours of downtime. That ratio is the actual story. The failure was trivial to fix and completely invisible.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to check instead
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Monitor the URL users actually load, and follow redirects to the end.&lt;/strong&gt; A 302 from a server-level rewrite is not evidence that your application runs. If your monitor reports on the first response in the chain, it is monitoring your rewrite rules.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Assert on the body, not the status code.&lt;/strong&gt; Match a string that only appears when the application really rendered — a title, a footer, a token in the markup. Byte length works too: a 503 page is a few hundred bytes, a real page is tens of kilobytes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Alert on relative traffic collapse, not on zero.&lt;/strong&gt; A drop from 306 to 19 requests in one hour is a far better signal than any absolute floor, and it fires within the hour instead of the next morning.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Treat long-lived pool children as suspect.&lt;/strong&gt; Pool workers are meant to be recycled. Any child running for hours with almost no CPU deserves a look. &lt;code&gt;ps -eo pid,ppid,etime,time,cmd&lt;/code&gt; plus &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/wchan&lt;/code&gt; costs nothing in a cron job, and a worker with &lt;code&gt;PPID 1&lt;/code&gt; is an orphan by definition.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cap every sleep and every outbound call.&lt;/strong&gt; The root cause under all of this is an unbounded wait. Timeouts on HTTP clients, bounded retry backoffs, lock waits that eventually give up. A worker allowed to sleep forever will eventually do exactly that.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Never delete a log file a process still holds open.&lt;/strong&gt; Rotate and signal. Otherwise the one artifact that would have told you why is writing into an unlinked inode.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;DNS, TCP, TLS and the certificate can all be perfect while the site returns 503 to every visitor.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;External processor is not available&lt;/code&gt; means the web server could not get a working connection to the PHP pool. It is a backend problem, not a network or certificate problem.&lt;/li&gt;
&lt;li&gt;A pool child whose parent has died gets &lt;code&gt;PPID 1&lt;/code&gt; and is supervised by nothing.&lt;/li&gt;
&lt;li&gt;If that orphan keeps the socket and pid file, the supervisor sees a live pool and refuses to spawn a real one. Neither alive nor dead, just occupying the identity.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/proc/&amp;lt;pid&amp;gt;/wchan&lt;/code&gt; naming a sleep function, with hours of elapsed time and seconds of CPU, is the confirmation.&lt;/li&gt;
&lt;li&gt;Health checks that follow redirects and assert on body content would have caught this in the first minute.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The site was down for fourteen hours. The fix was one signal and two deleted files. Everything expensive about this outage happened in the gap between a process existing and a process working.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Written from production work at Alesta WEB.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>php</category>
      <category>linux</category>
      <category>devops</category>
      <category>webdev</category>
    </item>
    <item>
      <title>Lowercasing "İ" Adds a Character: How Turkish Text Broke a Keyword Router</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Fri, 04 Sep 2026 06:41:21 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/lowercasing-i-adds-a-character-how-turkish-text-broke-a-keyword-router-3com</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/lowercasing-i-adds-a-character-how-turkish-text-broke-a-keyword-router-3com</guid>
      <description>&lt;p&gt;A classifier was sending questions to the wrong bucket. Not occasionally. I measured it against twenty real inputs and seven of them landed somewhere they did not belong. Thirty-five percent.&lt;/p&gt;

&lt;p&gt;The routing code was thirty lines long and did the obvious thing: lowercase the incoming text, then check whether any keyword for a category appears in it. First match wins.&lt;/p&gt;

&lt;p&gt;Every one of those thirty lines was wrong in a way that only shows up in Turkish, and two of the three problems will bite you in any language whose case mappings are not one-to-one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The symptom
&lt;/h2&gt;

&lt;p&gt;Users type free-form questions. The router picks a topic so that the right prompt and the right reference data get loaded. When it picks wrong, the answer is confidently about the wrong subject, which is worse than no answer.&lt;/p&gt;

&lt;p&gt;The reported complaint was "the answers feel shallow." The instinct is to blame the model. The model was fine. It was being handed the wrong context.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 1: lowercasing a dotted capital I produces two code points
&lt;/h2&gt;

&lt;p&gt;Here is the whole thing, measured rather than remembered:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="nv"&gt;$s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"İş"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;                              &lt;span class="c1"&gt;// "work", capital İ&lt;/span&gt;
&lt;span class="nv"&gt;$l&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;mb_strtolower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"UTF-8"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;// original      : U+0130 U+015F&lt;/span&gt;
&lt;span class="c1"&gt;// mb_strtolower : U+0069 U+0307 U+015F&lt;/span&gt;
&lt;span class="c1"&gt;// mb_strlen     : 2 -&amp;gt; 3&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE&lt;/code&gt; has no single-code-point lowercase form in the default Unicode case mapping. It maps to &lt;code&gt;U+0069 LATIN SMALL LETTER I&lt;/code&gt; followed by &lt;code&gt;U+0307 COMBINING DOT ABOVE&lt;/code&gt;. The string got longer. It still renders as "i̇ş" and looks entirely normal in your terminal, your editor, and your database client.&lt;/p&gt;

&lt;p&gt;So this happens:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="nb"&gt;mb_strpos&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;mb_strtolower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"İşten çıkarılır mıyım"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"UTF-8"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="s2"&gt;"iş"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;   &lt;span class="c1"&gt;// false&lt;/span&gt;
&lt;span class="nb"&gt;mb_strtolower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"İSTANBUL"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"UTF-8"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="s2"&gt;"istanbul"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;                  &lt;span class="c1"&gt;// false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Your keyword is two code points. The haystack has three. No match, no error, no warning. The comparison returns false and the request goes to whatever category happens to be checked next.&lt;/p&gt;

&lt;p&gt;The reason this is not an edge case: people capitalize the first word of a sentence. In Turkish a large number of everyday words begin with İ — the words for work, request, relationship, reputation, tender, transaction, need, cancellation. Any of them starting a sentence produces the three-code-point form. &lt;strong&gt;The broken form is the common form&lt;/strong&gt;, and the version your test uses — all lowercase, typed by you — is the rare one.&lt;/p&gt;

&lt;p&gt;That is why this survives a green test suite. You wrote &lt;code&gt;"işten çıkarılır mıyım"&lt;/code&gt; in the fixture, in lowercase, by hand. It matches. The user wrote &lt;code&gt;"İşten çıkarılır mıyım"&lt;/code&gt;. It does not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Unicode normalization will not save you
&lt;/h2&gt;

&lt;p&gt;The reflex is to reach for &lt;code&gt;Normalizer::normalize()&lt;/code&gt; and move on. I checked:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;lower    : U+0069 U+0307 U+015F
NFC      : U+0069 U+0307 U+015F
NFKC     : U+0069 U+0307 U+015F
casefold : U+0069 U+0307 U+015F
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;NFC composes a base character with a combining mark when a precomposed code point exists. There is no precomposed "latin small letter i with dot above" — a lowercase i already has a dot, so Unicode never needed one. The sequence stays two code points under every composing form, and &lt;code&gt;casefold()&lt;/code&gt; does not touch it either.&lt;/p&gt;

&lt;p&gt;NFD is more interesting, because it goes the other way and decomposes everything:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NFD("i̇ş") : U+0069 U+0307 U+0073 U+0327
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the cedilla under ş is its own code point too. That gives you one rule that handles both problems at once: decompose, then drop every nonspacing mark.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;unicodedata&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;normalize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;NFD&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
    &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;''&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;category&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;Mn&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ı&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;i&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;      &lt;span class="c1"&gt;# dotless ı has no combining mark to strip
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Measured output:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;'İş'         -&amp;gt; 'is'
'İSTANBUL'   -&amp;gt; 'istanbul'
'Değişiklik' -&amp;gt; 'degisiklik'
'İlişki'     -&amp;gt; 'iliski'
'Ölüm'       -&amp;gt; 'olum'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In PHP, where the &lt;code&gt;intl&lt;/code&gt; extension is frequently not loaded, the same thing without a normalizer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;tr_norm&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nv"&gt;$s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;mb_strtolower&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'UTF-8'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nv"&gt;$s&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;str_replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\u{0307}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;        &lt;span class="c1"&gt;// the leftover dot from İ&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nb"&gt;strtr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;
        &lt;span class="s1"&gt;'ı'&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'i'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s1"&gt;'ş'&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'s'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s1"&gt;'ğ'&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'g'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s1"&gt;'ü'&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'u'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s1"&gt;'ö'&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'o'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s1"&gt;'ç'&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'c'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="s1"&gt;'â'&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'a'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s1"&gt;'î'&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'i'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="s1"&gt;'û'&lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'u'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The single most important line is the &lt;code&gt;U+0307&lt;/code&gt; removal. Delete it and the bug is back.&lt;/p&gt;

&lt;p&gt;Run your keyword list through the same function, too. Both sides of a comparison must see the same normalization, always. A keyword table normalized at author time and input normalized at request time will drift the moment somebody edits the table by pasting from a document.&lt;/p&gt;

&lt;h2&gt;
  
  
  The other direction, for readers who do not write Turkish
&lt;/h2&gt;

&lt;p&gt;Turkish has two distinct letters where English has one: dotted &lt;code&gt;İ/i&lt;/code&gt; and dotless &lt;code&gt;I/ı&lt;/code&gt;. They are different letters, not styling. So the correct Turkish lowercase of &lt;code&gt;I&lt;/code&gt; is &lt;code&gt;ı&lt;/code&gt;, not &lt;code&gt;i&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;This is the well-known "Turkish I problem," and it usually appears as the mirror image of what I described. In Java, .NET, or any locale-sensitive &lt;code&gt;toLowerCase()&lt;/code&gt; running under a &lt;code&gt;tr-TR&lt;/code&gt; locale:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="s"&gt;"INFO"&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;toLowerCase&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;          &lt;span class="c1"&gt;// "ınfo" under tr-TR, "info" everywhere else&lt;/span&gt;
&lt;span class="s"&gt;"FILE"&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;toLowerCase&lt;/span&gt;&lt;span class="o"&gt;()&lt;/span&gt;          &lt;span class="c1"&gt;// "fıle"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Code that lowercases a protocol keyword, a header name, a file extension, or an SQL identifier and then compares it against an ASCII literal breaks on a machine whose locale happens to be Turkish. The fix there is the opposite of the one above: pass an explicit invariant locale, &lt;code&gt;toLowerCase(Locale.ROOT)&lt;/code&gt;, and never let ambient locale decide the semantics of a comparison.&lt;/p&gt;

&lt;p&gt;So there are two rules, pointing in opposite directions depending on what you are doing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Comparing &lt;strong&gt;protocol tokens and identifiers&lt;/strong&gt;: force a locale-invariant mapping. Never fold, never guess.&lt;/li&gt;
&lt;li&gt;Comparing &lt;strong&gt;human text for search or classification&lt;/strong&gt;: normalize aggressively, on both sides, with a function you control.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Confusing the two is how you end up with a system that is broken in one country and subtly wrong everywhere else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 2: unbounded substring matching
&lt;/h2&gt;

&lt;p&gt;With normalization fixed, the router still misrouted. The second bug had nothing to do with Unicode: the check was &lt;code&gt;strpos($text, $keyword) !== false&lt;/code&gt;, with no word boundary.&lt;/p&gt;

&lt;p&gt;Short keywords swallow longer words that merely contain them. Real collisions from the keyword table:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Keyword&lt;/th&gt;
&lt;th&gt;Means&lt;/th&gt;
&lt;th&gt;Also matches&lt;/th&gt;
&lt;th&gt;Which means&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;olum&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;death&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;olum&lt;/strong&gt;lu, &lt;strong&gt;olum&lt;/strong&gt;suz&lt;/td&gt;
&lt;td&gt;positive, negative&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;mal&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;goods, property&lt;/td&gt;
&lt;td&gt;nor*&lt;em&gt;mal&lt;/em&gt;&lt;em&gt;, **mal&lt;/em&gt;*zeme&lt;/td&gt;
&lt;td&gt;normal, material&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;is&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;work&lt;/td&gt;
&lt;td&gt;k*&lt;em&gt;is&lt;/em&gt;&lt;em&gt;i, deg&lt;/em&gt;&lt;em&gt;is&lt;/em&gt;&lt;em&gt;iklik, g&lt;/em&gt;&lt;em&gt;is&lt;/em&gt;*e&lt;/td&gt;
&lt;td&gt;person, change, counter&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ask&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;love&lt;/td&gt;
&lt;td&gt;m*&lt;em&gt;ask&lt;/em&gt;*e&lt;/td&gt;
&lt;td&gt;mask&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;din&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;religion&lt;/td&gt;
&lt;td&gt;ay*&lt;em&gt;din&lt;/em&gt;&lt;em&gt;, **din&lt;/em&gt;*len&lt;/td&gt;
&lt;td&gt;bright, rest&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;olum&lt;/code&gt; was the expensive one. "Olumlu" — "positive" — is among the most common words in any yes/no question. Every such question was routed to the death-and-inheritance category, because that category happened to sit near the top of the map.&lt;/p&gt;

&lt;p&gt;Word boundaries in an agglutinative language are not just &lt;code&gt;\b&lt;/code&gt; around a fixed string. Turkish stacks suffixes, and stems mutate: "lost" appears as &lt;em&gt;kayıp&lt;/em&gt;, &lt;em&gt;kaybettiğim&lt;/em&gt;, &lt;em&gt;kaybolan&lt;/em&gt; — the stem itself shifts from &lt;code&gt;kayıp&lt;/code&gt; to &lt;code&gt;kayb&lt;/code&gt;. So match a stem plus a bounded run of following letters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="nv"&gt;$pattern&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'/(?&amp;lt;!\p{L})'&lt;/span&gt; &lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="nb"&gt;preg_quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$stem&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'/'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="mf"&gt;.&lt;/span&gt; &lt;span class="s1"&gt;'\p{L}{0,6}(?!\p{L})/u'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;(?&amp;lt;!\p{L})&lt;/code&gt; and &lt;code&gt;(?!\p{L})&lt;/code&gt; are Unicode-aware boundary assertions — plain &lt;code&gt;\b&lt;/code&gt; reasons about &lt;code&gt;\w&lt;/code&gt;, whose meaning depends on flags and locale. The &lt;code&gt;{0,6}&lt;/code&gt; allows suffixes without letting the stem float in the middle of an unrelated word. And keep a stem list per concept rather than one string: &lt;code&gt;kayıp|kayb|kaybol|kaybet&lt;/code&gt;, because sound changes will not occur to you at the keyboard, only in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  Trap 3: map order is a priority ranking you did not know you wrote
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$map&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$topic&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$words&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$words&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$w&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;matches&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;$w&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nv"&gt;$topic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;// first hit wins&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Whatever order the array literal happens to be in &lt;em&gt;is&lt;/em&gt; your precedence rule. "Will my father's health improve" contains both a family keyword and a health keyword. Family was declared fourth, health sixth, so it went to the family bucket every time.&lt;/p&gt;

&lt;p&gt;Two changes fix the class of problem rather than the instance:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Order specific before general, deliberately, with a comment saying so.&lt;/li&gt;
&lt;li&gt;Stop returning on first hit. Collect every match, score them, take the best. A reasonable score is the number of distinct stems matched, weighted by stem specificity — a rare six-letter stem is worth more than a three-letter one that appears everywhere.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Scoring also gives you something first-match never can: a confidence number. When the top two topics score within noise of each other, you can fall back to a neutral path instead of committing to a coin flip.&lt;/p&gt;

&lt;h2&gt;
  
  
  Normalize for matching, never for identity
&lt;/h2&gt;

&lt;p&gt;One caution, because the recipe above is lossy on purpose:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;'ILIK'  -&amp;gt; 'ilik'
'ılık'  -&amp;gt; 'ilik'
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two different words collapse to the same key. That is exactly what you want for search and classification, and exactly what you must never do for identity.&lt;/p&gt;

&lt;p&gt;If you normalize usernames or email local parts this way before a uniqueness check or a login comparison, you have created an account-collision path: two visually and semantically distinct identifiers now compare equal. The same applies to case-insensitive uniqueness in general — the database collation, the application-level check and the login comparison have to agree on one rule, and that rule should be locale-invariant, not a fuzzy fold.&lt;/p&gt;

&lt;p&gt;Fold for relevance. Compare exactly for identity. Keep the two functions in separate files so nobody reuses one for the other by accident.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to verify, because you cannot eyeball this
&lt;/h2&gt;

&lt;p&gt;Every symptom in this article is invisible on screen. The three-code-point string looks identical to the two-code-point one. You cannot review your way to correctness here; you have to measure.&lt;/p&gt;

&lt;p&gt;What I did, and would do again:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Write a table of 20–30 real inputs with the expected category&lt;/strong&gt;, taken from actual traffic rather than invented. Run it, print the wrong-rate as a number. "Seven of twenty" is a fact you can act on; "seems better now" is not.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sweep for collisions mechanically.&lt;/strong&gt; For every keyword shorter than about five characters, generate common words containing it and feed them through the classifier. That is how &lt;code&gt;olum&lt;/code&gt; inside &lt;code&gt;olumlu&lt;/code&gt; surfaced — nobody was going to catch it by reading.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Print code points, not strings&lt;/strong&gt;, whenever a comparison fails for no visible reason. &lt;code&gt;printf("U+%04X ")&lt;/code&gt; over the characters takes ten seconds and ends the argument.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Re-run the same table after the fix&lt;/strong&gt; and report the new number. A fix that is not re-measured is a hypothesis.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;mb_strtolower('İ')&lt;/code&gt; returns &lt;code&gt;i&lt;/code&gt; + &lt;code&gt;U+0307&lt;/code&gt;. The string gets longer, looks the same, and stops matching.&lt;/li&gt;
&lt;li&gt;NFC and casefold do not collapse it. NFD plus stripping nonspacing marks does.&lt;/li&gt;
&lt;li&gt;Normalize the input and the keyword list with the same function.&lt;/li&gt;
&lt;li&gt;Match stems with Unicode-aware boundaries, not bare substrings.&lt;/li&gt;
&lt;li&gt;Map order is silent precedence. Score instead of returning on first hit.&lt;/li&gt;
&lt;li&gt;Fold for search; compare exactly for identity.&lt;/li&gt;
&lt;li&gt;Measure the wrong-rate before and after, on real inputs.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this is exotic Unicode trivia. It is one line of case mapping and one missing word boundary, and together they were quietly discarding a third of the routing decisions in a system that looked, from the outside, like a model quality problem.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Written from production work at Alesta WEB.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>php</category>
      <category>unicode</category>
      <category>i18n</category>
      <category>webdev</category>
    </item>
    <item>
      <title>The Word "null" Is Not a Null: How JSON Nulls Turn Into Text on Android</title>
      <dc:creator>Mahmut Gündüzalp</dc:creator>
      <pubDate>Sun, 30 Aug 2026 10:39:10 +0000</pubDate>
      <link>https://dev.to/mahmut_gndzalp_c736ac4b/the-word-null-is-not-a-null-how-json-nulls-turn-into-text-on-android-10kg</link>
      <guid>https://dev.to/mahmut_gndzalp_c736ac4b/the-word-null-is-not-a-null-how-json-nulls-turn-into-text-on-android-10kg</guid>
      <description>&lt;p&gt;A user opened our mobile app and saw this on their profile screen:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Membership: null
Points: null
Last payment: null
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same account on the website showed empty fields and a zero. No crash, no error dialog, no red line in the log. The app had received the data, parsed it successfully, and printed the four characters &lt;code&gt;n&lt;/code&gt;, &lt;code&gt;u&lt;/code&gt;, &lt;code&gt;l&lt;/code&gt;, &lt;code&gt;l&lt;/code&gt; into a &lt;code&gt;TextView&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The parsing code looked like it had already handled this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight kotlin"&gt;&lt;code&gt;&lt;span class="kd"&gt;val&lt;/span&gt; &lt;span class="py"&gt;plan&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;optString&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"membership"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the whole bug. On Android, when &lt;code&gt;membership&lt;/code&gt; is JSON &lt;code&gt;null&lt;/code&gt;, &lt;code&gt;optString&lt;/code&gt; does not return &lt;code&gt;""&lt;/code&gt;. It returns the string &lt;code&gt;"null"&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why it does that
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;org.json&lt;/code&gt; on Android represents JSON null with a sentinel object, &lt;code&gt;JSONObject.NULL&lt;/code&gt;. It is not Java &lt;code&gt;null&lt;/code&gt; — it is a real object that exists specifically so the library can tell "the key is absent" apart from "the key is present and its value is null."&lt;/p&gt;

&lt;p&gt;Android's implementation of &lt;code&gt;optString&lt;/code&gt; is, in essence:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="nf"&gt;optString&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Object&lt;/span&gt; &lt;span class="n"&gt;object&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="no"&gt;JSON&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;toString&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;object&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;   &lt;span class="c1"&gt;// ~ object.toString()&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;opt("membership")&lt;/code&gt; returns the sentinel. The sentinel is not Java &lt;code&gt;null&lt;/code&gt;, so the fallback branch is never reached. &lt;code&gt;toString()&lt;/code&gt; is called on it, and &lt;code&gt;JSONObject.NULL.toString()&lt;/code&gt; returns — by design, because it has to serialize back to valid JSON — the text &lt;code&gt;"null"&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The fallback works exactly as documented for a &lt;em&gt;missing&lt;/em&gt; key. It does nothing for a &lt;em&gt;null&lt;/em&gt; key. Those are two different states and the API gives them one parameter.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;getString&lt;/code&gt; has the same outcome. It does not throw on a null value; it hands you &lt;code&gt;"null"&lt;/code&gt; as well.&lt;/p&gt;

&lt;p&gt;Now compare the neighbours:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight kotlin"&gt;&lt;code&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;optInt&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"points"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;        &lt;span class="c1"&gt;// JSON null -&amp;gt; 0     ✅&lt;/span&gt;
&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;optBoolean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"active"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;false&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="c1"&gt;// JSON null -&amp;gt; false ✅&lt;/span&gt;
&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;optString&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"plan"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;       &lt;span class="c1"&gt;// JSON null -&amp;gt; "null" ❌&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;optInt&lt;/code&gt; and &lt;code&gt;optBoolean&lt;/code&gt; route through converters that return Java &lt;code&gt;null&lt;/code&gt; for the sentinel, so the fallback fires. Only the string path stringifies the sentinel. This inconsistency is why the bug survives review: three lines that look identical, and one of them behaves differently.&lt;/p&gt;

&lt;h3&gt;
  
  
  It also depends on which &lt;code&gt;org.json&lt;/code&gt; you are compiled against
&lt;/h3&gt;

&lt;p&gt;This is worth checking rather than assuming. The reference implementation from json.org — the &lt;code&gt;org.json:json&lt;/code&gt; jar you would pull into a plain JVM or server-side project — has a version of the same method that reads roughly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight java"&gt;&lt;code&gt;&lt;span class="kd"&gt;public&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="nf"&gt;optString&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="n"&gt;defaultValue&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
    &lt;span class="nc"&gt;Object&lt;/span&gt; &lt;span class="n"&gt;object&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;opt&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="no"&gt;NULL&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;equals&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;object&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;defaultValue&lt;/span&gt; &lt;span class="o"&gt;:&lt;/span&gt; &lt;span class="n"&gt;object&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;toString&lt;/span&gt;&lt;span class="o"&gt;();&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It checks the sentinel explicitly and returns your default. Same package, same class, same method signature, different answer.&lt;/p&gt;

&lt;p&gt;Android ships its own Harmony-derived implementation in the framework, and it wins on the classpath. So a unit test that runs on the JVM with the json.org jar can pass while the device fails. If your &lt;code&gt;test&lt;/code&gt; source set is green and your &lt;code&gt;androidTest&lt;/code&gt; source set is red on the same assertion, this is a candidate.&lt;/p&gt;

&lt;p&gt;Confirm it in ten seconds with an instrumented test rather than trusting either the docs or this article:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight kotlin"&gt;&lt;code&gt;&lt;span class="nd"&gt;@Test&lt;/span&gt; &lt;span class="k"&gt;fun&lt;/span&gt; &lt;span class="nf"&gt;optStringOnJsonNull&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;val&lt;/span&gt; &lt;span class="py"&gt;o&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;JSONObject&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"""{"a": null}"""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;assertEquals&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;optString&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;   &lt;span class="c1"&gt;// fails on device&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Where the null came from in the first place
&lt;/h2&gt;

&lt;p&gt;Fixing the client is only half of it, because the response should not have contained nulls for those fields at all.&lt;/p&gt;

&lt;p&gt;The API was PHP, and the payload was built from a query that looked something like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;
    &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;plan_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;total_paid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="k"&gt;MAX&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;paid_at&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;last_payment&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;members&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;
&lt;span class="k"&gt;LEFT&lt;/span&gt; &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;payments&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;member_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two separate null factories in one query:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;LEFT JOIN&lt;/code&gt; with no matching rows.&lt;/strong&gt; Every column from &lt;code&gt;payments&lt;/code&gt; comes back &lt;code&gt;NULL&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Aggregates over an empty set.&lt;/strong&gt; &lt;code&gt;SUM()&lt;/code&gt; and &lt;code&gt;MAX()&lt;/code&gt; return &lt;code&gt;NULL&lt;/code&gt;, not &lt;code&gt;0&lt;/code&gt; and not &lt;code&gt;''&lt;/code&gt;, when there are no rows to aggregate. &lt;code&gt;COUNT()&lt;/code&gt; is the exception — it returns &lt;code&gt;0&lt;/code&gt; — which is exactly why people assume the others do too.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The website never showed this because the view layer cast on the way out:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="k"&gt;echo&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)(&lt;/span&gt;&lt;span class="nv"&gt;$row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'total_paid'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;echo&lt;/span&gt; &lt;span class="nb"&gt;htmlspecialchars&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'plan_name'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;?? 0&lt;/code&gt; there is doing real work — &lt;code&gt;??&lt;/code&gt; catches SQL &lt;code&gt;NULL&lt;/code&gt; once PDO has turned it into PHP &lt;code&gt;null&lt;/code&gt;. Web output was clean. The JSON endpoint serialized the same row directly, &lt;code&gt;json_encode&lt;/code&gt; mapped PHP &lt;code&gt;null&lt;/code&gt; to JSON &lt;code&gt;null&lt;/code&gt;, and the app printed the word.&lt;/p&gt;

&lt;p&gt;This is the general shape: &lt;strong&gt;a null that is invisible in one rendering path becomes visible in another.&lt;/strong&gt; Two consumers of the same data, one of which happens to have a defensive cast in its template.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why the test accounts never caught it
&lt;/h3&gt;

&lt;p&gt;The seeded test users all had payment history. Every demo account had rows. The only way to produce this response was to log in as a member with &lt;em&gt;nothing&lt;/em&gt; — a brand new registration, or an account whose records had been archived.&lt;/p&gt;

&lt;p&gt;A user with zero rows is a boundary condition, and it is the one that seed scripts systematically do not create, because seed scripts exist to make screens look populated. If you write fixtures, make one account deliberately empty and put it in the smoke test.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fix both layers, not one
&lt;/h2&gt;

&lt;p&gt;The tempting move is to fix the server, since that is where the null originates, and declare the client fine. That is wrong for mobile specifically, and here is why.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Old app versions live forever.&lt;/strong&gt; You control your server. You do not control which build is on a user's phone. A client that renders &lt;code&gt;"null"&lt;/code&gt; when handed a JSON null is a permanently loaded gun pointed at every future endpoint, including ones written by someone who has never heard of this bug.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Caches outlive deploys.&lt;/strong&gt; An HTTP cache, a CDN, or a local response cache can hand the app a pre-fix payload well after the server is patched.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;New fields arrive constantly.&lt;/strong&gt; Server-side normalization is a policy you have to remember to apply. Client-side hardening is a helper you cannot forget to use, because it is the only way to read a string.&lt;/p&gt;

&lt;p&gt;So: both.&lt;/p&gt;

&lt;h3&gt;
  
  
  Server side — normalize once, at the boundary
&lt;/h3&gt;

&lt;p&gt;Recursively convert nulls at the point where the response is serialized, not in each handler:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight php"&gt;&lt;code&gt;&lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;json_clean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;mixed&lt;/span&gt; &lt;span class="nv"&gt;$value&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="kt"&gt;mixed&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;is_array&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$value&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$value&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nv"&gt;$k&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nv"&gt;$v&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="nv"&gt;$value&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;$k&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;json_clean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$v&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nv"&gt;$value&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nv"&gt;$value&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;// single exit point for every API response&lt;/span&gt;
&lt;span class="k"&gt;function&lt;/span&gt; &lt;span class="n"&gt;api_respond&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;array&lt;/span&gt; &lt;span class="nv"&gt;$payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nv"&gt;$status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nb"&gt;http_response_code&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$status&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="nb"&gt;header&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'Content-Type: application/json; charset=utf-8'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;echo&lt;/span&gt; &lt;span class="nb"&gt;json_encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;json_clean&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;$payload&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="no"&gt;JSON_UNESCAPED_UNICODE&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One caveat worth stating plainly: this flattens the difference between "no value" and "empty value". For a display API that is what you want. For an API where a client needs to distinguish "field not provided" from "field explicitly cleared" — a PATCH-style contract, for instance — do not do this globally. Fix those fields at the query instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="n"&gt;COALESCE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SUM&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;total_paid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="n"&gt;COALESCE&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;plan_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;    &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;plan_name&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;COALESCE&lt;/code&gt; is the honest fix; the recursive normalizer is the safety net under it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Client side — never call &lt;code&gt;optString&lt;/code&gt; directly again
&lt;/h3&gt;

&lt;p&gt;Wrap it once, ban the raw call in review:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight kotlin"&gt;&lt;code&gt;&lt;span class="kd"&gt;object&lt;/span&gt; &lt;span class="nc"&gt;Json&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;

    &lt;span class="k"&gt;fun&lt;/span&gt; &lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;JSONObject&lt;/span&gt;&lt;span class="p"&gt;?,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt; &lt;span class="p"&gt;==&lt;/span&gt; &lt;span class="k"&gt;null&lt;/span&gt; &lt;span class="p"&gt;||&lt;/span&gt; &lt;span class="p"&gt;!&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;has&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;||&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isNull&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;
        &lt;span class="kd"&gt;val&lt;/span&gt; &lt;span class="py"&gt;value&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;optString&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="c1"&gt;// a server can also send the *string* "null" — from string concatenation,&lt;/span&gt;
        &lt;span class="c1"&gt;// a logging wrapper, or an older client that echoed this bug back&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isEmpty&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;||&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="p"&gt;==&lt;/span&gt; &lt;span class="s"&gt;"null"&lt;/span&gt; &lt;span class="p"&gt;||&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt; &lt;span class="p"&gt;==&lt;/span&gt; &lt;span class="s"&gt;"undefined"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;fun&lt;/span&gt; &lt;span class="nf"&gt;number&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;JSONObject&lt;/span&gt;&lt;span class="p"&gt;?,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Long&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nc"&gt;Long&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt; &lt;span class="p"&gt;==&lt;/span&gt; &lt;span class="k"&gt;null&lt;/span&gt; &lt;span class="p"&gt;||&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;isNull&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;optLong&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;fallback&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;fun&lt;/span&gt; &lt;span class="nf"&gt;textOrNull&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;JSONObject&lt;/span&gt;&lt;span class="p"&gt;?,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt;
        &lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;obj&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;ifEmpty&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;null&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;isNull(key)&lt;/code&gt; is the load-bearing line. It returns true for both Java &lt;code&gt;null&lt;/code&gt; and the &lt;code&gt;JSONObject.NULL&lt;/code&gt; sentinel, which is the check &lt;code&gt;optString&lt;/code&gt; fails to make on your behalf.&lt;/p&gt;

&lt;p&gt;Filtering the literal &lt;code&gt;"null"&lt;/code&gt; string looks paranoid until you have seen a server do &lt;code&gt;"Welcome, " + name&lt;/code&gt; in a template and store the result. Once a &lt;code&gt;"null"&lt;/code&gt; has been written to a database by an upstream system, no amount of correctness downstream removes it — you have to strip it at read time.&lt;/p&gt;

&lt;p&gt;Then make the raw call unusable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight kotlin"&gt;&lt;code&gt;&lt;span class="c1"&gt;// build.gradle.kts — fail the build, not the user&lt;/span&gt;
&lt;span class="c1"&gt;// (or use Detekt's ForbiddenMethodCall rule)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A lint rule beats a code review comment, because the review comment only catches the calls written while the reviewer remembers the bug.&lt;/p&gt;

&lt;h2&gt;
  
  
  What about Gson, Moshi, kotlinx.serialization?
&lt;/h2&gt;

&lt;p&gt;Mostly they handle this correctly, with one trap of their own.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Moshi&lt;/strong&gt; and &lt;strong&gt;kotlinx.serialization&lt;/strong&gt; are aware of Kotlin nullability. Declare &lt;code&gt;val plan: String&lt;/code&gt; as non-null and feed it JSON null, and they throw at parse time — a loud, immediate, correctly-located failure. That is the behaviour you want.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Gson&lt;/strong&gt; does not. It constructs objects through unsafe reflection and bypasses Kotlin's null checks entirely, so a JSON null lands in a field the type system swears is non-null. Nothing fails at parse time. The NPE happens later, somewhere unrelated, and the stack trace points at the innocent code that touched the field rather than the parser that lied about it. That is arguably worse than &lt;code&gt;"null"&lt;/code&gt; on screen, because at least &lt;code&gt;"null"&lt;/code&gt; on screen tells you where it came from.&lt;/p&gt;

&lt;p&gt;If you are on Gson, declare the fields nullable (&lt;code&gt;String?&lt;/code&gt;) and handle the nulls explicitly. Fighting for non-null fields you cannot enforce buys nothing.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;org.json&lt;/code&gt; remains everywhere despite all three of these, for a simple reason: it is in the Android framework, weighs nothing, and needs no annotation processor. Small apps and old modules keep using it. It works fine — as long as you know that its string path does not have a null story.&lt;/p&gt;

&lt;h2&gt;
  
  
  The five-minute audit
&lt;/h2&gt;

&lt;p&gt;Worth running on any app that talks to an API you also control:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;grep -rn "optString(" app/src&lt;/code&gt; — every hit is a candidate. Replace with a helper that calls &lt;code&gt;isNull&lt;/code&gt; first.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;grep -rn "getString(" app/src&lt;/code&gt; on &lt;code&gt;JSONObject&lt;/code&gt; receivers — same problem, no fallback parameter to give false comfort.&lt;/li&gt;
&lt;li&gt;In the API, find aggregates without &lt;code&gt;COALESCE&lt;/code&gt;: &lt;code&gt;SUM(&lt;/code&gt;, &lt;code&gt;MAX(&lt;/code&gt;, &lt;code&gt;MIN(&lt;/code&gt;, &lt;code&gt;AVG(&lt;/code&gt; — and every &lt;code&gt;LEFT JOIN&lt;/code&gt; selecting columns from the right-hand table.&lt;/li&gt;
&lt;li&gt;Create one test account with zero related rows and open every screen with it. This finds more null bugs in ten minutes than reading code for a day.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;grep&lt;/code&gt; your database for the literal string &lt;code&gt;'null'&lt;/code&gt; in text columns. If you find any, something upstream has already been concatenating nulls into text, and that data needs cleaning independently of any code fix.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The underlying lesson is not about one library method. It is that &lt;code&gt;null&lt;/code&gt; crosses system boundaries badly. SQL &lt;code&gt;NULL&lt;/code&gt;, PHP &lt;code&gt;null&lt;/code&gt;, JSON &lt;code&gt;null&lt;/code&gt;, Java &lt;code&gt;null&lt;/code&gt;, Kotlin &lt;code&gt;null?&lt;/code&gt;, and a sentinel object that stringifies to &lt;code&gt;"null"&lt;/code&gt; are six different things wearing the same name, and every hop between two of them is a place where one quietly becomes another. Put the conversion in one function per boundary, and you only have to be right once.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;I build and run news, e-commerce and membership platforms at &lt;a href="https://alestaweb.com" rel="noopener noreferrer"&gt;alestaweb.com&lt;/a&gt;. This one took an afternoon and a screenshot from a user who assumed "null" was our membership tier.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>android</category>
      <category>kotlin</category>
      <category>json</category>
      <category>api</category>
    </item>
  </channel>
</rss>
