<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Vasyl</title>
    <description>The latest articles on DEV Community by Vasyl (@mrviduus).</description>
    <link>https://dev.to/mrviduus</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F333461%2Fea3cc6b2-e942-4848-8606-30c345279779.jpg</url>
      <title>DEV Community: Vasyl</title>
      <link>https://dev.to/mrviduus</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/mrviduus"/>
    <language>en</language>
    <item>
      <title>C# Without a Project File: File-Based Apps in .NET 11 (a 5-Minute Refresh)</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 17 Nov 2026 11:01:57 +0000</pubDate>
      <link>https://dev.to/mrviduus/c-without-a-project-file-file-based-apps-in-net-11-a-5-minute-refresh-1pif</link>
      <guid>https://dev.to/mrviduus/c-without-a-project-file-file-based-apps-in-net-11-a-5-minute-refresh-1pif</guid>
      <description>&lt;p&gt;Since .NET 10 you can run a C# file without a project: &lt;code&gt;dotnet run app.cs&lt;/code&gt;. No &lt;code&gt;.sln&lt;/code&gt;, no &lt;code&gt;.csproj&lt;/code&gt;. But it was one file only. In .NET 11 a file-based app can have many files, and that makes it useful for real small tools.&lt;/p&gt;

&lt;p&gt;I tried it on the classic Mars Rover kata. A rover starts at &lt;code&gt;0 0 N&lt;/code&gt;. &lt;code&gt;L&lt;/code&gt; and &lt;code&gt;R&lt;/code&gt; turn 90°, &lt;code&gt;M&lt;/code&gt; moves one cell forward. The program prints the final position.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4vsx9jp1x08vmj94raa4.jpg" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4vsx9jp1x08vmj94raa4.jpg" alt="NASA's Perseverance rover on Mars, taking a selfie with the Ingenuity helicopter in the background." width="800" height="588"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The real Mars rover: Perseverance, with the Ingenuity helicopter behind it. Ours only knows L, R and M. Credit: NASA/JPL-Caltech/MSSS&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Here is what we'll go through:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;One file&lt;/strong&gt;: run C# without a project&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;#:include&lt;/strong&gt;: split the app into files&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nested includes&lt;/strong&gt;: a file that includes other files&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Settings&lt;/strong&gt;: read a JSON file, and watch it crash&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Native AOT&lt;/strong&gt;: turn it off, or use the JSON source generator&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;
  
  
  1. One file: run C# without a project
&lt;/h2&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// app.cs&lt;/span&gt;
&lt;span class="nc"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"Rover is ready"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dotnet run app.cs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;That's all. The SDK creates a hidden project for you.&lt;/p&gt;
&lt;h2&gt;
  
  
  2. #:include: split the app into files
&lt;/h2&gt;

&lt;p&gt;Move the rover logic to its own file, and include it from &lt;code&gt;app.cs&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;mars-rover/
├── app.cs
├── settings.json
├── rover/rover.cs
└── helpers/directions.cs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// app.cs&lt;/span&gt;
&lt;span class="cp"&gt;#:include rover/rover.cs&lt;/span&gt;

&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;rover&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Rover&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"N"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;rover&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"LMLMLMLMM"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nc"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rover&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;// 0 1 N&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One rule: &lt;strong&gt;only the entry file (&lt;code&gt;app.cs&lt;/code&gt;) can have top-level statements.&lt;/strong&gt; Included files only add types.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Nested includes: a file that includes other files
&lt;/h2&gt;

&lt;p&gt;The rover includes the directions helper itself, so &lt;code&gt;app.cs&lt;/code&gt; doesn't need to know about it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// rover/rover.cs&lt;/span&gt;
&lt;span class="cp"&gt;#:include ../helpers/directions.cs&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Rover&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;facing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;_x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_y&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;_dir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Directions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;IndexOf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;facing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;void&lt;/span&gt; &lt;span class="nc"&gt;Execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;commands&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;commands&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;{&lt;/span&gt;
            &lt;span class="k"&gt;switch&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="p"&gt;{&lt;/span&gt;
                &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s1"&gt;'L'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;_dir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Directions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Left&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_dir&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
                &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s1"&gt;'R'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;_dir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Directions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Right&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_dir&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
                &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="s1"&gt;'M'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
                    &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dy&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Directions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Delta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_dir&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
                    &lt;span class="n"&gt;_x&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;dx&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;_y&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;dy&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
                    &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
                &lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ArgumentException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;$&lt;span class="s2"&gt;"Unknown command: {c}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
            &lt;span class="p"&gt;}&lt;/span&gt;
        &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;override&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="nc"&gt;ToString&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; $&lt;span class="s2"&gt;"{_x} {_y} {Directions.Name(_dir)}"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// helpers/directions.cs&lt;/span&gt;
&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Directions&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="nc"&gt;Ring&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"NESW"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;// clockwise&lt;/span&gt;
    &lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="k"&gt;readonly&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="nc"&gt;Deltas&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nc"&gt;IndexOf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;
        &lt;span class="nc"&gt;Ring&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;IndexOf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Length&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;ArgumentException&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;$&lt;span class="s2"&gt;"Unknown direction: {f}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nc"&gt;Left&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="nc"&gt;Right&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;Delta&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Deltas&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="kt"&gt;char&lt;/span&gt; &lt;span class="nc"&gt;Name&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nc"&gt;Ring&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each file says what it needs, like a &lt;code&gt;using&lt;/code&gt;, but for files.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Settings: read a JSON file, and watch it crash
&lt;/h2&gt;

&lt;p&gt;Put the start position in &lt;code&gt;settings.json&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nl"&gt;"X"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"Y"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"Facing"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"N"&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read it from the folder of &lt;code&gt;app.cs&lt;/code&gt;, so it works wherever you run the app from:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;dir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AppContext&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;GetData&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"EntryPointFileDirectoryPath"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="s2"&gt;"."&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;File&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ReadAllText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Combine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dir&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"settings.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;JsonSerializer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Deserialize&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nc"&gt;Start&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;record&lt;/span&gt; &lt;span class="nc"&gt;Start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="no"&gt;Y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="nc"&gt;Facing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it, and it crashes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;System.InvalidOperationException:
Reflection-based serialization has been disabled for this application.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  5. Native AOT: turn it off, or use the JSON source generator
&lt;/h2&gt;

&lt;p&gt;File-based apps have &lt;strong&gt;Native AOT on by default&lt;/strong&gt;, and AOT turns off reflection-based JSON. You have two options.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Option A: turn AOT off.&lt;/strong&gt; Add one line at the top, and the code from step 4 works as is:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="cp"&gt;#:property PublishAot=false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Option B: keep AOT, use the JSON source generator.&lt;/strong&gt; A few more lines, but you keep the fast startup and the native binary:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="cp"&gt;#!/usr/bin/env dotnet&lt;/span&gt;
&lt;span class="cp"&gt;#:include rover/rover.cs&lt;/span&gt;

&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nc"&gt;System&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Json&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nc"&gt;System&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Text&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Serialization&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;dir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;AppContext&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;GetData&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"EntryPointFileDirectoryPath"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="s2"&gt;"."&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;File&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;ReadAllText&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Combine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dir&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"settings.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;JsonSerializer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Deserialize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;AppJson&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Default&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Start&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;rover&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Rover&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;Y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Facing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;rover&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Length&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"LMLMLMLMM"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nc"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rover&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;record&lt;/span&gt; &lt;span class="nc"&gt;Start&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="no"&gt;X&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="no"&gt;Y&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="nc"&gt;Facing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nc"&gt;JsonSerializable&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Start&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;partial&lt;/span&gt; &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;AppJson&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;JsonSerializerContext&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ dotnet run app.cs
0 1 N
$ dotnet run app.cs -- RRMMLM
1 -2 E
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a quick tool, A is fine. If startup time matters, choose B. The &lt;code&gt;#!&lt;/code&gt; line on top removes analyzer warning CA2266, and on macOS/Linux it lets you run &lt;code&gt;chmod +x app.cs &amp;amp;&amp;amp; ./app.cs&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Notes for later
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;dotnet run app.cs&lt;/code&gt; runs a C# file without a project.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;#:include path.cs&lt;/code&gt; adds another file (.NET 11).&lt;/li&gt;
&lt;li&gt;Included files can include other files. Only the entry file has top-level code.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;EntryPointFileDirectoryPath&lt;/code&gt; gives you the folder of your app file.&lt;/li&gt;
&lt;li&gt;AOT is on by default: use &lt;code&gt;#:property PublishAot=false&lt;/code&gt; or the JSON source generator.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Would you use this for small tools, or stay with projects? Tell me in comments. 🙂&lt;/p&gt;

</description>
      <category>dotnet</category>
      <category>csharp</category>
      <category>tutorial</category>
      <category>programming</category>
    </item>
    <item>
      <title>What SKIP LOCKED Solves, and What It Leaves to You</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 27 Oct 2026 12:00:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/what-skip-locked-solves-and-what-it-leaves-to-you-bhk</link>
      <guid>https://dev.to/mrviduus/what-skip-locked-solves-and-what-it-leaves-to-you-bhk</guid>
      <description>&lt;p&gt;&lt;code&gt;SKIP LOCKED&lt;/code&gt; is an option for &lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt;. It tells the database: if another transaction has already locked a row, don't wait for it. Skip that row and return the next one that is free.&lt;/p&gt;

&lt;p&gt;It sounds like a small detail, but it was made for a very specific job: letting many workers take tasks from the same table without blocking each other. Oracle used it internally for its own message queues as early as the late 90s. Postgres added it in version 9.5 (2016), and MySQL in 8.0 (2018). SQL Server has the same idea as the &lt;code&gt;READPAST&lt;/code&gt; table hint.&lt;/p&gt;

&lt;p&gt;In the last few years it has become popular because of one promise: you may not need Redis or RabbitMQ for background jobs, because your database can be the queue.&lt;/p&gt;

&lt;p&gt;I wanted to understand how true that is. Then I noticed that my own side project already used it, and that one of its queues could leave a job stuck forever. The claim used &lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt;, so two workers could never pick up the same job. But nothing handled a worker that crashed after the claim. The job stayed &lt;code&gt;running&lt;/code&gt;, workers only looked for &lt;code&gt;queued&lt;/code&gt; jobs, and another check refused to create a new job for the same item while one was still running. One crash could block that item for good.&lt;/p&gt;

&lt;p&gt;That made me look more closely at what &lt;code&gt;SKIP LOCKED&lt;/code&gt; actually guarantees, and what is still left to you.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiteu9m27yoj1caa4owl6.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiteu9m27yoj1caa4owl6.gif" alt="Two terminals side by side. Worker A takes job 777 and is slow. Worker B takes jobs 778 to 786, then takes 777 again as attempt 2 after A's lease expires. A finishes late and prints LOST job 777." width="799" height="225"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Two workers, one queue table, a real run at 2x speed. A (left) is too slow, B (right) takes job 777 back as attempt 2, and A's late result is ignored. By the end of this post, every line here will make sense.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Here is what we'll go through:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;SKIP LOCKED&lt;/strong&gt;: how one statement stops two workers from taking the same job&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lease&lt;/strong&gt;: how a job comes back after a worker crashes&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Heartbeat&lt;/strong&gt;: how a slow worker keeps its job without losing it to a timeout&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fencing token&lt;/strong&gt;: whose result counts when two workers own one job&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Idempotency key&lt;/strong&gt;: how to avoid repeating a payment or an email&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The cost&lt;/strong&gt;: what the database pays for being your queue&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The verdict&lt;/strong&gt;: what I chose for my side project, and when this approach fits&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;
  
  
  1. SKIP LOCKED: claiming a job
&lt;/h2&gt;

&lt;p&gt;Suppose a worker finds a queued job and then updates its status in a separate statement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'queued'&lt;/span&gt; &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;777&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two workers can read the same row before either of them updates it. Both start processing job 777. If the job is "send a payment", the payment goes out twice.&lt;/p&gt;

&lt;p&gt;In Postgres, we can select and claim the job in one statement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;started_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;
  &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'queued'&lt;/span&gt;
  &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
  &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;SKIP&lt;/span&gt; &lt;span class="n"&gt;LOCKED&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;RETURNING&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The subquery locks the selected row until the transaction ends. If another worker runs the same query at that moment, it skips the locked row and takes the next queued job. Nobody waits, and nobody gets the same job.&lt;/p&gt;

&lt;p&gt;This is the part &lt;code&gt;SKIP LOCKED&lt;/code&gt; solves, and it solves it well.&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Lease: recovering after a crash
&lt;/h2&gt;

&lt;p&gt;Here is the detail that is easy to miss. The row lock lives only until the transaction commits. After that, nothing is locked. The only thing that keeps other workers away from the job is its status, &lt;code&gt;running&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;So if the worker crashes after the commit, the job stays &lt;code&gt;running&lt;/code&gt; forever. No error, no retry, no alert. This is exactly what happened in my side project.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;SKIP LOCKED&lt;/code&gt; protects the moment of the claim. What happens after that is your responsibility.&lt;/p&gt;

&lt;p&gt;A common fix is to give each claim a time limit, often called a &lt;strong&gt;lease&lt;/strong&gt;. The worker owns the job only for a certain time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'queued'&lt;/span&gt;
   &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;started_at&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="s1"&gt;'10 minutes'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the lease expires, another worker can retry the job. The queue recovers on its own, without someone resetting rows by hand. In my case I used one hour and a maximum of three attempts, after which the job is marked &lt;code&gt;failed&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The hard part is choosing the time. A worker waiting on a slow API looks the same as a worker that has crashed. If the lease expires while the first worker is still processing, a second worker starts the same job. Now the job has two owners.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Heartbeat: keeping a slow worker's job
&lt;/h2&gt;

&lt;p&gt;A &lt;strong&gt;heartbeat&lt;/strong&gt; reduces this risk. Instead of counting the lease from &lt;code&gt;started_at&lt;/code&gt;, the job stores &lt;code&gt;lease_until&lt;/code&gt;, and the worker keeps moving it forward while it is alive, for example every 30 seconds:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lease_until&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="s1"&gt;'1 minute'&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;777&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Other workers reclaim only jobs where &lt;code&gt;lease_until &amp;lt; now()&lt;/code&gt;. A slow worker that is still alive keeps its job, and a crashed worker loses it about a minute later.&lt;/p&gt;

&lt;p&gt;But a heartbeat can't remove the risk completely. A long GC pause or a lost connection can stop the heartbeat while the worker is still processing. The lease expires, another worker takes the job, and then the first one wakes up and continues.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Fencing token: who owns the result
&lt;/h2&gt;

&lt;p&gt;If two workers can end up with the same job, we need to decide whose result counts. The usual answer is a &lt;strong&gt;fencing token&lt;/strong&gt;, an idea well explained in Martin Kleppmann's &lt;em&gt;Designing Data-Intensive Applications&lt;/em&gt;. Each claim gets a number that only grows, and every final write checks that the number is still current:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- claim: the attempt number goes from 1 to 2&lt;/span&gt;
&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt; &lt;span class="n"&gt;RETURNING&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- finish: worker A still has attempt 1&lt;/span&gt;
&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'done'&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;777&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- the row already has attempt 2, so 0 rows change&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The older attempt updates nothing and knows it was replaced. The newer attempt saves its result.&lt;/p&gt;

&lt;p&gt;The cost is discipline. Every write that finishes a job has to check the number, and the result and the status should be saved in one transaction.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Idempotency key: when the work leaves the database
&lt;/h2&gt;

&lt;p&gt;A fencing token only protects writes that check it. If the old worker has already called a payment API or sent an email, the database can't undo that.&lt;/p&gt;

&lt;p&gt;For external calls you need an &lt;strong&gt;idempotency key&lt;/strong&gt;: the same job always sends the same key, for example &lt;code&gt;payment-777&lt;/code&gt;, and the receiving service ignores a second request with that key. Stripe supports this with an &lt;code&gt;Idempotency-Key&lt;/code&gt; header.&lt;/p&gt;

&lt;p&gt;This is why "exactly once" is mostly a myth in job processing. In practice you get &lt;strong&gt;"at least once"&lt;/strong&gt;, and you make repeats harmless: fencing inside the database, idempotency keys outside it.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. The cost: what the database pays
&lt;/h2&gt;

&lt;p&gt;When the queue lives in your main database, the database carries the load:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Bloat.&lt;/strong&gt; In Postgres, every &lt;code&gt;UPDATE&lt;/code&gt; creates a new version of the row. A queue table changes all the time, so &lt;code&gt;VACUUM&lt;/code&gt; has a lot of work, and a long-running transaction elsewhere can stop it from cleaning up. Brandur Leach wrote &lt;a href="https://brandur.org/postgres-queues" rel="noopener noreferrer"&gt;a detailed post&lt;/a&gt; about a queue that slowed down for exactly this reason.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Primary only.&lt;/strong&gt; You can't lock rows on a read replica, so all queue traffic goes to the primary. If the primary is down, so is your queue.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No strict order.&lt;/strong&gt; &lt;code&gt;SKIP LOCKED&lt;/code&gt; returns the next free row, not the next row in line. If jobs for one customer must run in order, you need extra design.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Connections.&lt;/strong&gt; Every worker holds a connection. Shopify used &lt;code&gt;SKIP LOCKED&lt;/code&gt; for &lt;a href="https://shopify.engineering/scaling-inventory-reservations" rel="noopener noreferrer"&gt;checkout reservations&lt;/a&gt;, and in their case the real limit was connection usage, not locking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Throughput.&lt;/strong&gt; Thousands of jobs per second are fine. Much higher volumes are where Kafka and similar systems make more sense.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  7. The verdict
&lt;/h2&gt;

&lt;h3&gt;
  
  
  What I chose for my side project
&lt;/h3&gt;

&lt;p&gt;After all this, I had a list: atomic claim, lease, heartbeat, fencing on every final write, idempotency keys. I could have built all of it.&lt;/p&gt;

&lt;p&gt;I didn't. My side project runs a single worker, so I enforced that instead. The worker takes a Postgres advisory lock at startup, and a second copy waits until the first one exits. I also added a metric and an alert for jobs that stay &lt;code&gt;running&lt;/code&gt; too long.&lt;/p&gt;

&lt;p&gt;That choice has its own caveat: if the database connection drops, the advisory lock is lost and isn't taken again. For a side project, that's acceptable. For a payment system, it wouldn't be. But it kept the design close to what the project actually needs, and it doesn't pretend to be safe for ten workers when there is only one.&lt;/p&gt;

&lt;h3&gt;
  
  
  So, is it a silver bullet?
&lt;/h3&gt;

&lt;p&gt;No. It's a very good tool when you know what comes with it.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;A table with SKIP LOCKED fits when&lt;/th&gt;
&lt;th&gt;Consider something else when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;You already run Postgres or MySQL&lt;/td&gt;
&lt;td&gt;You need very high throughput&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;You have thousands of jobs, not millions&lt;/td&gt;
&lt;td&gt;Many services must read the same events&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The job must be created in the same transaction as your data&lt;/td&gt;
&lt;td&gt;Jobs must run in strict order&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;You are ready to add leases and alerts for stuck jobs&lt;/td&gt;
&lt;td&gt;The primary database is already under pressure&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Try it yourself
&lt;/h2&gt;

&lt;p&gt;Everything above fits in one small worker of about 60 lines: a claim with &lt;code&gt;SKIP LOCKED&lt;/code&gt;, a 10-second lease, and &lt;code&gt;attempt&lt;/code&gt; as the fencing token. I wrote it in Node.js and in C#, and the GIF at the top is a real run of two copies against the same table. The slow one loses its lease, the other one takes the job as attempt 2, and the late write changes 0 rows.&lt;/p&gt;

&lt;p&gt;The heart of the C# version (Npgsql), shortened:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Claim one job: UPDATE ... WHERE id = (SELECT ... FOR UPDATE SKIP LOCKED) RETURNING id, attempt&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;var&lt;/span&gt; &lt;span class="n"&gt;claim&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;CreateCommand&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ClaimSql&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;var&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;claim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ExecuteReaderAsync&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(!&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ReadAsync&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;   &lt;span class="c1"&gt;// queue is empty&lt;/span&gt;
&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;GetInt32&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;GetInt32&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;

&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;DoWorkAsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;// Finish only if I still own the job: attempt is the fencing token&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;var&lt;/span&gt; &lt;span class="n"&gt;finish&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;CreateCommand&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"UPDATE jobs SET status = 'done' WHERE id = $1 AND attempt = $2"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="n"&gt;finish&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Parameters&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;AddWithValue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="n"&gt;finish&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Parameters&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;AddWithValue&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;finish&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ExecuteNonQueryAsync&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;==&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"LOST job &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: someone else owns it now"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The code, the real output and a step-by-step tutorial you can run in two terminals are &lt;a href="https://github.com/mrviduus/blog/tree/main/skip-locked-lab" rel="noopener noreferrer"&gt;on GitHub&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Notes for later
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;code&gt;SKIP LOCKED&lt;/code&gt; protects the claim. After the commit, only the status protects the job.&lt;/li&gt;
&lt;li&gt;A crash after the commit leaves a stuck job. Add a lease.&lt;/li&gt;
&lt;li&gt;A lease can't tell a dead worker from a slow one. Measure real job durations before you choose the timeout.&lt;/li&gt;
&lt;li&gt;A lease shorter than the work creates two owners. Use a fencing token on final writes.&lt;/li&gt;
&lt;li&gt;Fencing stops at the database. For external calls, use idempotency keys.&lt;/li&gt;
&lt;li&gt;"Exactly once" is really "at least once" plus safe repeats.&lt;/li&gt;
&lt;li&gt;A single worker is a valid design, as long as you enforce it and write it down.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Have you moved a queue from Redis to your database? What was the first thing that surprised you?&lt;/p&gt;




&lt;p&gt;&lt;em&gt;The side project in this post is &lt;a href="https://textstack.app" rel="noopener noreferrer"&gt;TextStack&lt;/a&gt;, a reader for technical books that explains hard terms in the context of the book. It's open source: &lt;a href="https://github.com/mrviduus/textstack" rel="noopener noreferrer"&gt;github.com/mrviduus/textstack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>database</category>
      <category>distributedsystems</category>
      <category>dotnet</category>
    </item>
    <item>
      <title>Behind Cloudflare and nginx, All My Users Had the Same IP</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 06 Oct 2026 12:00:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/behind-cloudflare-and-nginx-all-my-users-had-the-same-ip-3hae</link>
      <guid>https://dev.to/mrviduus/behind-cloudflare-and-nginx-all-my-users-had-the-same-ip-3hae</guid>
      <description>&lt;p&gt;My admin login page allows 5 wrong passwords per minute. After that, it should block you.&lt;/p&gt;

&lt;p&gt;I tried 9. It never blocked me.&lt;/p&gt;

&lt;p&gt;I checked the other rate limits in my app, and they did not work either. While I was fixing that, I found a second problem: my app saw every user in production as the same IP address. Nobody noticed, because the limiter was off. If I had only fixed the first bug, the limiter would have started to work with one counter for the whole internet.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the limiter did nothing
&lt;/h2&gt;

&lt;p&gt;In ASP.NET Core, every request goes through a chain of middleware, one step after another, and the order matters.&lt;/p&gt;

&lt;p&gt;One step is routing. It decides "this request is for the login endpoint". The rate limiter needs this answer, because every endpoint has its own limit.&lt;/p&gt;

&lt;p&gt;In my app, the rate limiter was placed before routing. So it asked "which endpoint is this?" too early, got no answer, and let the request go. There was no error and nothing in the logs.&lt;/p&gt;

&lt;p&gt;The fix was to move one line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;UseRouting&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="n"&gt;app&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;UseRateLimiter&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt; &lt;span class="c1"&gt;// must come after routing&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;(If you never call &lt;code&gt;UseRouting()&lt;/code&gt; yourself, ASP.NET adds it at the start for you, so you can't get this bug. I call it myself, later in the chain, and the limiter ended up above it.)&lt;/p&gt;

&lt;p&gt;After this fix the limits started to work, and three more problems showed up. They were there all the time, but they could not hurt anyone while the limiter did nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Every user had the same IP
&lt;/h2&gt;

&lt;p&gt;My limits count requests per IP address. But my app sits behind Cloudflare and nginx, so the app never talks to the user directly. The user's IP comes in a header, &lt;code&gt;X-Forwarded-For&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;nginx builds this header like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="k"&gt;proxy_set_header&lt;/span&gt; &lt;span class="s"&gt;X-Forwarded-For&lt;/span&gt; &lt;span class="nv"&gt;$proxy_add_x_forwarded_for&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;$proxy_add_x_forwarded_for&lt;/code&gt; takes the header it received and adds the address of whoever connected to nginx. In my setup that is cloudflared, running on the same machine. So every request arrived like this (the user IP here is an example):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;X-Forwarded-For: 203.0.113.42, 127.0.0.1
                 real user      cloudflared, added by nginx

before the fix, my app used:  127.0.0.1
after the fix, my app uses:   203.0.113.42
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;By default, ASP.NET's forwarded headers middleware reads only one entry, the last one. And the last one was always &lt;code&gt;127.0.0.1&lt;/code&gt;. So my "3 guest sign-ups per IP" limit would have been 3 guest sign-ups for everybody.&lt;/p&gt;

&lt;p&gt;There was a second problem in the same place. My config also trusted this header from anyone. In production the first problem hid it, but without that extra nginx hop, a client could write any IP in the header and get a fresh counter. I checked this in my test setup, and it worked.&lt;/p&gt;

&lt;p&gt;This is the fix for both, a bit shortened:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="n"&gt;builder&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Services&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Configure&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;ForwardedHeadersOptions&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;(&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ForwardedHeaders&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ForwardedHeaders&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;XForwardedFor&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt; &lt;span class="n"&gt;ForwardedHeaders&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;XForwardedProto&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ForwardLimit&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="c1"&gt;// read the whole list, not only the last entry&lt;/span&gt;

    &lt;span class="c1"&gt;// trust only my own hops: loopback and private networks&lt;/span&gt;
    &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;KnownIPNetworks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Clear&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;KnownIPNetworks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Net&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;IPNetwork&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"127.0.0.0/8"&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
    &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;KnownIPNetworks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Net&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;IPNetwork&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"::1/128"&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
    &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;KnownIPNetworks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Net&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;IPNetwork&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"10.0.0.0/8"&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
    &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;KnownIPNetworks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Net&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;IPNetwork&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"172.16.0.0/12"&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
    &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;KnownIPNetworks&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;System&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Net&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;IPNetwork&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;"192.168.0.0/16"&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now the middleware reads the list from the end, skips the addresses it trusts (my own proxies), and stops at the first one it doesn't trust. That is the real user. If a client puts a fake IP into the header, it ends up further left, and the middleware never reaches it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ten wrong passwords would lock out everyone
&lt;/h2&gt;

&lt;p&gt;My login limit was not per IP at all. It was one counter for the whole site, 10 tries per minute.&lt;/p&gt;

&lt;p&gt;While the limiter did nothing, it did not matter. With the limiter on, anyone could send ten wrong passwords and close login for all users, and password reset too, because it used the same limit. I changed it to one counter per IP. An attacker still gets 10 tries, and everyone else can still log in.&lt;/p&gt;

&lt;h2&gt;
  
  
  My tests were green because of the bug
&lt;/h2&gt;

&lt;p&gt;When the limits started to work, my integration tests started to fail. They send a lot of requests quickly, and they had passed only because nothing was ever blocked. I made the limits configurable, so CI can use higher numbers.&lt;/p&gt;

&lt;p&gt;I also added two tests that check the limiter itself, not the endpoints:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;send one request more than the limit allows, and expect &lt;code&gt;429 Too Many Requests&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;send requests as two different users through the same proxy, and expect two separate counters&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Before this, I had never seen my limiter return &lt;code&gt;429&lt;/code&gt;. I think that was the real warning, and I missed it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Check your own app
&lt;/h2&gt;

&lt;p&gt;Send more requests than your limit allows. Only to your own app, of course:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;i &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;seq &lt;/span&gt;1 11&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;curl &lt;span class="nt"&gt;-s&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; /dev/null &lt;span class="nt"&gt;-w&lt;/span&gt; &lt;span class="s2"&gt;"%{http_code}&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-X&lt;/span&gt; POST https://your-app/login
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You should see &lt;code&gt;429&lt;/code&gt; at the end. If you see only &lt;code&gt;200&lt;/code&gt;, your limit does not work. Then try again with a different fake &lt;code&gt;X-Forwarded-For&lt;/code&gt; header each time. If the &lt;code&gt;429&lt;/code&gt; goes away, anyone can bypass your limit.&lt;/p&gt;

&lt;p&gt;Have you ever checked that your rate limiter really returns &lt;code&gt;429&lt;/code&gt; in production? I am curious if I am the only one who didn't.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://vasyl.blog/2026/10/04/behind-cloudflare-and-nginx-all-my-users-had-the-same-ip/" rel="noopener noreferrer"&gt;vasyl.blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I found this while building &lt;a href="https://textstack.app" rel="noopener noreferrer"&gt;TextStack&lt;/a&gt;, a reader for technical books that explains hard terms in the book's context. The Android app is in closed testing, and I need a few more testers, so if you read on your phone, &lt;a href="https://play.google.com/apps/testing/app.textstack.mobile" rel="noopener noreferrer"&gt;you can join the beta here&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>security</category>
      <category>webdev</category>
      <category>dotnet</category>
      <category>devops</category>
    </item>
    <item>
      <title>Split-Brain Needs a Cluster. Mine Happened on One Server.</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Wed, 30 Sep 2026 12:30:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/split-brain-needs-a-cluster-mine-happened-on-one-server-1jf4</link>
      <guid>https://dev.to/mrviduus/split-brain-needs-a-cluster-mine-happened-on-one-server-1jf4</guid>
      <description>&lt;p&gt;Today I want to talk about split-brain. The name sounds dramatic, and the problem is dramatic too. One system splits into two halves, and each half thinks it is the boss. Both accept writes. The data slowly goes in different directions, and there is no clean way to merge it back.&lt;/p&gt;

&lt;h2&gt;
  
  
  How it happens
&lt;/h2&gt;

&lt;p&gt;A crash almost never causes it. A crash is the easy case: the node is dead, so it writes nothing. For split-brain you need everyone alive, but wrong about the world.&lt;/p&gt;

&lt;p&gt;There are two common ways to get there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A network partition.&lt;/strong&gt; Two nodes back each other up. The link between them dies. Each one thinks the other is dead, and both make themselves the leader.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A paused process.&lt;/strong&gt; The leader freezes for some time (a long GC pause, a VM migration). The cluster picks a new leader. Then the old one wakes up and keeps working. It does not know it is old. Nobody sends you a message saying "you are stale".&lt;/p&gt;

&lt;h2&gt;
  
  
  The classic cases
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Database pairs.&lt;/strong&gt; After a bad failover you have two primaries. Both take writes, and replication cannot fix it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Leader-based clusters.&lt;/strong&gt; After a partition the old leader keeps answering. Clients get stale reads, or writes that look accepted but later get lost.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Many data centers.&lt;/strong&gt; During a partition both sides keep taking writes. When the network is back, you have two histories of the same data.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Distributed locks.&lt;/strong&gt; A lock times out while its owner is paused. A second worker takes it. Now two processes hold the same lock.&lt;/p&gt;

&lt;p&gt;Different setups, same problem: two writers with an old idea of who owns the data.&lt;/p&gt;

&lt;h2&gt;
  
  
  How systems protect themselves
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Quorum.&lt;/strong&gt; Nothing important happens without a majority. This is why clusters have 3 or 5 nodes: only one side of a partition can win.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Terms.&lt;/strong&gt; In Raft every message has a term number. If a leader comes from an old term, the others just refuse it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fencing tokens.&lt;/strong&gt; Every claim gets a number that only grows. The storage checks this number at the moment of the write. So a stale writer is stopped where it writes, not where it decides.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;STONITH.&lt;/strong&gt; The brutal option: power off the other node, so it cannot write anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  My surprise: you do not need a cluster
&lt;/h2&gt;

&lt;p&gt;When I read about split-brain in Designing Data-Intensive Applications, I thought: interesting, but not my problem. My side project runs on one server with one Postgres. Nothing to split.&lt;/p&gt;

&lt;p&gt;Then I re-read my own postmortems and found this problem twice.&lt;/p&gt;

&lt;p&gt;First time: a deploy and a background rebuild both thought they owned the same build folder. The rebuild said "success", but that report was already stale, and the worker trusted it more than the disk. For 50 minutes Google saw 404 on every page.&lt;/p&gt;

&lt;p&gt;Second time: one reader raced with themselves. A heartbeat and a tab close both did read-then-insert on the same progress row. Both saw no row, both did INSERT, and the second one failed with error 23505.&lt;/p&gt;

&lt;p&gt;Both fixes use the fencing idea, just very small. For the first one, a timestamp that the wait step must beat. For the second one, a unique index, so the database rejects the second writer, and we catch the error and merge. No Raft, no quorum. Same main idea: check freshness at the place where you write.&lt;/p&gt;

&lt;h2&gt;
  
  
  If you get this in an interview
&lt;/h2&gt;

&lt;p&gt;"Explain split-brain" is a classic system design question. The textbook answer is two leaders in a cluster. A stronger answer: any two writers with an old idea of who owns the data, and it can happen on one server. Give the definition, name the defenses (quorum, terms, fencing tokens), and if you have a real story, tell it. A story beats a definition every time.&lt;/p&gt;

&lt;p&gt;Where did a distributed systems problem bite you in a system you thought was too small for it? I would love to hear your story.&lt;/p&gt;




&lt;p&gt;I build &lt;a href="https://textstack.app/" rel="noopener noreferrer"&gt;TextStack&lt;/a&gt;, a reading app for developers who learn AI engineering. Both postmortems from this post are public in the repo: &lt;a href="https://github.com/mrviduus/textstack" rel="noopener noreferrer"&gt;github.com/mrviduus/textstack&lt;/a&gt;, folder docs/incidents. Nothing here is made up.&lt;/p&gt;

&lt;p&gt;TextStack is starting closed testing now, and I need a few early testers. If you read technical books like DDIA and want early access, leave a comment here.&lt;/p&gt;

</description>
      <category>systemdesign</category>
      <category>postgres</category>
      <category>programming</category>
      <category>career</category>
    </item>
    <item>
      <title>My Uptime Monitor Was Set to Every 5 Minutes. In July It Ran 15 Times a Day.</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 18 Aug 2026 13:30:20 +0000</pubDate>
      <link>https://dev.to/mrviduus/my-uptime-monitor-was-set-to-every-5-minutes-in-july-it-ran-15-times-a-day-3if8</link>
      <guid>https://dev.to/mrviduus/my-uptime-monitor-was-set-to-every-5-minutes-in-july-it-ran-15-times-a-day-3if8</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for &lt;a href="https://dev.to/bugsmash"&gt;DEV's Summer Bug Smash: Smash Stories&lt;/a&gt; powered by &lt;a href="https://sentry.io/" rel="noopener noreferrer"&gt;Sentry&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;🔨 #bugsmash, week by week: &lt;a href="https://dev.to/mrviduus/nothing-crashed-nothing-errored-my-cpu-sat-at-390-for-an-hour-335a"&gt;a 390% CPU hour nobody noticed&lt;/a&gt;, &lt;a href="https://dev.to/mrviduus/stuck-at-preparing-00-forever-my-state-machine-had-no-exit-1pmb"&gt;a state machine with no exit&lt;/a&gt;, &lt;a href="https://dev.to/mrviduus/my-backup-leaked-156-gb-filled-the-disk-and-broke-itself-the-fix-was-one-letter-3ldd"&gt;a backup that leaked 156 GB&lt;/a&gt;, &lt;a href="https://dev.to/mrviduus/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline-12lo"&gt;a monitor that caught its own lies&lt;/a&gt;. Week five: the monitor that was quietly not running at all.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;I had not opened the TextStack repo properly in a while. I have been working through Claude Code for weeks, task by task, and each task looked small. Yesterday I opened the whole thing and looked at it as one project. It has grown into something I cannot hold in my head anymore. That is a separate problem and I will get to it at the end.&lt;/p&gt;

&lt;p&gt;While I was looking around I opened &lt;code&gt;.github/workflows/health-check.yml&lt;/code&gt;. It is the file that tells me my site is alive.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;schedule&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;cron&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;*/5&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every 5 minutes. It curls &lt;code&gt;/health&lt;/code&gt; on textstack.app and textstack.dev, then does two smoke checks, book listing and search. If any of them fails, the run goes red and I get a notification.&lt;/p&gt;

&lt;p&gt;288 runs a day. That is what the file says.&lt;/p&gt;

&lt;p&gt;Then I counted the actual runs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The number
&lt;/h2&gt;

&lt;p&gt;The workflow was created on 6 January 2026. Here is what it did, month by month.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Month&lt;/th&gt;
&lt;th&gt;Runs per day&lt;/th&gt;
&lt;th&gt;Percent of 288&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;January (from the 6th)&lt;/td&gt;
&lt;td&gt;107&lt;/td&gt;
&lt;td&gt;37%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;February&lt;/td&gt;
&lt;td&gt;74&lt;/td&gt;
&lt;td&gt;26%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;March&lt;/td&gt;
&lt;td&gt;73&lt;/td&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;April&lt;/td&gt;
&lt;td&gt;53&lt;/td&gt;
&lt;td&gt;18%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;May&lt;/td&gt;
&lt;td&gt;35&lt;/td&gt;
&lt;td&gt;12%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;June&lt;/td&gt;
&lt;td&gt;26&lt;/td&gt;
&lt;td&gt;9%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;July&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;August (first 11 days)&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;7%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;It went down every single month from January to July.&lt;/p&gt;

&lt;p&gt;In July my health check ran 15 times a day. I had asked for 288. That is one check every 96 minutes instead of every 5.&lt;/p&gt;

&lt;p&gt;And it never started at 100 percent either. Even in the first month it was already at about a third.&lt;/p&gt;

&lt;p&gt;I did not notice any of this for seven months.&lt;/p&gt;

&lt;h2&gt;
  
  
  What made me look
&lt;/h2&gt;

&lt;p&gt;On 6 August GitHub Actions went down. The incident opened at 15:22 UTC and was fully resolved at 02:04 UTC the next day, so almost 11 hours. At the peak, 71 percent of workflow runs were failing with infrastructure errors. My own work stopped for about four hours that afternoon, which is why I remembered the date.&lt;/p&gt;

&lt;p&gt;So I checked what my health check did in that window. Four runs. Two of them failed. The largest gap that day was 5 hours 25 minutes, from 18:33 to 23:59, and that gap sits inside the incident.&lt;/p&gt;

&lt;p&gt;At first I thought I had a clean story. GitHub went down, my monitor went down with it, blind for five hours.&lt;/p&gt;

&lt;p&gt;Then I looked at the same day before the incident started. There was already a gap of 3 hours 22 minutes, from 00:08 to 03:30, hours before anything broke. And on 3, 4 and 5 August the workflow ran 12, 13 and 13 times.&lt;/p&gt;

&lt;p&gt;So no. The outage did not blind me. The outage was just the day I happened to look. It had been like this since spring.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why
&lt;/h2&gt;

&lt;p&gt;This is documented. It is in the GitHub docs, in the part about the &lt;code&gt;schedule&lt;/code&gt; event.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Scheduled events can be delayed during periods of high load of GitHub Actions workflow runs, and if the load is high enough, some queued jobs may be dropped.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Dropped. Not delayed. Dropped.&lt;/p&gt;

&lt;p&gt;I knew that sentence existed. I read it as "sometimes it runs a bit late." It does not mean that. It means the run never happens and nothing tells you.&lt;/p&gt;

&lt;p&gt;There is a second half to this, and I want to be careful about how strongly I put it. GitHub itself has been under a lot of load this year. Commits went from about 1 billion in all of 2025 to 275 million per week by April 2026. Actions compute hit 2.1 billion minutes in a single week. Pull requests opened by AI agents went from about 4 million in September 2025 to more than 17 million in March 2026.&lt;/p&gt;

&lt;p&gt;My curve goes down while their curve goes up. That fits, and the mechanism is documented. But I only have my one repo, so I cannot prove the two curves are connected. I can only say the shape matches and GitHub says dropping is what happens under load.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part I did not expect
&lt;/h2&gt;

&lt;p&gt;Same repository, same seven months, same &lt;code&gt;schedule&lt;/code&gt; trigger. My daily database backup:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;schedule&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;cron&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;3&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;*'&lt;/span&gt;
&lt;span class="na"&gt;jobs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;backup&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;runs-on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;self-hosted&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;224 runs since January. Zero missing days. Not one. Every month it ran on exactly as many days as the month has, and in May and June it ran a couple of extra times because I triggered it by hand.&lt;/p&gt;

&lt;p&gt;So in the same repo one scheduled workflow degraded to 5 percent and the other stayed at 100 percent.&lt;/p&gt;

&lt;p&gt;I want to say the difference is &lt;code&gt;runs-on: self-hosted&lt;/code&gt;, because that is the obvious difference and it is the one I like. But the two workflows also differ in how often they ask for something. One asks 288 times a day, the other asks once. High frequency schedules are exactly what a queue sheds first. I cannot separate those two causes with the data I have, so I am not going to pretend I can.&lt;/p&gt;

&lt;p&gt;I did get one thing wrong before checking, and it is worth saying. I assumed self-hosted would not protect me on 6 August, because the root cause that day was invalid job assignments to runners, and job assignment is GitHub's side, not mine. That sounded right. The backup ran normally on 6 and 7 August anyway. My reasoning was fine and the answer was still wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  Three layers, all built by me, none of them checking anything
&lt;/h2&gt;

&lt;p&gt;Once I started pulling on this I found it was not one problem.&lt;/p&gt;

&lt;p&gt;My README says the project has uptime monitoring with UptimeRobot probes. There is a whole file for it, &lt;code&gt;docs/03-ops/uptime-monitoring.md&lt;/code&gt;. It is a good document. It has a table of three monitors with URLs, expected responses and intervals. It has numbered setup steps. It has an alert-response runbook with the exact commands to run for each kind of alert, down to which containers to check logs for.&lt;/p&gt;

&lt;p&gt;I do not have an UptimeRobot account. I never signed up. I wrote the whole document for the setup I was going to build, and then did not build it.&lt;/p&gt;

&lt;p&gt;The last section of that same file is the part that got me. It describes the GitHub Actions health check as internal belt-and-braces, useful when UptimeRobot itself has an outage, but redundant otherwise.&lt;/p&gt;

&lt;p&gt;Redundant. My documentation called it the backup to the real monitoring. It was the only monitoring. And it was running at 5 percent.&lt;/p&gt;

&lt;p&gt;Second layer: the cron says every 5 minutes. It runs every 96 minutes.&lt;/p&gt;

&lt;p&gt;Third layer: the Actions tab is green. A green tab means every run that happened passed. It looks exactly the same when almost no runs happened at all. There is no colour for "did not run."&lt;/p&gt;

&lt;p&gt;Three sources of confidence. All three mine. None of them lying on purpose, and none of them checking anything.&lt;/p&gt;

&lt;p&gt;This is the same shape as &lt;a href="https://dev.to/mrviduus/my-backup-leaked-156-gb-filled-the-disk-and-broke-itself-the-fix-was-one-letter-3ldd"&gt;a bug I wrote about a few weeks ago&lt;/a&gt;, where my backup verifier filled the disk of the machine it was verifying. Someone in the comments gave me the sentence for it: a verifier that shares a resource pool with the thing it verifies eventually becomes the largest consumer of it. This is one level up. A monitor that shares a platform with the thing it monitors inherits that platform's bad days, and stays silent about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I am not going to do
&lt;/h2&gt;

&lt;p&gt;The obvious fix is to sign up for an external monitoring service. Free tier, five minute checks, runs on somebody else's machines, done in twenty minutes.&lt;/p&gt;

&lt;p&gt;I am not going to do that, and I want to explain why, because the reason is the thing I actually learned this week.&lt;/p&gt;

&lt;p&gt;I opened this project yesterday and could not hold it in my head. It is one person's side project and it has a backend, a web app, an admin app, a mobile app, a worker, an MCP server, a browser extension, an SSG pipeline, and now eight workflows. Every one of those was a reasonable decision on the day I made it. Together they are more than I can carry.&lt;/p&gt;

&lt;p&gt;Every new service is not just the service. It is another account, another set of credentials, another page in the docs, another thing that is quietly broken in six months while I believe it is running. I know that last one is real now, because that is exactly what my UptimeRobot documentation was: a doc for a thing that did not exist.&lt;/p&gt;

&lt;p&gt;Adding a monitoring service to fix bad monitoring is how the pile got this big.&lt;/p&gt;

&lt;p&gt;So the rule I am taking out of this is not "add a monitor." It is narrower. Do not add a second thing where the first thing was never checked. And when you do keep something, keep it where you can see it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I am leaving it exactly where it is
&lt;/h2&gt;

&lt;p&gt;The obvious fix is to move the health check to &lt;code&gt;runs-on: self-hosted&lt;/code&gt;, next to the backup that has not missed a day in seven months. One line changed. 100 percent instead of 5 percent. I spent a day thinking that was the answer.&lt;/p&gt;

&lt;p&gt;It is the wrong answer, and working out why is the part of this I am actually glad about.&lt;/p&gt;

&lt;p&gt;My self-hosted runner is a machine in my flat. If that machine dies, a monitor running on it dies at the same moment and tells me nothing. I would find out on my next login, and I log in maybe once a week.&lt;/p&gt;

&lt;p&gt;GitHub is the opposite. When GitHub breaks, somebody whose job it is fixes it, usually within hours, without me. The 6 August outage lasted 11 hours and I did nothing about it.&lt;/p&gt;

&lt;p&gt;So the two options are not "5 percent reliable" versus "100 percent reliable." They are:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;A check that runs every 96 minutes and is not on my machine.&lt;/li&gt;
&lt;li&gt;A check that runs every 5 minutes and dies silently with my machine.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The second one has perfect uptime for every failure except the one failure I have no other way of seeing. My server dying is the thing I am most exposed to, because I am not looking at it. A monitor that shares its fate is not a monitor. It is a second thing that will be dead at the same time.&lt;/p&gt;

&lt;p&gt;So the degraded one is the one worth keeping. Not because 15 runs a day is good, but because it runs somewhere I do not control and do not have to maintain, and that is the whole property I need from it.&lt;/p&gt;

&lt;p&gt;What I give up is resolution. If the site goes down I might hear about it 96 minutes later instead of 5. For a project with my traffic that is fine, and I would rather know late than not know.&lt;/p&gt;

&lt;p&gt;The mistake I made for seven months was not putting the check on GitHub. That was right. The mistake was reading &lt;code&gt;*/5 * * * *&lt;/code&gt; as a promise and never checking whether it was kept. I optimised the number in the file and never looked at the number in reality.&lt;/p&gt;

&lt;p&gt;A monitor has two properties. How often it runs, and whether it can survive what it is watching. I spent all my attention on the first one. The second one is the one that decides whether it is a monitor at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trade used to be easier
&lt;/h2&gt;

&lt;p&gt;I want to be honest that I am less comfortable with this than I would have been a year ago.&lt;/p&gt;

&lt;p&gt;Leaning on GitHub used to be a decision you made once and forgot. It was background. It was there. That is what made "just put it on Actions" the obvious call for a solo project.&lt;/p&gt;

&lt;p&gt;It is not that obvious now, and the numbers say why. My own schedule went from 37 percent of what I asked for in January to 5 percent in July, and nothing about my repo changed in that time. GitHub had nine service-degrading incidents in May alone. On 6 August, Actions was broken for almost 11 hours with 71 percent of runs failing at the peak. In June, Microsoft started renting capacity from AWS to keep GitHub standing up, which is not a thing a company does when the situation is comfortable.&lt;/p&gt;

&lt;p&gt;The reason underneath is not a mystery either. Commits went from roughly 1 billion in all of 2025 to 275 million per week. Actions compute hit 2.1 billion minutes in one week. Pull requests opened by AI agents went from about 4 million to over 17 million in six months. The platform is absorbing a change in how code gets written, and the parts that get squeezed first are the low priority background ones. A 5 minute cron on a small public repo is exactly that.&lt;/p&gt;

&lt;p&gt;So I am keeping the trade, but I am not calling it free anymore. It has a cost that is going up, and the cost is paid in silence, which is the worst currency for a monitor.&lt;/p&gt;

&lt;p&gt;What I will do is check the number again in three months. If 5 percent has become 2 percent, then the check has stopped being a check and I will have to pay for something. I would rather find that out by counting than by an outage.&lt;/p&gt;

&lt;p&gt;The one thing I am changing is the documentation. &lt;code&gt;docs/03-ops/uptime-monitoring.md&lt;/code&gt; still describes an UptimeRobot setup I never built, and calls the GitHub check redundant. It is not redundant. It is the whole thing. That file now says what actually exists, roughly how often it really runs, and what it does not catch.&lt;/p&gt;

&lt;p&gt;That edit does not improve my uptime by one second. It just means that in six months I will not believe something that is not true.&lt;/p&gt;

&lt;p&gt;What I keep from all this is the number. 15 runs a day out of 288, for a month, without noticing. And the thing that finally told me was not any of my monitoring. It was me opening the repo and counting by hand.&lt;/p&gt;

&lt;p&gt;If you have a scheduled workflow you rely on, go count its runs. Not the failures. The runs. It takes two minutes and I would like to know whether my curve is mine alone.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I build TextStack, an open-source reader for technical books, in .NET. The full write-up lives on &lt;a href="https://vasyl.blog/2026/08/11/uptime-monitor-every-5-minutes-ran-15-times-a-day/" rel="noopener noreferrer"&gt;my blog&lt;/a&gt;. &lt;a href="https://github.com/mrviduus/textstack" rel="noopener noreferrer"&gt;github.com/mrviduus/textstack&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>bugsmash</category>
      <category>github</category>
      <category>devops</category>
    </item>
    <item>
      <title>Nobody Alerts on Silence: Wiring Sentry Into an LLM Pipeline</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 11 Aug 2026 12:00:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline-12lo</link>
      <guid>https://dev.to/mrviduus/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline-12lo</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for &lt;a href="https://dev.to/bugsmash"&gt;DEV's Summer Bug Smash: Clear the Lineup&lt;/a&gt; powered by &lt;a href="https://sentry.io/" rel="noopener noreferrer"&gt;Sentry&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;🔨 #bugsmash, week by week: &lt;a href="https://dev.to/mrviduus/nothing-crashed-nothing-errored-my-cpu-sat-at-390-for-an-hour-335a"&gt;a 390% CPU hour nobody noticed&lt;/a&gt;, &lt;a href="https://dev.to/mrviduus/stuck-at-preparing-00-forever-my-state-machine-had-no-exit-1pmb"&gt;a state machine with no exit&lt;/a&gt;, &lt;a href="https://dev.to/mrviduus/my-backup-leaked-156-gb-filled-the-disk-and-broke-itself-the-fix-was-one-letter-3ldd"&gt;a backup that leaked 156 GB&lt;/a&gt;. Week four is the finale: I wired monitoring into the pipeline that produced all three — and its best catch was itself.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Project Overview
&lt;/h2&gt;

&lt;p&gt;TextStack is an open-source reader for technical books, built in .NET: an ASP.NET Core API, a background Worker, PostgreSQL + pgvector, React on top. The LLM pipeline does translation, word explanations, "Ask this book" RAG, and three production agents (Enrichment, Librarian, Tutor), routed between a local Ollama and OpenAI by a config-driven router. The code is public: &lt;a href="https://github.com/mrviduus/textstack" rel="noopener noreferrer"&gt;github.com/mrviduus/textstack&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Bug Fix or Performance Improvement
&lt;/h2&gt;

&lt;p&gt;Three weeks ago a user's PDF fell through my LLM router onto a CPU-only Ollama container instead of GPT-4.1, and my CPU sat at 390% for an hour. Zero exceptions. Zero error logs. Zero alerts. And when I went to see what my existing observability had recorded, the answer was &lt;em&gt;nothing at all&lt;/em&gt;: the OTLP exporter pointed at an Aspire dashboard container that is profile-gated and doesn't run in production. Every span my services had ever produced in prod had been fired into a closed socket.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Observability you never read is indistinguishable from observability you never installed.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The one-line config fix was submission #1. This submission is the fix for the &lt;em&gt;class&lt;/em&gt; of bug — a system that has no way to make a sound when it does the wrong thing successfully:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the router now records &lt;strong&gt;why&lt;/strong&gt; it picked a provider, not just which one;&lt;/li&gt;
&lt;li&gt;expensive tasks landing on the default provider fire a throttled Sentry alert;&lt;/li&gt;
&lt;li&gt;provider failures the client deliberately swallows now report before returning their empty response;&lt;/li&gt;
&lt;li&gt;the Worker probes provider reachability at startup and a circuit breaker stops a dead provider from eating 50 × 90 s of wall-clock per start;&lt;/li&gt;
&lt;li&gt;and an environment-tag fix so a laptop can never masquerade as production again (that story is below — it earned its own PR).&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;Four PRs, all merged to main; 1,363 unit tests, full CI green:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://github.com/mrviduus/textstack/pull/445" rel="noopener noreferrer"&gt;#445 — the Sentry integration&lt;/a&gt;: SDK for API + Worker, route-reason spans, agent transactions, allowlist scrubber&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/mrviduus/textstack/pull/446" rel="noopener noreferrer"&gt;#446 — the first leak it found in itself&lt;/a&gt;: EF Core SQL riding in breadcrumb &lt;em&gt;messages&lt;/em&gt;, past a green-tested scrubber&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/mrviduus/textstack/pull/447" rel="noopener noreferrer"&gt;#447 — the reader-facing race it found in me&lt;/a&gt;: &lt;code&gt;23505&lt;/code&gt; on reading-progress upserts, real users losing their place in books&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/mrviduus/textstack/pull/448" rel="noopener noreferrer"&gt;#448 — the readiness probe, the circuit breaker, and the environment-tag fix&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  My Improvements
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The router now says why.&lt;/strong&gt; Route resolution was a &lt;code&gt;??&lt;/code&gt; chain that produced a string — &lt;em&gt;identical&lt;/em&gt; whether an operator deliberately routed a task or it fell off the end onto the default. That chain doesn't just fail to record intent; it destroys it. So it returns two things now:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="k"&gt;private&lt;/span&gt; &lt;span class="n"&gt;RouteDecision&lt;/span&gt; &lt;span class="nf"&gt;ResolveRoute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="n"&gt;featureTag&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;matched&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;RegistryKey&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;featureTag&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;??&lt;/span&gt; &lt;span class="nf"&gt;ConfigRouteKey&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;featureTag&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;matched&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="n"&gt;not&lt;/span&gt; &lt;span class="k"&gt;null&lt;/span&gt;
        &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;RouteDecision&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;matched&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;RouteReason&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RouteMatched&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;RouteDecision&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s"&gt;"Ai:DefaultProvider"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;??&lt;/span&gt; &lt;span class="s"&gt;"openai"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                            &lt;span class="n"&gt;RouteReason&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DefaultFallback&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every LLM call tags its span with &lt;code&gt;ai.task&lt;/code&gt;, &lt;code&gt;ai.provider.resolved&lt;/code&gt;, and &lt;code&gt;ai.provider.reason&lt;/code&gt; = &lt;code&gt;route_matched | default_fallback&lt;/code&gt;. "Which model answered this, and did anyone choose it on purpose?" is now a trace query instead of a CPU graph.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Alert arithmetic.&lt;/strong&gt; &lt;code&gt;pdf.parse&lt;/code&gt; resolves a route once per &lt;em&gt;page&lt;/em&gt; with parallelism six — my first version would have turned the original incident into 106 identical Sentry events. Every alarm goes through a throttle keyed on &lt;code&gt;(task, provider, reason)&lt;/code&gt;: first hit fires immediately, then one event per hour per distinct problem. The unit test literally counts to 106 and asserts one claim.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;No silent fallback, ever — in either direction.&lt;/strong&gt; When the breaker finds Ollama dead, tasks are skipped and stay queued; nothing auto-switches to a paid provider, because that converts an outage into unbounded spend. Provider choice stays 100% config-driven.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;And the first live run found a hole in my own fix.&lt;/strong&gt; The startup probe opens the circuit on a one-minute backoff; the backfill worker wakes after a two-minute start delay — by then the circuit is legitimately half-open, and my single up-front gate waved the whole batch through. A per-book re-check turned 38 calls into one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Metadata backfill: enriching 38 user books
Metadata backfill: aborting after 0 enriched / 1 failed — provider 'ollama'
  is unavailable; the remaining candidates stay queued
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Tests check what you imagined; a live run checks what's there.&lt;/p&gt;

&lt;h2&gt;
  
  
  Best Use of Sentry
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Error Monitoring — what the first 24 hours in production caught:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The OpenAI account was out of credits.&lt;/strong&gt; &lt;code&gt;HTTP 429 (insufficient_quota)&lt;/code&gt; on &lt;code&gt;/translate&lt;/code&gt; and &lt;code&gt;/explain&lt;/code&gt; — the entire paid surface had been failing for readers for twelve hours. No version of my logs would have surfaced that before a user complained.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Readers were losing their place in books.&lt;/strong&gt; &lt;code&gt;PUT /me/progress&lt;/code&gt; threw &lt;code&gt;23505: duplicate key value violates unique constraint&lt;/code&gt; ten times in four hours: a textbook read-then-insert race (session heartbeat + &lt;code&gt;sendBeacon&lt;/code&gt; on unload + second device), milliseconds wide, invisible in tests. Fixed in &lt;a href="https://github.com/mrviduus/textstack/pull/447" rel="noopener noreferrer"&gt;#447&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Custom tags&lt;/strong&gt; (&lt;code&gt;ai.task&lt;/code&gt;, &lt;code&gt;ai.provider&lt;/code&gt;, &lt;code&gt;ai.failure&lt;/code&gt;, &lt;code&gt;agent.name&lt;/code&gt;, &lt;code&gt;agent.outcome&lt;/code&gt;) go through an &lt;strong&gt;allowlist scrubber&lt;/strong&gt; — every tag not explicitly blessed dies at the edge, so a future &lt;code&gt;SetTag("prompt", userText)&lt;/code&gt; can never leak. A Sentry issue answers "which feature, on which model, is broken?" without opening a trace.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tracing&lt;/strong&gt; covers agent runs and RAG indexing at 100% sampling (they're the reason I installed this), HTTP at 20%, health checks at 0%. I rejected the deprecated OTel bridge &lt;em&gt;specifically because&lt;/em&gt; spans leaving through the OpenTelemetry SDK bypass &lt;code&gt;BeforeSend&lt;/code&gt; — my OTel pipeline carries raw client IPs and full SQL text that must never leave the box. A tiny &lt;code&gt;TraceScope&lt;/code&gt; dual-writes an &lt;code&gt;Activity&lt;/code&gt; and a Sentry span instead, so everything Sentry receives passes my scrubber.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Breadcrumbs caught my scrubber lying — twice.&lt;/strong&gt; A live event's breadcrumb trail contained SQL: EF Core interpolates the query into the breadcrumb &lt;em&gt;message&lt;/em&gt;, not the structured &lt;code&gt;data&lt;/code&gt; bag my scrubber nulled (and my unit tests were green the whole time, asserting exactly the wrong thing). Fixed by dropping EF command breadcrumbs outright — then production found the &lt;em&gt;same&lt;/em&gt; leak in a second channel: EF logs a failed command at &lt;code&gt;Error&lt;/code&gt; level and Sentry's &lt;code&gt;ILogger&lt;/code&gt; integration promotes it to an event, SQL in the message again. A scrubber written against one egress path will be bypassed by the next one. Both doors are closed in &lt;a href="https://github.com/mrviduus/textstack/pull/446" rel="noopener noreferrer"&gt;#446&lt;/a&gt;, and dropping loses no signal — the exception middleware already reports the same failure with the SQLSTATE and constraint name, no SQL.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Release + environment tags as forensics.&lt;/strong&gt; The most interesting issue of the first day showed a dead Ollama starving a metadata pipeline: thirty events, tagged &lt;code&gt;environment: Production&lt;/code&gt;. I read it as an outage and started writing the fix. It was my laptop — a dev &lt;code&gt;.env&lt;/code&gt; with &lt;code&gt;ASPNETCORE_ENVIRONMENT=Production&lt;/code&gt; plus the production DSN I'd pasted in to verify the integration. What broke the spell was Sentry's own metadata: &lt;code&gt;linux-arm64&lt;/code&gt; runtime on an x86_64 prod, and a &lt;code&gt;release&lt;/code&gt; tag pointing at a commit that had never been deployed. An environment tag is a claim a process makes about itself, not a fact. Now &lt;code&gt;SENTRY_RELEASE&lt;/code&gt; comes from the &lt;code&gt;GIT_SHA&lt;/code&gt; build arg — every CI-built image has one, no &lt;code&gt;dotnet run&lt;/code&gt; ever does — and a Production claim without a release gets renamed &lt;code&gt;production-unverified&lt;/code&gt; (&lt;a href="https://github.com/mrviduus/textstack/pull/448" rel="noopener noreferrer"&gt;#448&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;And the meta-lesson that justified the whole exercise: I verified the integration by sending real events at the real DSN and &lt;em&gt;reading the captured payloads in the UI&lt;/em&gt; — that's how the leaks, the inferred-geo surprise, and the middleware capture path all surfaced. A monitoring system whose first act is to indict itself is one you can start trusting.&lt;/p&gt;

&lt;p&gt;What's the most embarrassing thing your monitoring has ever caught — and was it in the code, or in you?&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I build TextStack, an open-source reader for technical books, in .NET. The full write-up lives on &lt;a href="https://vasyl.blog/2026/08/08/nobody-alerts-on-silence-wiring-sentry-into-an-llm-pipeline/" rel="noopener noreferrer"&gt;my blog&lt;/a&gt;. &lt;a href="https://github.com/mrviduus/textstack" rel="noopener noreferrer"&gt;github.com/mrviduus/textstack&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>bugsmash</category>
      <category>dotnet</category>
      <category>sentry</category>
    </item>
    <item>
      <title>My Backup Leaked 156 GB, Filled the Disk, and Broke Itself. The Fix Was One Letter.</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 04 Aug 2026 12:00:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/my-backup-leaked-156-gb-filled-the-disk-and-broke-itself-the-fix-was-one-letter-3ldd</link>
      <guid>https://dev.to/mrviduus/my-backup-leaked-156-gb-filled-the-disk-and-broke-itself-the-fix-was-one-letter-3ldd</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for &lt;a href="https://dev.to/bugsmash"&gt;DEV's Summer Bug Smash: Smash Stories&lt;/a&gt; powered by &lt;a href="https://sentry.io/" rel="noopener noreferrer"&gt;Sentry&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;🔨 &lt;strong&gt;#bugsmash, week by week.&lt;/strong&gt; Every week I take one real production bug from &lt;a href="https://textstack.app" rel="noopener noreferrer"&gt;textstack.app&lt;/a&gt; — my public-domain e-reader side project — and write up the full detective story: symptom, wrong suspects, root cause, fix. This one is about a backup that ate its own disk. &lt;em&gt;Originally published on &lt;a href="https://vasyl.blog/2026/08/03/my-backup-leaked-156-gb-filled-the-disk-and-broke-itself-the-fix-was-one-letter/" rel="noopener noreferrer"&gt;vasyl.blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;My nightly backup has a verify step. It restores the fresh dump into a throwaway postgres container and runs sanity queries — because a backup you never restored is a hope, not a backup.&lt;/p&gt;

&lt;p&gt;One week it started flapping. &lt;code&gt;postgres did not become ready&lt;/code&gt;. That was the whole error. The dump itself succeeded. The storage tarball succeeded. Only the gate failed — the step whose entire job is to tell me the backup is real.&lt;/p&gt;

&lt;p&gt;Then one morning the box stopped answering SSH.&lt;/p&gt;

&lt;h2&gt;
  
  
  The innocent suspect
&lt;/h2&gt;

&lt;p&gt;First check, obviously: disk.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;df&lt;/span&gt; &lt;span class="nt"&gt;-h&lt;/span&gt; /
&lt;span class="go"&gt;Filesystem      Size  Used Avail Use%
/dev/sda1        96G   56G   36G  61%
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;61%. Plenty of room. Disk is innocent. Move on.&lt;/p&gt;

&lt;p&gt;Except &lt;code&gt;df -h /&lt;/code&gt; answers a narrower question than the one I asked. On this box, Docker's data-root lives on a separate partition — &lt;code&gt;/mnt/data&lt;/code&gt;. Everything Docker writes — images, containers, volumes — goes there, not to &lt;code&gt;/&lt;/code&gt;. And &lt;code&gt;/mnt/data&lt;/code&gt; looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight console"&gt;&lt;code&gt;&lt;span class="gp"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;df&lt;/span&gt; &lt;span class="nt"&gt;-h&lt;/span&gt; /mnt/data
&lt;span class="go"&gt;Filesystem      Size  Used Avail Use%
/dev/sdb1       196G  183G     0 100%
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Zero bytes free. And &lt;code&gt;docker system df&lt;/code&gt; pointed at the culprit: 156 GB of dangling anonymous volumes. Fifty-seven of them. All identical. All pgdata.&lt;/p&gt;

&lt;h2&gt;
  
  
  One volume per run, every run, for 34 days
&lt;/h2&gt;

&lt;p&gt;Here's the mechanism, and it's built from three facts that are each harmless on their own.&lt;/p&gt;

&lt;p&gt;Fact one: the official postgres image declares &lt;code&gt;VOLUME /var/lib/postgresql/data&lt;/code&gt;. If you don't mount something there yourself, Docker silently creates an &lt;em&gt;anonymous volume&lt;/em&gt; for every container you start.&lt;/p&gt;

&lt;p&gt;Fact two: &lt;code&gt;--rm&lt;/code&gt; only fires when the container exits on its own. A run that gets killed or times out never reaches that point.&lt;/p&gt;

&lt;p&gt;Fact three: my cleanup path for exactly those killed runs did &lt;code&gt;docker rm -f&lt;/code&gt;. Without &lt;code&gt;-v&lt;/code&gt;. That removes the container and leaves its anonymous volume behind — a full initialized pgdata directory, orphaned, every time.&lt;/p&gt;

&lt;p&gt;One leaked volume per bad run. Daily backups. Thirty-four days. 156 GB.&lt;/p&gt;

&lt;h2&gt;
  
  
  The loop closes
&lt;/h2&gt;

&lt;p&gt;This is the part I find beautiful, in the way you can only appreciate after the incident is over.&lt;/p&gt;

&lt;p&gt;The docker root fills up. A fresh throwaway postgres can no longer &lt;code&gt;initdb&lt;/code&gt; — nowhere to write its data dir. So the verify step fails. A failed, timed-out run is exactly the kind that skips &lt;code&gt;--rm&lt;/code&gt; and goes through the leaky cleanup. Which leaks another volume. Which leaves the disk fuller than before.&lt;/p&gt;

&lt;p&gt;The backup broke the very step that verified the backup. A self-reinforcing failure, powered entirely by its own cleanup code.&lt;/p&gt;

&lt;p&gt;One thing that mattered a lot at 11pm: real data was never at risk. Prod postgres and file storage bind-mount to &lt;code&gt;/&lt;/code&gt; — the partition sitting comfortably at 61%. The only thing bloating &lt;code&gt;/mnt/data&lt;/code&gt; was fifty-seven copies of a database that existed for ninety seconds each, just to prove a dump restores.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;One letter, in two places:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# -v removes the container's ANONYMOUS volume too. postgres declares an&lt;/span&gt;
&lt;span class="c"&gt;# anonymous VOLUME at /var/lib/postgresql/data, so every run that reaches&lt;/span&gt;
&lt;span class="c"&gt;# `docker rm -f` (a killed/timed-out run where --rm never fired) otherwise&lt;/span&gt;
&lt;span class="c"&gt;# leaks a full pgdata volume.&lt;/span&gt;
cleanup&lt;span class="o"&gt;()&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt;
  docker &lt;span class="nb"&gt;rm&lt;/span&gt; &lt;span class="nt"&gt;-fv&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$CONTAINER&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null 2&amp;gt;&amp;amp;1 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;true&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="nb"&gt;trap &lt;/span&gt;cleanup EXIT
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;docker rm -f&lt;/code&gt; became &lt;code&gt;docker rm -fv&lt;/code&gt; — in the trap handler and in the reap of leaked verify containers from previous runs. Plus a catch-all for the runs even a trap can't cover (SIGKILL):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Belt-and-suspenders: drop any dangling anonymous volumes orphaned before&lt;/span&gt;
&lt;span class="c"&gt;# this fix (or by an OOM-killed `docker rm`). Named volumes are untouched.&lt;/span&gt;
docker volume prune &lt;span class="nt"&gt;-f&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null 2&amp;gt;&amp;amp;1 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And because the original failure hid behind a bare &lt;code&gt;postgres did not become ready&lt;/code&gt;, the verify script now diagnoses itself: on readiness failure it dumps the container's &lt;code&gt;docker logs&lt;/code&gt;, &lt;code&gt;df -h&lt;/code&gt;, and &lt;code&gt;docker system df&lt;/code&gt; straight into the CI log, and bails early if the container dies during startup instead of waiting out the full window. The next time this class of bug shows up, the error message will contain its own root cause.&lt;/p&gt;

&lt;p&gt;Live remediation was one command. &lt;code&gt;docker volume prune&lt;/code&gt; reclaimed 163 GB. &lt;code&gt;/mnt/data&lt;/code&gt; went from 100% to 17%.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I keep
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;df -h /&lt;/code&gt; is not "the disk."&lt;/strong&gt; If Docker's data-root lives on its own partition, the partition you check by habit can say 61% while the one that matters says 100%.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Any image with a &lt;code&gt;VOLUME&lt;/code&gt; declaration is a leak waiting for a missing &lt;code&gt;-v&lt;/code&gt;.&lt;/strong&gt; You don't opt into anonymous volumes; they happen to you. Every &lt;code&gt;docker rm&lt;/code&gt; without &lt;code&gt;-v&lt;/code&gt; on such a container strands one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;--rm&lt;/code&gt; is happy-path cleanup.&lt;/strong&gt; The trap handler is the real cleanup — and it must be at least as thorough as &lt;code&gt;--rm&lt;/code&gt; would have been, which means it needs &lt;code&gt;-v&lt;/code&gt; too.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A verify step needs its own observability.&lt;/strong&gt; Mine guarded the backups for months and then failed with seven words and no evidence. Any gate that can fail should dump the state needed to diagnose the failure, in the failure itself.&lt;/p&gt;

&lt;p&gt;And one for the road: a full disk still lets you SSH in. When the box stopped answering entirely, that was a clean manual reboot — not the disk. An SSH &lt;em&gt;connect&lt;/em&gt; timeout is a host-offline signal. Knowing which symptom belongs to which failure saved me from chasing a second ghost that night.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>bugsmash</category>
      <category>docker</category>
      <category>devops</category>
    </item>
    <item>
      <title>Stuck at "Preparing… 0/0" Forever. My State Machine Had No Exit.</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 28 Jul 2026 12:00:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/stuck-at-preparing-00-forever-my-state-machine-had-no-exit-1pmb</link>
      <guid>https://dev.to/mrviduus/stuck-at-preparing-00-forever-my-state-machine-had-no-exit-1pmb</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for &lt;a href="https://dev.to/bugsmash"&gt;DEV's Summer Bug Smash: Clear the Lineup&lt;/a&gt; powered by &lt;a href="https://sentry.io/" rel="noopener noreferrer"&gt;Sentry&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The project
&lt;/h2&gt;

&lt;p&gt;TextStack is an open-source reader for technical books, built in .NET. It has an "Ask this book" feature: RAG over the book you uploaded. Before you can ask, the book must be indexed: split into chunks, and for PDFs each page goes through a paid vision parse that takes minutes. The code is public: &lt;a href="https://github.com/mrviduus/textstack" rel="noopener noreferrer"&gt;github.com/mrviduus/textstack&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug
&lt;/h2&gt;

&lt;p&gt;A user clicks "Ask this book". The screen shows "Preparing… 0/0". And for some books it stayed there forever. Not slow. Forever. The retry button did nothing. The only cure was me poking the database by hand.&lt;/p&gt;

&lt;p&gt;Here is what the old flow did:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;POST /me/books/{id}/index
  -&amp;gt; flip rag_status to Indexing        (in the endpoint)
  -&amp;gt; run the whole chunking inline      (minutes of vision parsing, still in the HTTP request)
  -&amp;gt; return 202 when done
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two problems living inside one line: the work ran inside the HTTP request, and the status flip happened before the work.&lt;/p&gt;

&lt;p&gt;A vision parse of a big PDF runs for many minutes. Cloudflare cuts the request long before that. The API container also restarts on every deploy. In both cases the process doing the work dies quietly, and the row stays like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rag_status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Indexing&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rag_chunk_count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And here is the real bug. The retry path only re-claimed rows in status &lt;code&gt;NotIndexed&lt;/code&gt; or &lt;code&gt;Failed&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// old claim: an Indexing row can never be claimed again&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;book&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;RagStatus&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="n"&gt;RagIndexStatus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;NotIndexed&lt;/span&gt; &lt;span class="n"&gt;or&lt;/span&gt; &lt;span class="n"&gt;RagIndexStatus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Failed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;Indexing&lt;/code&gt; with zero chunks was a state with no exit. Not failed, so no retry. Not ready, so no answers. A dead end that looks like progress. "Preparing… 0/0", forever.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix, in three pieces
&lt;/h2&gt;

&lt;p&gt;The shape of the fix: the endpoint only claims and answers, a background worker does the work, and every path ends in a terminal state. (Full code: &lt;a href="https://github.com/mrviduus/textstack/blob/main/backend/src/Worker/Services/RagIndexingService.cs" rel="noopener noreferrer"&gt;RagIndexingService.cs&lt;/a&gt; and &lt;a href="https://github.com/mrviduus/textstack/blob/main/backend/src/Worker/Services/RagIndexingWorker.cs" rel="noopener noreferrer"&gt;RagIndexingWorker.cs&lt;/a&gt;.)&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. The endpoint returns 202 immediately.&lt;/strong&gt; It flips the status and goes home. No paid work on the HTTP path, nothing for Cloudflare to kill.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. A worker claims the row atomically.&lt;/strong&gt; No locks, no queue infrastructure. One UPDATE where the WHERE clause is the whole concurrency story:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;claimed&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Database&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ExecuteSqlInterpolatedAsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"""
&lt;/span&gt;    &lt;span class="n"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;user_books&lt;/span&gt;
    &lt;span class="n"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;rag_indexing_started_at&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;bookId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="n"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;rag_status&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;              &lt;span class="p"&gt;--&lt;/span&gt; &lt;span class="n"&gt;Indexing&lt;/span&gt;
      &lt;span class="n"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;rag_chunk_count&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;
      &lt;span class="n"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;rag_indexing_started_at&lt;/span&gt; &lt;span class="n"&gt;IS&lt;/span&gt; &lt;span class="n"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="s"&gt;""", ct);
&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;claimed&lt;/span&gt; &lt;span class="p"&gt;==&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;             &lt;span class="c1"&gt;// someone else owns it, walk away&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If two workers pick the same row, one gets rowcount 1 and proceeds, the other gets 0 and returns. Postgres is the referee.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. A sweep recovers dead rows.&lt;/strong&gt; Every 30 seconds the worker also checks: any row still &lt;code&gt;Indexing&lt;/code&gt; with zero chunks whose &lt;code&gt;rag_indexing_started_at&lt;/code&gt; is older than 15 minutes means the process that claimed it died mid-parse. That row flips to a terminal &lt;code&gt;Failed&lt;/code&gt; with a human-readable reason:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;user_books&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;rag_status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rag_error&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'indexing interrupted, retry'&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;rag_status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;rag_chunk_count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;rag_indexing_started_at&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;staleCutoff&lt;/span&gt;&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Note what it does NOT do: auto-requeue. A vision parse is real paid spend. The row fails loudly, the user re-triggers deliberately. Retry is a human decision when every retry costs money.&lt;/p&gt;

&lt;h2&gt;
  
  
  The detail that bit me anyway
&lt;/h2&gt;

&lt;p&gt;The failure path had its own bug waiting. When indexing dies because the worker is shutting down, the cancellation token is already cancelled. If you write the &lt;code&gt;Failed&lt;/code&gt; status using that same token, the error write gets cancelled too, and you are back to the forever-Indexing dead end you just fixed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// the terminal write runs on a FRESH context with CancellationToken.None:&lt;/span&gt;
&lt;span class="c1"&gt;// the token that killed the work must not kill the record of its death&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;var&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;dbFactory&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;CreateDbContextAsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;CancellationToken&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;None&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The same cancellation also produces a second trap: the chunker returns 0 chunks on shutdown, which looks identical to a genuinely empty book. Without checking &lt;code&gt;ct.IsCancellationRequested&lt;/code&gt;, a perfectly good book gets labeled "No chapters to index" and a user believes their upload is broken. Zero is the least trustworthy number in a distributed system.&lt;/p&gt;

&lt;h2&gt;
  
  
  Before and after
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Before&lt;/th&gt;
&lt;th&gt;After&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Work runs&lt;/td&gt;
&lt;td&gt;inside the HTTP request&lt;/td&gt;
&lt;td&gt;in a background worker&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Request dies mid-parse&lt;/td&gt;
&lt;td&gt;row stuck Indexing forever&lt;/td&gt;
&lt;td&gt;sweep flips it to Failed in 15 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Retry button&lt;/td&gt;
&lt;td&gt;dead for stuck rows&lt;/td&gt;
&lt;td&gt;works, Failed is claimable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Shutdown during parse&lt;/td&gt;
&lt;td&gt;"No chapters to index" (wrong)&lt;/td&gt;
&lt;td&gt;"indexing interrupted, retry"&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Concurrency control&lt;/td&gt;
&lt;td&gt;none needed (and none possible)&lt;/td&gt;
&lt;td&gt;one atomic UPDATE, rowcount as the lock&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  What I took away
&lt;/h2&gt;

&lt;p&gt;A status enum is not a state machine until every state has an exit. &lt;code&gt;Indexing&lt;/code&gt; was a real status, rendered nicely in the UI, and it was also a trap: no transition out except success. The fix was not clever code. It was drawing the diagram and asking one question about every state: how does a row leave here if the process dies right now?&lt;/p&gt;

&lt;p&gt;And the operational rule on top: never let an HTTP request own work that outlives an HTTP request. The request is a messenger, not a worker.&lt;/p&gt;

&lt;p&gt;What is the longest a "temporary" status has survived in your database? I found rows that had been "Preparing" for weeks.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I build TextStack, an open-source reader for technical books, in .NET. This fix is from the indexing pipeline behind its "Ask this book" feature. &lt;a href="https://github.com/mrviduus/textstack" rel="noopener noreferrer"&gt;github.com/mrviduus/textstack&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>bugsmash</category>
      <category>dotnet</category>
      <category>postgres</category>
    </item>
    <item>
      <title>Nothing Crashed. Nothing Errored. My CPU Sat at 390% for an Hour.</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 21 Jul 2026 12:00:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/nothing-crashed-nothing-errored-my-cpu-sat-at-390-for-an-hour-335a</link>
      <guid>https://dev.to/mrviduus/nothing-crashed-nothing-errored-my-cpu-sat-at-390-for-an-hour-335a</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for &lt;a href="https://dev.to/bugsmash"&gt;DEV's Summer Bug Smash: Smash Stories&lt;/a&gt; powered by &lt;a href="https://sentry.io/" rel="noopener noreferrer"&gt;Sentry&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A user uploaded a 106-page PDF and clicked "Ask this book".&lt;/p&gt;

&lt;p&gt;The screen said "Preparing… 0/0".&lt;/p&gt;

&lt;p&gt;It said that for an hour.&lt;/p&gt;

&lt;p&gt;No errors. No failed requests. No exceptions in the logs. Just one graph I still think about: a CPU-only container, pinned at 390%. Almost exactly four cores, flat, like a ruler.&lt;/p&gt;

&lt;p&gt;The strange part: I pay for GPT-4.1 exactly so this kind of work never runs on my CPU. And the OpenAI dashboard showed zero traffic. Something was doing the work. Just not the thing I configured.&lt;/p&gt;

&lt;h2&gt;
  
  
  What was supposed to happen
&lt;/h2&gt;

&lt;p&gt;TextStack is a reading app with an "Ask this book" feature. It is RAG over the book you uploaded. For table-heavy PDFs, plain text extraction is useless, so each page goes through vision parsing: page image in, Markdown out. That job belongs to GPT-4.1.&lt;/p&gt;

&lt;p&gt;The provider is picked by a routing config. Simplified:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"Ai"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"DefaultProvider"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"ollama"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"Routes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
    &lt;/span&gt;&lt;span class="nl"&gt;"pdf.parse"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"openai-pdf"&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If a task has a route, it goes to that provider. If it doesn't, it falls back to the default. The default is a local Ollama, which I use for small cheap tasks.&lt;/p&gt;

&lt;p&gt;Keep that fallback in mind. It is the villain of this story.&lt;/p&gt;

&lt;h2&gt;
  
  
  Debugging by absence
&lt;/h2&gt;

&lt;p&gt;Most debugging starts with an error. This bug gave me none. The strongest clue was a thing that was missing.&lt;/p&gt;

&lt;p&gt;Here is what I could see:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The indexing job was running. Not failed, not stuck in a queue. Running.&lt;/li&gt;
&lt;li&gt;Zero requests to OpenAI. Not slow requests, not errors. Zero.&lt;/li&gt;
&lt;li&gt;The Ollama container was burning four cores. On a box with no GPU.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So the vision parsing was happening. On the local model. On CPU. I measured it later: about 42 seconds per page. The book had 106 pages. That is roughly 74 minutes of parsing for one book, all of it awaited inline by the indexing worker. The whole RAG pipeline sat behind it. From the outside it looked frozen at "Preparing… 0/0". From the inside it was working very hard on the wrong hardware.&lt;/p&gt;

&lt;p&gt;Nothing failed. That was the problem. A crash would have paged me in a minute. A fallback that "works" can burn for an hour.&lt;/p&gt;

&lt;h2&gt;
  
  
  The root cause hurt a little
&lt;/h2&gt;

&lt;p&gt;A few days earlier I shipped a reliability fix. Indexing used to run chunking inside the HTTP request, and it could get permanently stuck if the request died. So I moved the whole thing into a background Worker. Good change. I would do it again.&lt;/p&gt;

&lt;p&gt;But the API and the Worker are separate processes, each with its own &lt;code&gt;appsettings.json&lt;/code&gt;. The API config had the route:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"Routes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"pdf.parse"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"openai-pdf"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The Worker config did not.&lt;/p&gt;

&lt;p&gt;So when the parsing moved from the API into the Worker, the router looked up &lt;code&gt;pdf.parse&lt;/code&gt;, found nothing, and did what fallbacks do. It quietly handed a vision parsing job for a 106-page book to a local model on a CPU-only container.&lt;/p&gt;

&lt;p&gt;The work moved. The config did not. My reliability fix caused the regression.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;One config block in the Worker's &lt;code&gt;appsettings.json&lt;/code&gt;, mirroring the API:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="nl"&gt;"Routes"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"pdf.parse"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"openai-pdf"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;plus the matching &lt;code&gt;Ai:Pdf&lt;/code&gt; settings block. No code change. The container dropped from 390% to idle, OpenAI traffic came back, the book indexed in minutes.&lt;/p&gt;

&lt;p&gt;Five minutes to write. Much longer to find.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this bug hid so well
&lt;/h2&gt;

&lt;p&gt;Three reasons, and I think they generalize:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A fallback doesn't fail. It succeeds with the wrong tool.&lt;/strong&gt; Every health check was green. The job was making progress. Slowly, expensively, but progress. There was no moment where the system could say "this is wrong", because by its own rules nothing was.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Nobody alerts on silence.&lt;/strong&gt; I had alerts for errors and for high latency on my API. I had nothing for "an expensive external provider we depend on received zero calls today". Absence of a signal is the hardest thing to notice, because there is nothing to point at.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Config lives per process, but my mental model didn't.&lt;/strong&gt; In my head there was one routing table. In reality there were two files, and they agreed only by accident, until they didn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed in how I work
&lt;/h2&gt;

&lt;p&gt;Three rules I took from this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;When work moves between processes, config moves with it.&lt;/strong&gt; That is now a checklist item on any "move X to the Worker" change: diff the relevant config sections between the two processes before shipping.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make routing loud.&lt;/strong&gt; One log line per job that says which provider was resolved and why: &lt;code&gt;pdf.parse -&amp;gt; ollama (no route, DefaultProvider fallback)&lt;/code&gt;. That line would have turned an hour of confusion into a 30-second fix.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;For expensive work, prefer fail-fast over fallback.&lt;/strong&gt; A missing route for a cheap task can fall back. A missing route for vision parsing of a whole book should throw. A silent fallback is a decision the system makes without telling you. If the choice of provider matters, it should not be made silently.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The uncomfortable lesson on top: the bug was introduced by a fix I was proud of. Reliability work changes where code runs, and "where" is exactly what config is about.&lt;/p&gt;

&lt;p&gt;Where in your system could a fallback take over silently, and how long would it take you to notice? Mine took a 106-page PDF and a CPU graph. I would love to hear yours in the comments.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I build TextStack, an open-source reader for technical books, in .NET. This bug is from the RAG indexing pipeline behind its "Ask this book" feature. &lt;a href="https://github.com/mrviduus/textstack" rel="noopener noreferrer"&gt;github.com/mrviduus/textstack&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>bugsmash</category>
      <category>dotnet</category>
      <category>ai</category>
    </item>
    <item>
      <title>Your RAG Eval Isn't Flaky. Your Retrieval Is Non-Deterministic.</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 14 Jul 2026 14:48:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/your-rag-eval-isnt-flaky-your-retrieval-is-non-deterministic-42ab</link>
      <guid>https://dev.to/mrviduus/your-rag-eval-isnt-flaky-your-retrieval-is-non-deterministic-42ab</guid>
      <description>&lt;p&gt;Same query.&lt;br&gt;
Same documents.&lt;br&gt;
Same model.&lt;br&gt;
And the RAG eval can still hand back a different Recall@8.&lt;/p&gt;

&lt;p&gt;Not because the model is flaky. Because of an &lt;code&gt;ORDER BY&lt;/code&gt; clause.&lt;/p&gt;

&lt;p&gt;I didn't find this by watching a metric wobble. I found it reading the retrieval code, and realized the score would drift run to run even if the model never changed.&lt;/p&gt;

&lt;p&gt;This came out of a habit I've adopted recently: I write the eval before the feature. Reviewing the retrieval pipeline behind my "Ask this Book" feature, I saw it: the retrieval layer wasn't deterministic.&lt;/p&gt;
&lt;h2&gt;
  
  
  Order isn't presentation. It's part of the input.
&lt;/h2&gt;

&lt;p&gt;My RAG implementation is intentionally simple: plain PostgreSQL and .NET. Two retrieval strategies over the same table:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;semantic search using pgvector&lt;/li&gt;
&lt;li&gt;lexical search using PostgreSQL full-text search&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The results are merged with Reciprocal Rank Fusion (RRF).&lt;/p&gt;

&lt;p&gt;Here's the important part: &lt;strong&gt;RRF doesn't care about the retrieval scores. It only cares about rank.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If one retriever returns&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;A
B
C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;instead of&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;B
A
C
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;RRF produces different fused scores. Different fused scores mean a different Top-K. Different Top-K means different Recall@K.&lt;/p&gt;

&lt;p&gt;In RRF, order isn't a display detail. Order is data.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug
&lt;/h2&gt;

&lt;p&gt;My lexical query ended like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Looks perfectly reasonable. Except &lt;code&gt;ts_rank_cd&lt;/code&gt; produces ties surprisingly often. Multiple chunks can have exactly the same score.&lt;/p&gt;

&lt;p&gt;And SQL only guarantees the ordering you explicitly request. If multiple rows compare equal, PostgreSQL is free to return them in any order.&lt;/p&gt;

&lt;p&gt;Nothing changed. Same database. Same query. Same model. Only the order of equally-ranked rows. Yet that's enough for RRF to assign different ranks, producing different fused scores and a different evaluation result.&lt;/p&gt;

&lt;p&gt;The semantic retrieval had the same issue. Distance ties are much rarer than lexical ties, but "rare" isn't good enough for an evaluation pipeline.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;The fix was almost embarrassingly small.&lt;/p&gt;

&lt;p&gt;Before:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A deterministic tie-breaker on both retrieval queries. Now equal-scoring rows always appear in the same order, RRF receives the same input every run, and the Top-K stays identical.&lt;/p&gt;

&lt;p&gt;Notice what didn't happen. The retrieval didn't become better. &lt;strong&gt;It became reproducible.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this matters
&lt;/h2&gt;

&lt;p&gt;We spend a lot of effort making the model deterministic during evaluation: temperature 0, fixed datasets, golden answers, reproducible prompts.&lt;/p&gt;

&lt;p&gt;But it's easy to assume everything underneath the model is already deterministic. Often it isn't. Retrieval. Ranking. Sampling. Data loading. Any non-deterministic stage in the pipeline can quietly invalidate your eval.&lt;/p&gt;

&lt;p&gt;A fluctuating eval isn't just annoying. It's dangerous. Eventually you stop trusting the number, even when it's pointing at a real problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  The lesson I took away
&lt;/h2&gt;

&lt;p&gt;Before debugging the model, debug determinism. An evaluation can only be as deterministic as the pipeline feeding it. Same query. Same rows. Same order. Only then can you trust what your eval is telling you.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I build &lt;a href="https://textstack.app" rel="noopener noreferrer"&gt;TextStack&lt;/a&gt;, an open-source reader for technical books, in .NET. This is from the retrieval layer behind its "Ask this Book" feature. Code on &lt;a href="https://github.com/mrviduus/textstack" rel="noopener noreferrer"&gt;github.com/mrviduus/textstack&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>dotnet</category>
      <category>ai</category>
      <category>rag</category>
      <category>postgres</category>
    </item>
    <item>
      <title>AI Wrote a Thread-Safe Counter. The CPU Made It 5x Slower.</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 07 Jul 2026 13:00:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/ai-wrote-a-thread-safe-counter-the-cpu-made-it-5x-slower-45n6</link>
      <guid>https://dev.to/mrviduus/ai-wrote-a-thread-safe-counter-the-cpu-made-it-5x-slower-45n6</guid>
      <description>&lt;p&gt;&lt;em&gt;Meet the cache line.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I asked an AI assistant for a simple thing: &lt;strong&gt;per-thread counters.&lt;/strong&gt; Four threads, each incrementing its own slot in an array. No shared variables. No locks needed. The code it wrote was clean and correct:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="n"&gt;counters&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="c1"&gt;// thread t does: counters[t]++&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every thread writes only to its own element. There is no race here. Any code review would pass it. Every test would pass too.&lt;/p&gt;

&lt;p&gt;Then I measured it against a version that does &lt;em&gt;exactly the same work&lt;/em&gt;, and the correct code lost by 5x.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers
&lt;/h2&gt;

&lt;p&gt;This is from my machine (Apple Silicon, .NET 10, 4 threads, 200 million increments per thread):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Round 1:  adjacent (one cache line):  464 ms   padded (line per thread):  85 ms   ratio: 5.4x
Round 2:  adjacent (one cache line):  397 ms   padded (line per thread):  85 ms   ratio: 4.7x
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same loop. Same number of increments. Same "each thread touches only its own counter". The only difference between the two versions is &lt;em&gt;where the counters live in memory&lt;/em&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the hardware actually does
&lt;/h2&gt;

&lt;p&gt;A CPU never reads one byte from memory. It moves data in fixed blocks called &lt;strong&gt;cache lines&lt;/strong&gt; — 64 bytes on x86, 128 bytes on Apple Silicon. Ask for one &lt;code&gt;long&lt;/code&gt; and the whole block it lives in travels into the core's cache. Think of a cook whose ingredients are in a basement fridge: going downstairs is expensive, so you never carry one carrot — you carry the whole crate.&lt;/p&gt;

&lt;p&gt;(Don't take my word for the 128: run &lt;code&gt;sysctl hw.cachelinesize&lt;/code&gt; on an M-series Mac.)&lt;/p&gt;

&lt;p&gt;Usually this works for you. Array elements sit side by side, so scanning an array is fast: you touch one element and the next fifteen arrive in the same crate for free.&lt;/p&gt;

&lt;p&gt;But with multiple cores there is a rule: &lt;strong&gt;to write into a cache line, a core must own it exclusively.&lt;/strong&gt; The moment core 1 writes, every other core's copy of that line is declared stale. And the unit of ownership is not your variable. It is the whole line.&lt;/p&gt;

&lt;p&gt;Now look at my four counters. Four &lt;code&gt;long&lt;/code&gt; values, 8 bytes each, side by side — 32 bytes. They all fit in &lt;em&gt;one&lt;/em&gt; cache line. Thread 1 increments its counter and takes ownership of the line. A nanosecond later thread 2 increments &lt;em&gt;its own, different&lt;/em&gt; counter — and has to rip the same line back. The line ping-pongs between cores on every single write. Four threads that share nothing in the code are fighting over one crate in the hardware.&lt;/p&gt;

&lt;p&gt;This is called &lt;strong&gt;false sharing&lt;/strong&gt;. False, because no data is actually shared. The fight is real anyway.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcvrlihqsni4dj3ke8x0t.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcvrlihqsni4dj3ke8x0t.png" alt="Four threads writing to four counters in one cache line ping-pong on every write (464 ms), versus padded layout with one counter per cache line (85 ms)" width="800" height="533"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Four counters in one cache line vs. one counter per line — same code, 5x difference.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  The demo fix is one attribute
&lt;/h2&gt;

&lt;p&gt;Give every thread its own cache line. Pad each counter so the next one starts in a different line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;StructLayout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;LayoutKind&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Explicit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Size&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;128&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="nc"&gt;PaddedCounter&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;FieldOffset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;64&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt; &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the entire difference between 464 ms and 85 ms. We pay a little memory — 128 bytes per counter instead of 8 — and get back the parallelism we thought we already had. (I use 128, not 64, for two reasons: Apple Silicon lines are 128 bytes, and on x86 the adjacent-line prefetcher likes to drag neighboring lines along.)&lt;/p&gt;

&lt;h2&gt;
  
  
  Run it yourself
&lt;/h2&gt;

&lt;p&gt;The full demo is about 70 lines, no project file needed — with the .NET 10 SDK you can run a single file directly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;dotnet run &lt;span class="nt"&gt;-c&lt;/span&gt; Release FalseSharingDemo.cs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// FalseSharingDemo.cs — .NET 10 file-based app&lt;/span&gt;
&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;System.Diagnostics&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="nn"&gt;System.Runtime.InteropServices&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;const&lt;/span&gt; &lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="n"&gt;Iterations&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;200_000_000&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;threadCount&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Environment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ProcessorCount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;round&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;round&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;round&lt;/span&gt;&lt;span class="p"&gt;++)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;slow&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;MeasureAdjacent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;fast&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;MeasurePadded&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="n"&gt;Console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;WriteLine&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;$"adjacent: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;slow&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TotalMilliseconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;F0&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt; ms   padded: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;fast&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;TotalMilliseconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;F0&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt; ms   ratio: &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;slow&lt;/span&gt; &lt;span class="p"&gt;/&lt;/span&gt; &lt;span class="n"&gt;fast&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;F1&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s"&gt;x"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="n"&gt;TimeSpan&lt;/span&gt; &lt;span class="nf"&gt;MeasureAdjacent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="n"&gt;counters&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;RunThreads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;Iterations&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;++)&lt;/span&gt;
            &lt;span class="n"&gt;counters&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;]++;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;counters&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
    &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="n"&gt;TimeSpan&lt;/span&gt; &lt;span class="nf"&gt;MeasurePadded&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;counters&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;PaddedCounter&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;RunThreads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;Iterations&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;++)&lt;/span&gt;
            &lt;span class="n"&gt;counters&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;++;&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;counters&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;static&lt;/span&gt; &lt;span class="n"&gt;TimeSpan&lt;/span&gt; &lt;span class="nf"&gt;RunThreads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Func&lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;long&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="n"&gt;sink&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;threads&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="n"&gt;Thread&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
    &lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;sw&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;Stopwatch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;StartNew&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;threadCount&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;++)&lt;/span&gt;
    &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="kt"&gt;int&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="n"&gt;threads&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nf"&gt;Thread&lt;/span&gt;&lt;span class="p"&gt;(()&lt;/span&gt; &lt;span class="p"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Interlocked&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Add&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;ref&lt;/span&gt; &lt;span class="n"&gt;sink&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;body&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)));&lt;/span&gt;
        &lt;span class="n"&gt;threads&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;Start&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;foreach&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;var&lt;/span&gt; &lt;span class="n"&gt;th&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;threads&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;th&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Join&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="n"&gt;sw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;Stop&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
    &lt;span class="n"&gt;GC&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;KeepAlive&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sink&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;sw&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Elapsed&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;StructLayout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;LayoutKind&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Explicit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Size&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="m"&gt;128&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;span class="k"&gt;struct&lt;/span&gt; &lt;span class="nc"&gt;PaddedCounter&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nf"&gt;FieldOffset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="m"&gt;64&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt; &lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="kt"&gt;long&lt;/span&gt; &lt;span class="n"&gt;Value&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Where this hides in real code
&lt;/h2&gt;

&lt;p&gt;You will not write four counters in a loop at work. But you will write, or an AI will write for you:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Cache statistics.&lt;/strong&gt; Almost every in-memory cache keeps hits, misses, evictions. The natural implementation is fields next to each other, or a &lt;code&gt;long[]&lt;/code&gt; with one slot per shard, updated with &lt;code&gt;Interlocked.Increment&lt;/code&gt; from every thread. Fields next to each other means one cache line. This is literally the demo above, running in your production service.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Sharded counters.&lt;/strong&gt; The cruel version: you sharded a counter &lt;em&gt;specifically to make it parallel&lt;/em&gt;, put the shards in one array — and they still share lines. You did the architecture work and the hardware quietly undid it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;LRU metadata.&lt;/strong&gt; A compact &lt;code&gt;long[] lastAccessTicks&lt;/code&gt; per cache slot means every cache &lt;em&gt;read&lt;/em&gt; becomes a &lt;em&gt;write&lt;/em&gt; into a hot shared array. A read-heavy cache that is slow because of writes is a fun thing to debug.&lt;/p&gt;

&lt;p&gt;And sometimes the problem ships inside the library. &lt;code&gt;ConcurrentDictionary&lt;/code&gt; — the base of most homemade .NET caches — internally keeps a counter per lock stripe in a plain array (&lt;code&gt;_countPerLock&lt;/code&gt; in the source). Under very hot multi-threaded writes those neighbors &lt;em&gt;can&lt;/em&gt; end up sharing lines. I have not benchmarked the real-world impact — but the layout is right there in the source, and now you know what to look for.&lt;/p&gt;

&lt;h2&gt;
  
  
  When you should NOT care
&lt;/h2&gt;

&lt;p&gt;Honesty section. False sharing hurts when the writes are &lt;em&gt;hot&lt;/em&gt; — millions of updates per second from several threads. If your cache updates its stats a thousand times per second, you will never notice, and padding everything "just in case" is cargo cult. The rule is the same as always: measure first. The demo above is the measurement; adapt it to your data layout and see if your ratio is 1.0x or 5x.&lt;/p&gt;

&lt;h2&gt;
  
  
  The actual point
&lt;/h2&gt;

&lt;p&gt;The AI-generated code was not wrong. It compiled, it was race-free, it passed every test I could write for its correctness. An entire code review process could bless it. The 5x was invisible at every layer we normally check.&lt;/p&gt;

&lt;p&gt;That is what changed with AI-assisted coding, and it is why hardware fundamentals became more valuable, not less. The model will happily generate a thread-safe counter, a sharded cache, an LRU eviction policy — and none of its correctness guarantees say anything about cache lines. Correctness and mechanical sympathy are different layers. Tests catch the first. Only understanding catches the second.&lt;/p&gt;

&lt;p&gt;You do not need to memorize cache sizes. You need to know the crate exists. Keep data that is used together close. Keep data that is &lt;em&gt;written by different threads&lt;/em&gt; apart. That one rule, read in both directions, is most of "cache-aware" programming.&lt;/p&gt;

&lt;p&gt;The tools write the code now. Knowing why it is slow is still our job.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://vasyl.blog/2026/07/03/false-sharing-meet-the-cache-line/" rel="noopener noreferrer"&gt;vasyl.blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>dotnet</category>
      <category>csharp</category>
      <category>performance</category>
    </item>
    <item>
      <title>An AI Feature Has No "Tests Pass" Moment. So I Write the Eval First.</title>
      <dc:creator>Vasyl</dc:creator>
      <pubDate>Tue, 23 Jun 2026 12:00:00 +0000</pubDate>
      <link>https://dev.to/mrviduus/an-ai-feature-has-no-tests-pass-moment-so-i-write-the-eval-first-1f7p</link>
      <guid>https://dev.to/mrviduus/an-ai-feature-has-no-tests-pass-moment-so-i-write-the-eval-first-1f7p</guid>
      <description>&lt;p&gt;I was building an "Ask This Book" feature: readers can ask questions about a book while they're reading it.&lt;/p&gt;

&lt;p&gt;One requirement sounded simple:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A reader on chapter 3 must never receive spoilers from chapter 30.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;My first instinct was the same as everyone else's: tell the model not to spoil future chapters. Something like:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Please don't reveal information from chapters the reader hasn't reached yet."&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;And honestly, it mostly worked.&lt;/p&gt;

&lt;p&gt;The problem is that "mostly" is useless. A user only needs one spoiler.&lt;/p&gt;

&lt;p&gt;That was the moment I realized the feature had no definition of done.&lt;/p&gt;

&lt;p&gt;With normal software, something pushes back. The compiler complains. The tests fail. The types don't line up.&lt;/p&gt;

&lt;p&gt;With an LLM feature, none of that happens. The output looks plausible by default — fluent, confident, well formatted — even when it's wrong.&lt;/p&gt;

&lt;p&gt;So "it looked right in the demo" quietly becomes the finish line.&lt;/p&gt;

&lt;p&gt;That's exactly why I write the eval before I write the feature.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Eval Is the Specification
&lt;/h2&gt;

&lt;p&gt;Most teams treat evals as QA. Build the feature, ship something that works, add evals later.&lt;/p&gt;

&lt;p&gt;I increasingly think that's backwards. For AI systems, the eval is often the only concrete definition of success.&lt;/p&gt;

&lt;p&gt;The moment I wrote the spoiler eval, I had to define failure: spoiler leakage must be zero. Not low. Not acceptable. Zero.&lt;/p&gt;

&lt;p&gt;And that requirement immediately exposed a problem. No prompt can guarantee zero.&lt;/p&gt;

&lt;p&gt;Prompts are probabilistic. Users can phrase questions differently. Models can interpret instructions differently. Future model updates can behave differently. You cannot get a hard guarantee from a soft instruction.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Eval Changed the Architecture
&lt;/h2&gt;

&lt;p&gt;Once the eval demanded zero spoilers, the solution stopped being a prompt problem. It became a retrieval problem.&lt;/p&gt;

&lt;p&gt;Instead of telling the model not to reveal future chapters, I prevented future chapters from entering the context at all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;chapter_ord&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="o"&gt;@&lt;/span&gt;&lt;span class="n"&gt;maxChapterOrd&lt;/span&gt;&lt;span class="nv"&gt;`&lt;/span&gt;&lt;span class="se"&gt;``&lt;/span&gt;&lt;span class="nv"&gt;

Anything beyond the reader's progress never enters the retrieval set. The model can't leak information it never saw.

And the eval that checks it is just as blunt — a retrieved chunk past the reader's progress is a leak:



&lt;/span&gt;&lt;span class="se"&gt;``&lt;/span&gt;&lt;span class="nv"&gt;`&lt;/span&gt;&lt;span class="n"&gt;csharp&lt;/span&gt;
&lt;span class="o"&gt;//&lt;/span&gt; &lt;span class="n"&gt;One&lt;/span&gt; &lt;span class="n"&gt;retrieved&lt;/span&gt; &lt;span class="n"&gt;chunk&lt;/span&gt; &lt;span class="n"&gt;past&lt;/span&gt; &lt;span class="n"&gt;the&lt;/span&gt; &lt;span class="n"&gt;reader&lt;/span&gt;&lt;span class="s1"&gt;'s progress = one spoiler leak.
public static int LeakCount(IEnumerable&amp;lt;RetrievedChunk&amp;gt; retrieved, int gateChapterOrd) =&amp;gt;
    retrieved.Count(c =&amp;gt; c.ChapterOrd &amp;gt; gateChapterOrd);
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Across the adversarial test cases, that number has to be zero. That's the moment the idea really clicked for me: the eval didn't test the design. It produced the design.&lt;/p&gt;

&lt;p&gt;A measurable failure condition forced a better architecture than I would have built if I had started with prompt engineering.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Same Thing Happened to Retrieval Quality
&lt;/h2&gt;

&lt;p&gt;The spoiler requirement wasn't the only eval. I also defined two other targets before building the feature:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Retrieval must surface the correct passage near the top of the results.&lt;/li&gt;
&lt;li&gt;Answers must remain grounded in the passages they cite.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Because those requirements were measurable, every change received a verdict instead of an opinion.&lt;/p&gt;

&lt;p&gt;A single semantic search wasn't clearing the bar. So I ended up combining two retrieval approaches:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;vector search for semantic similarity&lt;/li&gt;
&lt;li&gt;full-text search for exact names, phrases, and quotations&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The results are fused using Reciprocal Rank Fusion — less mysterious than it sounds. Each chunk scores the sum of 1/(k+rank) across the lists it appears in, so anything ranked highly by both retrievers floats to the top:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="c1"&gt;// ranked highly by both vector AND lexical -&amp;gt; floats to the top.&lt;/span&gt;
&lt;span class="n"&gt;scores&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;item&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="p"&gt;+=&lt;/span&gt; &lt;span class="m"&gt;1.0&lt;/span&gt; &lt;span class="p"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt; &lt;span class="p"&gt;+&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="p"&gt;+&lt;/span&gt; &lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// i is 0-based; RRF rank is 1-based&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I didn't choose hybrid retrieval because it's fashionable. I chose it because it moved the number. The eval said the system wasn't good enough. The architecture changed until it was.&lt;/p&gt;

&lt;h2&gt;
  
  
  A Note on the Stack
&lt;/h2&gt;

&lt;p&gt;None of this is a no-dependencies flex. The judge that scores grounding is a custom evaluator on Microsoft.Extensions.AI.Evaluation:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight csharp"&gt;&lt;code&gt;&lt;span class="k"&gt;public&lt;/span&gt; &lt;span class="k"&gt;sealed&lt;/span&gt; &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;RubricEvaluator&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Rubric&lt;/span&gt; &lt;span class="n"&gt;rubric&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;IEvaluator&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I lean on the Microsoft stack on purpose. What I keep hand-rolled is the part that decides quality — the retrieval, the fusion, the spoiler gate. The line I draw isn't "no libraries." It's no agent framework hiding the parts that determine whether the thing actually works.&lt;/p&gt;

&lt;h2&gt;
  
  
  Eval-First Development
&lt;/h2&gt;

&lt;p&gt;Traditional software development gives us confidence almost for free. Compilers. Type systems. Unit tests. Integration tests.&lt;/p&gt;

&lt;p&gt;AI systems don't. The difficult part isn't implementing the feature. The difficult part is defining what "correct" means.&lt;/p&gt;

&lt;p&gt;That's why I increasingly think of eval-first development as the AI equivalent of TDD. With traditional software, tests verify the implementation. With AI systems, evals often define the implementation.&lt;/p&gt;

&lt;p&gt;Build the feature first and the eval later, and the eval can only grade what you've already built. Build the eval first and it starts shaping the system itself.&lt;/p&gt;

&lt;p&gt;It defines done. It tells you when you've regressed. And sometimes it forces a better architecture than the one you originally had in mind.&lt;/p&gt;

&lt;p&gt;Otherwise you're not shipping a feature. You're shipping a guess that happened to demo well.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Want to go deeper on evals? I've written a separate, more hands-on series on building production AI on .NET: &lt;a href="https://vasyl.blog/2026/06/10/what-are-ai-evals/" rel="noopener noreferrer"&gt;what evals actually are&lt;/a&gt;, &lt;a href="https://vasyl.blog/2026/06/10/error-analysis-for-evals/" rel="noopener noreferrer"&gt;error analysis&lt;/a&gt;, &lt;a href="https://vasyl.blog/2026/06/10/golden-datasets-that-dont-lie/" rel="noopener noreferrer"&gt;golden datasets&lt;/a&gt;, &lt;a href="https://vasyl.blog/2026/06/10/llm-as-judge-done-right/" rel="noopener noreferrer"&gt;LLM-as-judge&lt;/a&gt;, and &lt;a href="https://vasyl.blog/2026/06/10/evals-in-ci-and-production/" rel="noopener noreferrer"&gt;evals in CI and production&lt;/a&gt;. This post was originally published on &lt;a href="https://vasyl.blog/2026/06/17/evals-before-rag/" rel="noopener noreferrer"&gt;my blog&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>dotnet</category>
      <category>csharp</category>
      <category>machinelearning</category>
    </item>
  </channel>
</rss>
