<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Priyanshu Jha</title>
    <description>The latest articles on DEV Community by Priyanshu Jha (@priyanshu_jha_eee132b17e5).</description>
    <link>https://dev.to/priyanshu_jha_eee132b17e5</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4156860%2F6b514e8f-ed6f-4828-b48f-5a528a0471dc.png</url>
      <title>DEV Community: Priyanshu Jha</title>
      <link>https://dev.to/priyanshu_jha_eee132b17e5</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/priyanshu_jha_eee132b17e5"/>
    <language>en</language>
    <item>
      <title>When stdout is nil: stopping a WinRM panic in GitLab Fleeting</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Fri, 13 Nov 2026 18:16:16 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/when-stdout-is-nil-stopping-a-winrm-panic-in-gitlab-fleeting-3o5l</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/when-stdout-is-nil-stopping-a-winrm-panic-in-gitlab-fleeting-3o5l</guid>
      <description>&lt;p&gt;A nil writer is one of those values that looks like "I don't care about this output" and behaves like "please crash." In GitLab Fleeting's WinRM connector, that's exactly what happened. The library always copied command output to stdout. When stdout was nil, the process panicked.&lt;/p&gt;

&lt;p&gt;This is the story of &lt;a href="https://gitlab.com/gitlab-org/fleeting/fleeting/-/merge_requests/151" rel="noopener noreferrer"&gt;gitlab-org/fleeting/fleeting!151&lt;/a&gt;, my fix for that panic, opened during the October 2026 Hackathon. I'll cover the issue, the test that reproduced it before the fix existed, the one-guard change, and the fake WinRM transport that made the tests possible.&lt;/p&gt;

&lt;h2&gt;
  
  
  The issue: a panic on a path that meant "no stdout"
&lt;/h2&gt;

&lt;p&gt;The issue was &lt;a href="https://gitlab.com/gitlab-org/fleeting/fleeting/-/work_items/66" rel="noopener noreferrer"&gt;gitlab-org/fleeting/fleeting#66&lt;/a&gt;, opened by Brandon Bruce. It came from GitLab Runner's &lt;code&gt;instance_ready_command&lt;/code&gt; path for Windows autoscaling (support case 759678).&lt;/p&gt;

&lt;p&gt;Fleeting is the library GitLab Runner uses to provision and connect to autoscaled instances. The WinRM connector is how it talks to Windows. &lt;code&gt;winRMClient.Run&lt;/code&gt; took &lt;code&gt;opts.Stdout&lt;/code&gt; and handed it to the winrm library. The library copies command output to that writer unconditionally. When &lt;code&gt;Stdout&lt;/code&gt; was nil, the copy was a nil pointer dereference.&lt;/p&gt;

&lt;p&gt;Nil &lt;code&gt;Stderr&lt;/code&gt; was already handled. Nil &lt;code&gt;Stdout&lt;/code&gt; was not. An earlier MR, &lt;a href="https://gitlab.com/gitlab-org/fleeting/fleeting/-/merge_requests/117" rel="noopener noreferrer"&gt;!117&lt;/a&gt;, had tightened stderr-writer handling and concurrent writes, and had left this hole.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe1x085tcujmn5a3hewe6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe1x085tcujmn5a3hewe6.png" alt="A nil stdout writer, a panic, and the io.Discard guard" width="800" height="997"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The plan: reproduce it first
&lt;/h2&gt;

&lt;p&gt;I wanted this issue for the October 2026 Hackathon, and I didn't want to guess about the panic. I commented with a plan that started from a failing test:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I would like to work on this for the October 2026 Hackathon. I reproduced the panic on &lt;code&gt;main&lt;/code&gt; with a unit test that drives &lt;code&gt;winRMClient.Run&lt;/code&gt; through a fake &lt;code&gt;winrm.Transporter&lt;/code&gt; with &lt;code&gt;Stdout: nil&lt;/code&gt; (it crashes in &lt;code&gt;io.Copy&lt;/code&gt; inside &lt;code&gt;RunWithContextWithInput&lt;/code&gt;). Planned fix: default a nil &lt;code&gt;opts.Stdout&lt;/code&gt; to &lt;code&gt;io.Discard&lt;/code&gt; in &lt;code&gt;Run&lt;/code&gt;, plus that regression test.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The contributor platform assigned me, and I opened the MR the same day.&lt;/p&gt;

&lt;h2&gt;
  
  
  The guard
&lt;/h2&gt;

&lt;p&gt;The production change is a nil check at the top of &lt;code&gt;Run&lt;/code&gt;, before anything is passed to the library:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;winRMClient&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt; &lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Context&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt; &lt;span class="nc"&gt;RunOptions&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="c1"&gt;// winrm copies the command output to stdout unconditionally, so a nil&lt;/span&gt;
    &lt;span class="c1"&gt;// writer would panic.&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Stdout&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Stdout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;io&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Discard&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;stderrWriter&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;stderrOmitWriter&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Stderr&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;stderrWriter&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;newStderrOmitWriter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;io.Discard&lt;/code&gt; is the right default here: the caller asked to run a command and didn't provide a place for stdout, so the output should be thrown away, not written into a nil interface. Stderr already had its own guard, a &lt;code&gt;stderrOmitWriter&lt;/code&gt;. I didn't try to unify the two. I made stdout safe in the same &lt;em&gt;place&lt;/em&gt; — at the start of &lt;code&gt;Run&lt;/code&gt; — with the simplest writer that cannot panic.&lt;/p&gt;

&lt;h2&gt;
  
  
  A fake SOAP transport, so the test doesn't need Windows
&lt;/h2&gt;

&lt;p&gt;A unit test for this can't talk to a real WinRM endpoint. The winrm client speaks SOAP: it creates a shell, starts a command, and receives streams. I added &lt;code&gt;connector/winrm_test.go&lt;/code&gt; with a fake transport that answers those three requests and ignores the rest:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="c1"&gt;// fakeWinRMTransport answers WinRM SOAP requests with canned responses for a&lt;/span&gt;
&lt;span class="c1"&gt;// single command that writes "hello" to stdout and exits with exitCode.&lt;/span&gt;
&lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="n"&gt;fakeWinRMTransport&lt;/span&gt; &lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fakeWinRMTransport&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;Transport&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;winrm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Endpoint&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fakeWinRMTransport&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nc"&gt;Post&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;winrm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;soap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SoapMessage&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;string&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kt"&gt;error&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;switch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Contains&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"transfer/Create"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fakeWinRMCreateShellResponse&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Contains&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"shell/Command"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fakeWinRMCommandResponse&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Contains&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"shell/Receive"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fakeWinRMReceiveResponse&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;
    &lt;span class="k"&gt;default&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="c1"&gt;// Signal and Delete responses are not parsed by the client.&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The receive response carries a base64 stdout stream of &lt;code&gt;hello&lt;/code&gt; (&lt;code&gt;aGVsbG8=&lt;/code&gt;) and an exit code of 0. The client is built with &lt;code&gt;TransportDecorator&lt;/code&gt; so it uses the fake instead of the network:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="n"&gt;params&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;winrm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NewParameters&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"PT60S"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"en-US"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;153600&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;TransportDecorator&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;winrm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Transporter&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;fakeWinRMTransport&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two examples sit on top of that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight go"&gt;&lt;code&gt;&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nc"&gt;TestWinRMRunNilWriters&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nf"&gt;newFakeWinRMClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;require&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NotPanics&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;func&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Background&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;RunOptions&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nc"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"echo hello"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;require&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NoError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;func&lt;/span&gt; &lt;span class="nc"&gt;TestWinRMRunWritesStdout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;testing&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;T&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="nf"&gt;newFakeWinRMClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="kd"&gt;var&lt;/span&gt; &lt;span class="n"&gt;stdout&lt;/span&gt; &lt;span class="n"&gt;strings&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Builder&lt;/span&gt;
    &lt;span class="n"&gt;err&lt;/span&gt; &lt;span class="o"&gt;:=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Background&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;RunOptions&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nc"&gt;Command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"echo hello"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;Stdout&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;stdout&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;require&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;NoError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;require&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Equal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"hello"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stdout&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first one is the regression test. Without the guard it panics with &lt;code&gt;invalid memory address or nil pointer dereference&lt;/code&gt;. With the guard it passes. The second one is there so the discard path can't accidentally swallow output when a writer &lt;em&gt;is&lt;/em&gt; provided.&lt;/p&gt;

&lt;p&gt;I ran &lt;code&gt;go test ./... -count=1&lt;/code&gt; and &lt;code&gt;golangci-lint run ./connector/&lt;/code&gt; locally, and both were clean.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq78n2gbiod2g80jeti5w.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fq78n2gbiod2g80jeti5w.png" alt="The fake WinRM SOAP transport the tests use" width="800" height="433"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Review and merge
&lt;/h2&gt;

&lt;p&gt;GitLab Duo reviewed the MR and had no change requests. It checked that a nil &lt;code&gt;Stdout&lt;/code&gt; is replaced with &lt;code&gt;io.Discard&lt;/code&gt; before it reaches &lt;code&gt;runWinRMCmd&lt;/code&gt;, that other callers of &lt;code&gt;runWinRMCmd&lt;/code&gt; (&lt;code&gt;DialWinRM&lt;/code&gt;, &lt;code&gt;DialRun&lt;/code&gt;, &lt;code&gt;deployTunnelBinary&lt;/code&gt;) already pass non-nil writers, that the fake transport's substring routing doesn't overlap, and that &lt;code&gt;TestWinRMRunWritesStdout&lt;/code&gt; writes to a &lt;code&gt;strings.Builder&lt;/code&gt; from a single goroutine.&lt;/p&gt;

&lt;p&gt;I marked the MR ready for a maintainer. Arran Walker reviewed it, thanked me, and merged it on 14 October 2026, two days after I opened it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Reproduce a panic with a test before you write the fix.&lt;/strong&gt; The issue comment already said where it crashed (&lt;code&gt;io.Copy&lt;/code&gt; inside &lt;code&gt;RunWithContextWithInput&lt;/code&gt;). A &lt;code&gt;NotPanics&lt;/code&gt; example on &lt;code&gt;main&lt;/code&gt; turns that into something you can watch go from red to green.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Guard nil writers at the boundary you own.&lt;/strong&gt; The panic was inside a library that always writes to stdout. Fleeting can't change that library's contract, but it can stop passing nil in.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. &lt;code&gt;io.Discard&lt;/code&gt; is the nil stdout.&lt;/strong&gt; If the caller didn't provide a writer, throwing the bytes away is the behavior they asked for. Don't invent a more clever sink.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Fake the protocol at the transport boundary.&lt;/strong&gt; A WinRM unit test doesn't need a Windows host if you can answer Create, Command, and Receive with canned SOAP. Keep the routing strings tight so they don't overlap.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Keep a positive example next to the panic example.&lt;/strong&gt; &lt;code&gt;TestWinRMRunNilWriters&lt;/code&gt; proves we don't crash. &lt;code&gt;TestWinRMRunWritesStdout&lt;/code&gt; proves we still write when we should.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://gitlab.com/gitlab-org/fleeting/fleeting/-/work_items/66" rel="noopener noreferrer"&gt;gitlab-org/fleeting/fleeting#66&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Merge request: &lt;a href="https://gitlab.com/gitlab-org/fleeting/fleeting/-/merge_requests/151" rel="noopener noreferrer"&gt;gitlab-org/fleeting/fleeting!151&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The earlier stderr-handling work: &lt;a href="https://gitlab.com/gitlab-org/fleeting/fleeting/-/merge_requests/117" rel="noopener noreferrer"&gt;gitlab-org/fleeting/fleeting!117&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks to Brandon Bruce for the issue, and to Arran Walker for the review and the merge.&lt;/p&gt;

&lt;p&gt;If you've been thinking about contributing to GitLab, the community issues are a great place to start: &lt;a href="https://contributors.gitlab.com" rel="noopener noreferrer"&gt;contributors.gitlab.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>gitlab</category>
      <category>opensource</category>
      <category>go</category>
      <category>devops</category>
    </item>
    <item>
      <title>Cross-slot by default: moving Redis Cluster permission into ClusterUtil.batch</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Fri, 13 Nov 2026 18:15:49 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/cross-slot-by-default-moving-redis-cluster-permission-into-clusterutilbatch-336f</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/cross-slot-by-default-moving-redis-cluster-permission-into-clusterutilbatch-336f</guid>
      <description>&lt;p&gt;Redis Cluster does not let you run a multi-key command unless every key hashes to the same slot. That's a good default. It's also easy to forget, which is why GitLab has a validator that raises if you pipeline keys across slots, and a helper to allow it when you mean to.&lt;/p&gt;

&lt;p&gt;This is the story of &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/265488" rel="noopener noreferrer"&gt;gitlab-org/gitlab!265488&lt;/a&gt;. It moves that permission into &lt;code&gt;Gitlab::Redis::ClusterUtil.batch&lt;/code&gt;, so the methods that already batch across slots don't make every caller opt in. I'll cover the issue, the change I shipped, the wrappers I deliberately left alone, and the review that kept the parent issue open.&lt;/p&gt;

&lt;h2&gt;
  
  
  The issue: a permission every caller had to remember
&lt;/h2&gt;

&lt;p&gt;The issue was &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/work_items/474372" rel="noopener noreferrer"&gt;gitlab-org/gitlab#474372&lt;/a&gt;, opened by Sylvester Chin. It started as a suggestion on an earlier MR, &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/159542#note_2014931413" rel="noopener noreferrer"&gt;!159542&lt;/a&gt;: if &lt;code&gt;ClusterUtil&lt;/code&gt; is the place that pipelines keys, &lt;code&gt;ClusterUtil&lt;/code&gt; should be the place that allows cross-slot commands.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Gitlab::Redis::ClusterUtil.batch&lt;/code&gt; is used by &lt;code&gt;batch_get&lt;/code&gt;, &lt;code&gt;batch_del&lt;/code&gt;, and &lt;code&gt;batch_unlink&lt;/code&gt;. Those methods take a list of keys, slice them, and pipeline a command for each slice. The keys can hash to different slots. That's expected. What's less expected is that every caller still had to wrap the call:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="nc"&gt;Gitlab&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Instrumentation&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;RedisClusterValidator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;allow_cross_slot_commands&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nc"&gt;Gitlab&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;ClusterUtil&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;batch_get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;keys&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Forget the wrapper, and you get a &lt;code&gt;CrossSlotError&lt;/code&gt; for a method whose whole job is to batch keys. The issue also listed a second, broader cleanup: &lt;code&gt;Rails.cache.delete_multi&lt;/code&gt;, &lt;code&gt;fetch_multi&lt;/code&gt;, and &lt;code&gt;read_multi&lt;/code&gt; call sites could drop their wrappers once the permission lived lower down.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fptqjcvbmxuquy8gmlmnk.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fptqjcvbmxuquy8gmlmnk.png" alt="Callers wrapping ClusterUtil, and the wrapper moving inside batch" width="800" height="713"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The plan, and what I decided not to do in the same MR
&lt;/h2&gt;

&lt;p&gt;I commented on the issue with this plan:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Plan: wrap the pipelined calls in &lt;code&gt;Gitlab::Redis::ClusterUtil.batch&lt;/code&gt; (used by &lt;code&gt;batch_get&lt;/code&gt;/&lt;code&gt;batch_del&lt;/code&gt;/&lt;code&gt;batch_unlink&lt;/code&gt;) with &lt;code&gt;Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands&lt;/code&gt;, so callers no longer need to wrap them. I'll then drop the now-redundant wrappers at call sites that only call &lt;code&gt;ClusterUtil&lt;/code&gt;, and add a spec showing the batch methods work without an outer &lt;code&gt;allow_cross_slot_commands&lt;/code&gt; block. I'll keep the diff small and leave the &lt;code&gt;Rails.cache.*_multi&lt;/code&gt; callers for a follow-up unless you'd like them in the same MR.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The contributor platform assigned me, and I opened the MR with a smaller scope than that plan's middle sentence. I did not drop the wrappers at existing call sites.&lt;/p&gt;

&lt;p&gt;The reason is in the MR description. Callers such as &lt;code&gt;Gitlab::AvatarCache&lt;/code&gt;, &lt;code&gt;Gitlab::SetCache&lt;/code&gt;, &lt;code&gt;Gitlab::RepositoryHashCache&lt;/code&gt;, &lt;code&gt;Gitlab::ReactiveCacheSetCache&lt;/code&gt;, and &lt;code&gt;Gitlab::SidekiqStatus&lt;/code&gt; don't only go through &lt;code&gt;ClusterUtil&lt;/code&gt;. Their non-cluster branches still run commands like &lt;code&gt;redis.unlink(*keys)&lt;/code&gt; and &lt;code&gt;redis.mget(*keys)&lt;/code&gt;, which need the wrapper on their own. Removing it because the cluster path no longer needs it would have been a nice-looking cleanup and a real bug on the other branch.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;Rails.cache.*_multi&lt;/code&gt; callers mentioned in the issue were left for a follow-up, as planned.&lt;/p&gt;

&lt;h2&gt;
  
  
  The change
&lt;/h2&gt;

&lt;p&gt;The production change is a wrap around the existing loop in &lt;code&gt;ClusterUtil.batch&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nc"&gt;Gitlab&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Instrumentation&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;RedisClusterValidator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;allow_cross_slot_commands&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
    &lt;span class="n"&gt;entries&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;each_slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pipeline_batch_size&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flat_map&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;subset&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
      &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pipelined&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="n"&gt;pipeline&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;subset&lt;/span&gt;
      &lt;span class="k"&gt;end&lt;/span&gt;
    &lt;span class="k"&gt;end&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;batch_get&lt;/code&gt;, &lt;code&gt;batch_del&lt;/code&gt;, and &lt;code&gt;batch_unlink&lt;/code&gt; all go through this method, so they inherit the permission. The validator tracks nesting, so a caller that still has its own &lt;code&gt;allow_cross_slot_commands&lt;/code&gt; block keeps working. I used that fact rather than touching those callers.&lt;/p&gt;

&lt;p&gt;The specs had to change in two ways. First, the existing batch examples no longer wrap themselves in &lt;code&gt;allow_cross_slot_commands&lt;/code&gt;. If the production change were missing, those examples would fail, which is the point:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s1"&gt;'gets multiple keys without an outer allow_cross_slot_commands block'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nc"&gt;Gitlab&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;with&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
    &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;described_class&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;batch_get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;array&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Second, I added examples on &lt;code&gt;.batch&lt;/code&gt; itself that check the flag is on while the block runs, and that it is restored afterwards, including when the block raises:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s1"&gt;'allows cross slot commands while yielding and restores the state afterwards'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="n"&gt;allowed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

  &lt;span class="nc"&gt;Gitlab&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;with&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
    &lt;span class="n"&gt;described_class&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;_pipeline&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_subset&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
      &lt;span class="n"&gt;allowed&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;validator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;allow_cross_slot_commands?&lt;/span&gt;
      &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;end&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;

  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;allowed&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;validator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;allow_cross_slot_commands?&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;be&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;

&lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s1"&gt;'restores the state when the block raises'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nc"&gt;Gitlab&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Redis&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Cache&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;with&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;described_class&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;raise&lt;/span&gt; &lt;span class="s1"&gt;'boom'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;raise_error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'boom'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;

  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;validator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;allow_cross_slot_commands?&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;be&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A helper that flips a process-wide (or thread-local) flag and does not restore it on an exception is a gift to the next test, and to the next request.&lt;/p&gt;

&lt;p&gt;The description also has a local validation step: in a Rails console, call &lt;code&gt;batch_get(%w[a b c], redis)&lt;/code&gt; inside &lt;code&gt;Gitlab::Redis::Cache.with&lt;/code&gt; &lt;em&gt;without&lt;/em&gt; wrapping it, and it should no longer raise &lt;code&gt;CrossSlotError&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The review: LGTM, and don't close the issue
&lt;/h2&gt;

&lt;p&gt;I marked the MR ready, then sent a gentle ping a few days later. Saahil Sabu Hameed reviewed it, approved it, and asked for one description change:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The issue also asks for a cleanup.&lt;/p&gt;

&lt;p&gt;All usages of &lt;code&gt;Rails.cache.delete_multi&lt;/code&gt; or &lt;code&gt;.fetch_multi&lt;/code&gt; or &lt;code&gt;.read_multi&lt;/code&gt; can be cleaned up to not need &lt;code&gt;Gitlab::Instrumentation::RedisClusterValidator.allow_cross_slot_commands&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Lets update the MR description to say "Related to" instead of "Closes"? You can open up another MR for the cleanup if you are interested in working on that as well.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's the right call. This MR made the batch helpers safe to call on their own. It did not finish the issue. Closing #474372 would have made the leftover work harder to find. I updated the description to "Related to," and said I was happy to take the &lt;code&gt;Rails.cache&lt;/code&gt; cleanup as a follow-up.&lt;/p&gt;

&lt;p&gt;Saahil asked Drew Blessing for the backend maintainer review. Drew approved it and added it to the merge train. The MR merged on 3 November 2026.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fldw4rczxc3cxuqsa95cf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fldw4rczxc3cxuqsa95cf.png" alt="The leftover wrappers, and the issue left open for the Rails.cache cleanup" width="799" height="683"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Put the permission next to the operation that needs it.&lt;/strong&gt; If &lt;code&gt;ClusterUtil.batch&lt;/code&gt; is the thing that pipelines keys across slots, callers shouldn't have to remember to allow that. The next call site will forget.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. A wrapper that still protects another branch is not redundant.&lt;/strong&gt; I had planned to drop wrappers at call sites that "only" use &lt;code&gt;ClusterUtil&lt;/code&gt;. Looking at those files, they also have non-cluster &lt;code&gt;mget&lt;/code&gt; and &lt;code&gt;unlink&lt;/code&gt; paths. I left them alone. A cleanup that is only safe on one branch is not a cleanup.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Don't close an issue you only partly solved.&lt;/strong&gt; "Related to" is more honest than "Closes" when the issue also asks for a &lt;code&gt;Rails.cache&lt;/code&gt; cleanup. Saahil caught that in review.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. If you flip a flag, test that it flips back when the block raises.&lt;/strong&gt; The happy-path restore is easy to write and easy to get right. The exception path is the one that leaks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Nesting matters.&lt;/strong&gt; Because the validator already tracked nested &lt;code&gt;allow_cross_slot_commands&lt;/code&gt; blocks, I could add the inner wrap without breaking callers that still have an outer one. That's what made a small MR safe.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/work_items/474372" rel="noopener noreferrer"&gt;gitlab-org/gitlab#474372&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Merge request: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/265488" rel="noopener noreferrer"&gt;gitlab-org/gitlab!265488&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Where the idea came from: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/159542#note_2014931413" rel="noopener noreferrer"&gt;gitlab-org/gitlab!159542&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GitLab's Redis development guide, including multi-key commands: &lt;a href="https://docs.gitlab.com/ee/development/redis.html#multi-key-commands" rel="noopener noreferrer"&gt;docs.gitlab.com/development/redis&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks to Sylvester Chin for the issue, Saahil Sabu Hameed for the review and for keeping the leftover work visible, and Drew Blessing for the maintainer review.&lt;/p&gt;

&lt;p&gt;If you've been thinking about contributing to GitLab, the community issues are a great place to start: &lt;a href="https://contributors.gitlab.com" rel="noopener noreferrer"&gt;contributors.gitlab.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>gitlab</category>
      <category>opensource</category>
      <category>redis</category>
      <category>ruby</category>
    </item>
    <item>
      <title>A job that could never succeed: skipping note emails for deleted recipients</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Fri, 13 Nov 2026 18:15:35 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/a-job-that-could-never-succeed-skipping-note-emails-for-deleted-recipients-n23</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/a-job-that-could-never-succeed-skipping-note-emails-for-deleted-recipients-n23</guid>
      <description>&lt;p&gt;Some bugs don't fail for the user who is still around. They fail for a user who is already gone, in a background job that cannot possibly succeed, and they leave a trail of retries in the dead set.&lt;/p&gt;

&lt;p&gt;This is the story of &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/268706" rel="noopener noreferrer"&gt;gitlab-org/gitlab!268706&lt;/a&gt;. It stops GitLab from trying to send a note notification after the recipient has been deleted, and it went through a review that made both the guard and the specs stricter. I'll cover the issue, the first version, and the two review comments that changed it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The issue: the same failure, on the other lookup
&lt;/h2&gt;

&lt;p&gt;The issue was &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/work_items/638487" rel="noopener noreferrer"&gt;gitlab-org/gitlab#638487&lt;/a&gt;, opened by Danny Bailey. It is a follow-up, and that matters.&lt;/p&gt;

&lt;p&gt;An earlier MR, &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/267804" rel="noopener noreferrer"&gt;!267804&lt;/a&gt;, had fixed the same shape of failure for &lt;em&gt;deleted notes&lt;/em&gt;. &lt;code&gt;Emails::Notes#setup_note_mail&lt;/code&gt; used to load the note with a find that raises when the record is missing. If the note was deleted after a &lt;code&gt;Notify#note_*_email&lt;/code&gt; job was queued but before the job ran, &lt;code&gt;ActiveRecord::RecordNotFound&lt;/code&gt; was raised, &lt;code&gt;ActionMailer::MailDeliveryJob&lt;/code&gt; retried it, and the job ended in the dead set. The fix was to load the note with &lt;code&gt;Note.find_by_id&lt;/code&gt; and return &lt;code&gt;false&lt;/code&gt; when it was missing, so the action returned before calling &lt;code&gt;mail&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The recipient lookup was left out of scope. It still used &lt;code&gt;User.find(recipient_id)&lt;/code&gt;. If the recipient was deleted in that same window, the job took the same path: raise, retry, dead set, and it could never succeed.&lt;/p&gt;

&lt;p&gt;Danny's proposal was to mirror the note guard: look the recipient up with &lt;code&gt;User.find_by_id&lt;/code&gt;, and skip delivery when it is missing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftru94ciqojj2ebw2dqz9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftru94ciqojj2ebw2dqz9.png" alt="From a queued note email to the dead set, and the skip after the fix" width="800" height="347"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The plan
&lt;/h2&gt;

&lt;p&gt;I commented on the issue with this plan:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I plan to mirror the deleted-note guard from !267804: load the recipient with &lt;code&gt;User.find_by_id&lt;/code&gt; in &lt;code&gt;Emails::Notes#setup_note_mail&lt;/code&gt; and return early when missing, with a focused spec.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Danny assigned me the same day: "Sure thing, I've assigned you, go for it." Matching an existing, just-merged pattern in the same method is the easiest kind of change to review, as long as the new guard really behaves like the old one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Version 1: the same method, one mailer
&lt;/h2&gt;

&lt;p&gt;The production change in the first commit (&lt;code&gt;d89428c0&lt;/code&gt;) was two lines in &lt;code&gt;setup_note_mail&lt;/code&gt;. After the note is loaded and the project and group are derived, the recipient is no longer loaded with &lt;code&gt;find&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="vi"&gt;@recipient&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find_by_id&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;recipient_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="k"&gt;unless&lt;/span&gt; &lt;span class="vi"&gt;@recipient&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If that returns &lt;code&gt;false&lt;/code&gt;, every note email action returns before calling &lt;code&gt;mail&lt;/code&gt;, and &lt;code&gt;SentNotification.record_note&lt;/code&gt; is never reached. The YARD comment on the method now says it returns false when the note &lt;em&gt;or&lt;/em&gt; the recipient was deleted before the job ran.&lt;/p&gt;

&lt;p&gt;The first spec covered &lt;code&gt;note_issue_email&lt;/code&gt; only. It used a &lt;code&gt;non_existing_record_id&lt;/code&gt; as the recipient, shared the &lt;code&gt;'no email is sent'&lt;/code&gt; example, and asserted that &lt;code&gt;deliver_now&lt;/code&gt; did not create a &lt;code&gt;SentNotification&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;context&lt;/span&gt; &lt;span class="s1"&gt;'when the recipient is deleted before the email is sent'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nf"&gt;let_it_be&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:note&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:note_on_issue&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;noteable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;issue&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;project&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;project&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="n"&gt;subject&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;described_class&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;note_issue_email&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;non_existing_record_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;note&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="n"&gt;it_behaves_like&lt;/span&gt; &lt;span class="s1"&gt;'no email is sent'&lt;/span&gt;

  &lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s1"&gt;'does not record a sent notification'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;subject&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;deliver_now&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;not_to&lt;/span&gt; &lt;span class="n"&gt;change&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;SentNotification&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One small workflow bump: my first &lt;code&gt;@gitlab-bot ready&lt;/code&gt; ran before GitLab had indexed the diff, so no reviewer was assigned. I posted ready again once the change was visible, and the pipeline was running.&lt;/p&gt;

&lt;h2&gt;
  
  
  The review: return earlier, and test the path that actually failed
&lt;/h2&gt;

&lt;p&gt;Danny left two non-blocking suggestions, both of which made the change better, and I applied both in &lt;code&gt;82c370a6&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Don't derive a project for a mail you will not send
&lt;/h3&gt;

&lt;p&gt;The first version looked up the recipient &lt;em&gt;after&lt;/em&gt; setting &lt;code&gt;@project&lt;/code&gt; and &lt;code&gt;@group&lt;/code&gt;. Danny's nitpick:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;consider looking up the recipient right after the note guard, so we bail out before deriving &lt;code&gt;@project&lt;/code&gt;/&lt;code&gt;@group&lt;/code&gt; when there's nothing to send.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's a small move with a clear reason. If the recipient is gone, there is no mail, and there should be no &lt;code&gt;SentNotification&lt;/code&gt;. Doing the extra work first doesn't break anything, but the method reads as "set up the mail, then maybe notice we can't send it." The final order is note, then recipient, then project and group:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;setup_note_mail&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;note_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;recipient_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="vi"&gt;@note&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;note_id&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_a?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Note&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;?&lt;/span&gt; &lt;span class="n"&gt;note_id&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nc"&gt;Note&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find_by_id&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;note_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="k"&gt;unless&lt;/span&gt; &lt;span class="vi"&gt;@note&lt;/span&gt;

  &lt;span class="vi"&gt;@recipient&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;User&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;find_by_id&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;recipient_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="k"&gt;unless&lt;/span&gt; &lt;span class="vi"&gt;@recipient&lt;/span&gt;

  &lt;span class="vi"&gt;@project&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="vi"&gt;@note&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;project&lt;/span&gt;
  &lt;span class="vi"&gt;@group&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="vi"&gt;@note&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;noteable&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;try&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="vi"&gt;@group&lt;/span&gt; &lt;span class="o"&gt;||=&lt;/span&gt; &lt;span class="vi"&gt;@note&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;noteable&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;resource_parent&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vi"&gt;@note&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;noteable&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;try&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:resource_parent&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;is_a?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

  &lt;span class="vi"&gt;@sent_notification&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;SentNotification&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;record_note&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="vi"&gt;@note&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;recipient_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="vi"&gt;@project&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="vi"&gt;@group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="vi"&gt;@note&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;persisted?&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  2. Cover every note mailer, through &lt;code&gt;deliver_later&lt;/code&gt;
&lt;/h3&gt;

&lt;p&gt;The issue's failure is not &lt;code&gt;deliver_now&lt;/code&gt;. It is a job that raises inside &lt;code&gt;ActionMailer::MailDeliveryJob&lt;/code&gt;, is retried, and lands in the dead set. Danny pointed at a shared example that already existed for deleted notes:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;could this cover all six note mailers and use &lt;code&gt;'a note email for a deleted note'&lt;/code&gt;, like the deleted-note context just above? That shared example runs the job through &lt;code&gt;deliver_later&lt;/code&gt; and asserts it completes without raising, which is the failure from #638487.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;My first spec used &lt;code&gt;'no email is sent'&lt;/code&gt; and only &lt;code&gt;note_issue_email&lt;/code&gt;. That shows no mail goes out on the synchronous path. It does not show that the &lt;em&gt;job&lt;/em&gt; completes without raising, and it does not show that &lt;code&gt;note_commit_email&lt;/code&gt;, &lt;code&gt;note_merge_request_email&lt;/code&gt;, &lt;code&gt;note_snippet_email&lt;/code&gt;, &lt;code&gt;note_design_email&lt;/code&gt;, and &lt;code&gt;note_wiki_page_email&lt;/code&gt; take the same early return. Those six methods all go through &lt;code&gt;setup_note_mail&lt;/code&gt;, but a regression in one of them is exactly the kind of thing a parameterized example is for.&lt;/p&gt;

&lt;p&gt;The final spec runs all six through the shared example, keeps the &lt;code&gt;SentNotification&lt;/code&gt; check, and adds the same case for &lt;code&gt;note_epic_email&lt;/code&gt; in &lt;code&gt;ee/spec/mailers/notify_spec.rb&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;context&lt;/span&gt; &lt;span class="s1"&gt;'when the recipient is deleted before the email is sent'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nf"&gt;let_it_be&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:note&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:note_on_issue&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;noteable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;issue&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;project&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;project&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:mailer_method&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
    &lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
      &lt;span class="n"&gt;note_commit_email&lt;/span&gt;
      &lt;span class="n"&gt;note_issue_email&lt;/span&gt;
      &lt;span class="n"&gt;note_merge_request_email&lt;/span&gt;
      &lt;span class="n"&gt;note_snippet_email&lt;/span&gt;
      &lt;span class="n"&gt;note_design_email&lt;/span&gt;
      &lt;span class="n"&gt;note_wiki_page_email&lt;/span&gt;
    &lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;

  &lt;span class="n"&gt;with_them&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
    &lt;span class="n"&gt;subject&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;described_class&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;public_send&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mailer_method&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;non_existing_record_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;note&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="n"&gt;it_behaves_like&lt;/span&gt; &lt;span class="s1"&gt;'a note email for a deleted note'&lt;/span&gt;

    &lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s1"&gt;'does not record a sent notification'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
      &lt;span class="n"&gt;expect&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;subject&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;deliver_now&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;not_to&lt;/span&gt; &lt;span class="n"&gt;change&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;SentNotification&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
    &lt;span class="k"&gt;end&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Danny reviewed the new version, approved it, and asked Serena Fang for a second backend look. Serena approved it and set merge when pipeline succeeds. The MR merged on 10 November 2026.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxx9tfomht9s4k0w2dpju.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxx9tfomht9s4k0w2dpju.png" alt="Review: an earlier return, and every note mailer through deliver_later" width="800" height="691"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. When a just-merged MR leaves a lookup out of scope, that's often the next issue.&lt;/strong&gt; The deleted-note guard was the template. I didn't need a new design. I needed to apply the same pattern to the line that was skipped.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. &lt;code&gt;find&lt;/code&gt; and &lt;code&gt;find_by_id&lt;/code&gt; are not interchangeable in jobs.&lt;/strong&gt; &lt;code&gt;find&lt;/code&gt; raises &lt;code&gt;RecordNotFound&lt;/code&gt;. A mail job treats that as a transient failure and retries it. If the record is gone for good, the retries are wasted work and the dead set is noise. &lt;code&gt;find_by_id&lt;/code&gt; returns &lt;code&gt;nil&lt;/code&gt;, and you can leave quietly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Put the guard before the work, not after.&lt;/strong&gt; Looking up the recipient after deriving &lt;code&gt;@project&lt;/code&gt; and &lt;code&gt;@group&lt;/code&gt; was correct, but it wasn't kind to the next reader. If you're going to return, return first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Test the failure mode from the issue, not a convenient stand-in.&lt;/strong&gt; &lt;code&gt;'no email is sent'&lt;/code&gt; plus &lt;code&gt;deliver_now&lt;/code&gt; is weaker than &lt;code&gt;'a note email for a deleted note'&lt;/code&gt; plus &lt;code&gt;deliver_later&lt;/code&gt;. The bug was a job that raised. The spec should be a job that doesn't.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Parameterize across every entry point that shares the guard.&lt;/strong&gt; Six FOSS mailers and the EE epic mailer all call &lt;code&gt;setup_note_mail&lt;/code&gt;. One example on &lt;code&gt;note_issue_email&lt;/code&gt; would have left the rest to faith.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/work_items/638487" rel="noopener noreferrer"&gt;gitlab-org/gitlab#638487&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Merge request: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/268706" rel="noopener noreferrer"&gt;gitlab-org/gitlab!268706&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The deleted-note fix this follows: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/267804" rel="noopener noreferrer"&gt;gitlab-org/gitlab!267804&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks to Danny Bailey for the issue, for pointing at the shared example I should have used first, and for the early-return nudge, and to Serena Fang for the maintainer review.&lt;/p&gt;

&lt;p&gt;If you've been thinking about contributing to GitLab, the community issues are a great place to start: &lt;a href="https://contributors.gitlab.com" rel="noopener noreferrer"&gt;contributors.gitlab.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>gitlab</category>
      <category>opensource</category>
      <category>rails</category>
      <category>ruby</category>
    </item>
    <item>
      <title>A prefix away from deleting a live file: fixing Geo's in-use check</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Fri, 13 Nov 2026 18:15:16 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/a-prefix-away-from-deleting-a-live-file-fixing-geos-in-use-check-35jp</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/a-prefix-away-from-deleting-a-live-file-fixing-geos-in-use-check-35jp</guid>
      <description>&lt;p&gt;GitLab Geo keeps a secondary site in sync with a primary. When a file is deleted on the primary, the secondary has to delete its copy too. That sounds simple, until two records share a path, or the path the secondary stored is not quite the path the "is this still in use?" check looks at.&lt;/p&gt;

&lt;p&gt;This is the story of &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/265937" rel="noopener noreferrer"&gt;gitlab-org/gitlab!265937&lt;/a&gt;, which stopped Geo from treating a perfectly live object-storage file as unused. I'll cover the issue, the plan, the two-path check, a local-path trap that made a one-sided fix dangerous, and the review that tightened the specs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The issue: a safety check that never matched
&lt;/h2&gt;

&lt;p&gt;The issue was &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/work_items/636176" rel="noopener noreferrer"&gt;gitlab-org/gitlab#636176&lt;/a&gt;, opened by Chloe Fons. It came out of work to roll out a feature flag, &lt;code&gt;geo_blob_removal_uses_registry_location&lt;/code&gt; (&lt;a href="https://gitlab.com/gitlab-org/gitlab/-/work_items/635220" rel="noopener noreferrer"&gt;#635220&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;Some background. On a Geo secondary, each blob registry row stores a &lt;code&gt;blob_path&lt;/code&gt;: the file's path relative to its uploader root, or its object storage key. When a delete event arrives without a path, &lt;code&gt;Geo::DestroyWorker&lt;/code&gt; falls back to that stored location and &lt;code&gt;Geo::FileRegistryRemovalService&lt;/code&gt; deletes the file. Before it does, &lt;code&gt;blob_location_in_use?&lt;/code&gt; asks whether a &lt;em&gt;newer&lt;/em&gt; record reuses the same path, so a live file is never removed.&lt;/p&gt;

&lt;p&gt;With object storage replication and a &lt;code&gt;bucket_prefix&lt;/code&gt;, that check broke down. &lt;code&gt;ObjectStorage::Concern#store_path&lt;/code&gt; adds the prefix, so the path saved on the registry is &lt;code&gt;&amp;lt;bucket_prefix&amp;gt;/&amp;lt;key&amp;gt;&lt;/code&gt;. The values the in-use overrides compare against never include it:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Blob type&lt;/th&gt;
&lt;th&gt;What the in-use check compares&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Uploads&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Upload#path&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;LFS objects&lt;/td&gt;
&lt;td&gt;the LFS oid&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Helm metadata caches, NuGet symbols&lt;/td&gt;
&lt;td&gt;&lt;code&gt;object_storage_key&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So the check never matched for those types. Terraform state versions only read the last two path segments, so they were not affected. For the others, the fallback could delete an object that a newer record still used.&lt;/p&gt;

&lt;p&gt;This had to land before the flag was enabled by default. A wrong "in use" result only leaves an orphaned file. A wrong "not in use" result deletes a file that a live record uses.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fznvpuduzpn896na19ovo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fznvpuduzpn896na19ovo.png" alt="How a prefixed path missed the in-use check" width="800" height="1615"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The plan, before writing any code
&lt;/h2&gt;

&lt;p&gt;I commented on the issue with a plan that followed Chloe's write-up:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Planned approach, following the implementation plan: in &lt;code&gt;ee/app/models/concerns/geo/blob_replicator_strategy.rb&lt;/code&gt;, &lt;code&gt;registry_location_params&lt;/code&gt; will call a new private &lt;code&gt;stored_location_in_use?(blob_path, uploader_class)&lt;/code&gt; that returns true when &lt;code&gt;blob_location_in_use?&lt;/code&gt; matches either the stored path or the path with the uploader's &lt;code&gt;"#{bucket_prefix}/"&lt;/code&gt; removed (checking both, since a local path can start with the prefix by chance). The returned location keeps the prefix for the removal service. I'll add specs with a bucket prefix for an upload replicator and an object storage key type (Helm metadata cache / NuGet symbol).&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Chloe assigned me and highlighted one detail:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Your plan sounds good. If you could also follow the Specs section as written, especially the third user-upload example, that'd be awesome.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That third example is the reason the fix checks &lt;em&gt;both&lt;/em&gt; paths, and it's the one I was most careful to keep.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two-path check
&lt;/h2&gt;

&lt;p&gt;The production change is small, and it sits in one method. &lt;code&gt;registry_location_params&lt;/code&gt; used to call &lt;code&gt;blob_location_in_use?&lt;/code&gt; on the stored path alone. It now calls a new private method, and the hash it returns still carries the prefixed path, because the removal service needs that key to find the object:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Object storage keys start with the bucket prefix, but the columns the in-use checks match never include it.&lt;/span&gt;
&lt;span class="c1"&gt;# A local path can start with the prefix too, so a match with or without it counts.&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;stored_location_in_use?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;blob_path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;uploader_class&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="n"&gt;bucket_prefix&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;uploader_class&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;constantize&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;object_store_options&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;bucket_prefix&lt;/span&gt;
  &lt;span class="n"&gt;paths&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;blob_path&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="n"&gt;paths&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;blob_path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;delete_prefix&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"#{bucket_prefix}/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;bucket_prefix&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;present?&lt;/span&gt;

  &lt;span class="n"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uniq&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;any?&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nf"&gt;blob_location_in_use?&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;uploader_class&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why both paths, not only the stripped one? A local path can start with the prefix by chance. If &lt;code&gt;remote_directory&lt;/code&gt; is &lt;code&gt;gitlab/uploads&lt;/code&gt;, the prefix is &lt;code&gt;uploads&lt;/code&gt;, and a local avatar path looks like &lt;code&gt;uploads/-/system/...&lt;/code&gt;. If you only check the shorter path, a live local file whose path &lt;em&gt;is&lt;/em&gt; the stored path can look unused. The issue is explicit about the trade-off: fail safe (leave an orphan) rather than fail loose (delete a live file).&lt;/p&gt;

&lt;p&gt;The existing &lt;code&gt;ObjectStorage::Concern.without_bucket_prefix&lt;/code&gt; helper does not fit here. For a path outside the prefix, &lt;code&gt;Pathname#relative_path_from&lt;/code&gt; returns &lt;code&gt;../../...&lt;/code&gt;, which is not a useful comparison key. The default &lt;code&gt;blob_location_in_use?&lt;/code&gt; and its replicator overrides stayed unchanged.&lt;/p&gt;

&lt;h2&gt;
  
  
  Specs that fail without the fix
&lt;/h2&gt;

&lt;p&gt;The issue laid out the examples to add, and I followed that list.&lt;/p&gt;

&lt;p&gt;The shared example &lt;code&gt;a blob replicator whose blob path can be reused&lt;/code&gt; is already used by the LFS, Helm metadata cache, NuGet symbol, and Terraform replicator specs. I added a &lt;code&gt;#replicate_destroy with a bucket prefix&lt;/code&gt; block that stubs the prefix to &lt;code&gt;my/prefix&lt;/code&gt;, stores the prefixed path on the registry, deletes the original record, and then checks two outcomes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;When a record reuses the unprefixed path, &lt;code&gt;Geo::FileRegistryRemovalService&lt;/code&gt; is constructed with &lt;code&gt;nil, nil&lt;/code&gt; — no location, so the file is left alone.&lt;/li&gt;
&lt;li&gt;When nothing reuses the path, the service receives the &lt;em&gt;prefixed&lt;/em&gt; absolute path and the uploader class, so the object can actually be found and removed.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;User uploads needed their own copy of those two examples in &lt;code&gt;user_upload_replicator_spec.rb&lt;/code&gt;, because avatars use &lt;code&gt;AvatarUploader&lt;/code&gt; and can reuse a path. The third example is the local-path trap:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;context&lt;/span&gt; &lt;span class="s1"&gt;'when a local path starts with the prefix'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nf"&gt;let&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:bucket_prefix&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'uploads'&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="nf"&gt;let&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:stored_blob_path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;local_path&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s1"&gt;'passes no location when an upload reuses the stored path'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
    &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;local_path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;start_with&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"#{bucket_prefix}/"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:upload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;model_record&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;local_path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nf"&gt;expect_removal_service_with&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;nil&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;secondary_side_replicator&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replicate_destroy&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The prefix is &lt;code&gt;uploads&lt;/code&gt;, the stored path is the local avatar path unchanged, and a new upload reuses it. If the code only checked the stripped path, this example would fail.&lt;/p&gt;

&lt;p&gt;I was honest in the MR about what I had and hadn't run. I had a sparse checkout without the GitLab Rails and database environment, so I ran RuboCop on the three changed files and left the new examples to CI. The command from the issue is &lt;code&gt;bin/rspec ee/spec/replicators/geo/ ee/spec/workers/geo/destroy_worker_spec.rb&lt;/code&gt;. I also wrote down how to confirm the &lt;code&gt;nil, nil&lt;/code&gt; examples fail without the production change: revert the strategy file and rerun.&lt;/p&gt;

&lt;h2&gt;
  
  
  The review: cleaner specs, and one more example
&lt;/h2&gt;

&lt;p&gt;Chloe's first pass was "a great start," with a handful of spec suggestions and a question:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I wonder if we should also add one example that creates an upload with the &lt;code&gt;:object_storage&lt;/code&gt; factory trait and the prefix stubbed, and asserts that &lt;code&gt;replicator.deleted_params[:blob_path]&lt;/code&gt; starts with the prefix.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's a different path from &lt;code&gt;replicate_destroy&lt;/code&gt;. &lt;code&gt;deleted_params&lt;/code&gt; is what gets &lt;em&gt;written&lt;/em&gt; to the registry at sync time. If the stored path never starts with the prefix, the in-use fix has nothing to strip. I added a &lt;code&gt;#deleted_params with a bucket prefix&lt;/code&gt; example that creates the upload with &lt;code&gt;:remote_store&lt;/code&gt; (the &lt;code&gt;:object_storage&lt;/code&gt; trait), stubs object storage and the prefix, and checks the blob path starts with &lt;code&gt;my/prefix/&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The other comments were about making the shared examples leaner, and I applied all of them in &lt;code&gt;55222358&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Remove &lt;code&gt;uploader_class:&lt;/code&gt; and the attribute &lt;code&gt;select&lt;/code&gt; from the shared-example registry. Those examples are only used by LFS, Helm, NuGet, and Terraform, none of which have an uploader class.&lt;/li&gt;
&lt;li&gt;Move &lt;code&gt;blob_path&lt;/code&gt; and &lt;code&gt;uploader_class&lt;/code&gt; out of the inner &lt;code&gt;describe&lt;/code&gt;, so they're defined once.&lt;/li&gt;
&lt;li&gt;Drop explicit &lt;code&gt;model_record&lt;/code&gt; loads that weren't needed, because later lines instantiate the model anyway.&lt;/li&gt;
&lt;li&gt;In the user-upload examples, don't read &lt;code&gt;model_record.model&lt;/code&gt; into a &lt;code&gt;let&lt;/code&gt; before delete. After &lt;code&gt;model_record.delete&lt;/code&gt;, &lt;code&gt;model_record.model&lt;/code&gt; still loads, and the specs now use &lt;code&gt;create(:upload, model: model_record.model, path: local_path)&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Chloe reviewed the new version and approved it. Chloe also merged the MR, on 30 October 2026.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzx505t4xmll4j8ag6yah.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzx505t4xmll4j8ag6yah.png" alt="The two-path check, and the local path that starts with the prefix" width="800" height="608"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Asymmetry matters more than symmetry.&lt;/strong&gt; A check that protects deletions should bias toward "in use." An orphaned file can be cleaned up later. A deleted live file cannot.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Strip a prefix, but also keep the original.&lt;/strong&gt; The bug was "the stored path has a prefix the comparison column doesn't." The trap is "sometimes the stored path &lt;em&gt;is&lt;/em&gt; a local path that starts with the same letters." Both comparisons have to pass.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Follow the issue's spec plan, especially the awkward example.&lt;/strong&gt; The third user-upload example exists to fail a one-sided fix. Chloe asked for it by name, and it earned its place.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. A review that deletes setup is as useful as one that adds it.&lt;/strong&gt; Moving lets up, dropping unused registry fields, and not pre-loading associations made the examples easier to read and less brittle.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Say what you couldn't run locally.&lt;/strong&gt; I didn't have a full GitLab Rails environment. Writing that down, plus the exact rspec command and how to prove the examples fail without the fix, made the review easier to start.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/work_items/636176" rel="noopener noreferrer"&gt;gitlab-org/gitlab#636176&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Merge request: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/265937" rel="noopener noreferrer"&gt;gitlab-org/gitlab!265937&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Feature flag rollout this had to land before: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/work_items/635220" rel="noopener noreferrer"&gt;gitlab-org/gitlab#635220&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks to Chloe Fons for an issue that already contained the method, the trade-off, and the examples, and for a review that made the specs match that care.&lt;/p&gt;

&lt;p&gt;If you've been thinking about contributing to GitLab, the community issues are a great place to start: &lt;a href="https://contributors.gitlab.com" rel="noopener noreferrer"&gt;contributors.gitlab.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>gitlab</category>
      <category>opensource</category>
      <category>ruby</category>
      <category>geo</category>
    </item>
    <item>
      <title>From a 500 to a real error: validating comment length across 9 GitLab GraphQL mutations</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Fri, 13 Nov 2026 09:52:42 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/from-a-500-to-a-real-error-validating-comment-length-across-9-gitlab-graphql-mutations-60d</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/from-a-500-to-a-real-error-validating-comment-length-across-9-gitlab-graphql-mutations-60d</guid>
      <description>&lt;p&gt;Here's a bug you've probably shipped yourself. The model has a validation, the API docs mention the limit, and the API never checks it. Everything works until someone sends input that's too long, and then the user gets "Internal server error" instead of a message they can act on.&lt;/p&gt;

&lt;p&gt;This is the story of &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/269616" rel="noopener noreferrer"&gt;gitlab-org/gitlab!269616&lt;/a&gt;. It added a length check to the &lt;code&gt;comment&lt;/code&gt; argument of nine GraphQL mutations in GitLab's vulnerability management, and it went through four rounds of CI fixes and review. The MR touched 20 files, most of them specs and generated docs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The issue: a documented limit nobody enforced
&lt;/h2&gt;

&lt;p&gt;The issue was &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/issues/637728" rel="noopener noreferrer"&gt;gitlab-org/gitlab#637728&lt;/a&gt;, opened by Lorenz van Herwaarden. It started with a GitLab Duo review comment on an earlier MR. Here's the gist:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Vulnerabilities::StateTransition&lt;/code&gt; has &lt;code&gt;validates :comment, length: { maximum: 50_000 }&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Several GraphQL mutations say "(maximum 50,000 characters)" in their &lt;code&gt;comment&lt;/code&gt; description, but don't enforce it.&lt;/li&gt;
&lt;li&gt;A longer comment reaches &lt;code&gt;Vulnerabilities::StateTransition.create!&lt;/code&gt;, which raises &lt;code&gt;ActiveRecord::RecordInvalid&lt;/code&gt;. Nothing rescues it, so the user gets a generic "Internal server error".&lt;/li&gt;
&lt;li&gt;Bulk dismiss is worse. It uses &lt;code&gt;StateTransition.insert_all!&lt;/code&gt;, which &lt;strong&gt;skips model validations&lt;/strong&gt;, so an oversized comment only fails at the database check constraint (&lt;code&gt;char_length(comment) &amp;lt;= 50000&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The issue also explained why rescuing the error in the services was the wrong fix:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;A rescue would run after &lt;code&gt;CreateService&lt;/code&gt; may already have saved the vulnerability.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;So the fix had to happen &lt;em&gt;before&lt;/em&gt; anything is written: at the GraphQL argument.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr30wwsqoer8x9jou52tf.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr30wwsqoer8x9jou52tf.png" alt="What happened to an oversized comment, before and after" width="800" height="698"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The plan
&lt;/h2&gt;

&lt;p&gt;I asked to work on it with this plan:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;My plan is to add &lt;code&gt;validates: { length: { maximum: 50_000 } }&lt;/code&gt; to the &lt;code&gt;comment&lt;/code&gt; arguments on the security-finding and vulnerability state-transition mutations (including the shared &lt;code&gt;VulnerabilityStateTransitions&lt;/code&gt; concern), following the existing GraphQL argument pattern, then add a spec per mutation for a comment over 50,000 characters so the error is returned before anything is written.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Lorenz assigned me and improved the scope right away. The issue had missed two mutations that also take a &lt;code&gt;comment&lt;/code&gt;, &lt;code&gt;securityFindingRevertToDetected&lt;/code&gt; and &lt;code&gt;vulnerabilitiesDismiss&lt;/code&gt;, so Lorenz added them to the description. Lorenz also asked me to include Subashis Chakraborty from the team as a reviewer.&lt;/p&gt;

&lt;p&gt;The final list was nine mutations:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;vulnerabilityConfirm&lt;/code&gt;, &lt;code&gt;vulnerabilityResolve&lt;/code&gt;, &lt;code&gt;vulnerabilityDismiss&lt;/code&gt;, &lt;code&gt;vulnerabilityRevertToDetected&lt;/code&gt; (through one shared concern)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;securityFindingConfirm&lt;/code&gt;, &lt;code&gt;securityFindingResolve&lt;/code&gt;, &lt;code&gt;securityFindingDismiss&lt;/code&gt;, &lt;code&gt;securityFindingRevertToDetected&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;vulnerabilitiesDismiss&lt;/code&gt; (bulk)&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The fix: validate at the boundary
&lt;/h2&gt;

&lt;p&gt;GraphQL-Ruby can validate an argument before the resolver runs. GitLab already uses this in other mutations, so the change is one line per argument. Here's the shared concern that covers four of the mutations:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;argument&lt;/span&gt; &lt;span class="ss"&gt;:comment&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="nc"&gt;GraphQL&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Types&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;validates&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;maximum&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Vulnerabilities&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;StateTransition&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="no"&gt;COMMENT_MAX_LENGTH&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Comment why vulnerability was #{state_transition_name_past_tense} (maximum 50,000 characters)."&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the comment is too long, GraphQL returns a validation error and the resolver never runs. No vulnerability, no state transition, no half-written data. This also covers bulk dismiss, because the check runs before &lt;code&gt;insert_all!&lt;/code&gt; is ever reached.&lt;/p&gt;

&lt;p&gt;Each mutation got a request spec that proves two things: the error comes back, and nothing is written. Here's the bulk dismiss one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;context&lt;/span&gt; &lt;span class="s2"&gt;"when the comment exceeds the maximum length"&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="nf"&gt;let&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:comment&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;'a'&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Vulnerabilities&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;StateTransition&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="no"&gt;COMMENT_MAX_LENGTH&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s2"&gt;"returns a validation error and does not dismiss the vulnerabilities"&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
    &lt;span class="n"&gt;expect&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nf"&gt;post_graphql_mutation&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mutation&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;current_user&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;current_user&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
      &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;not_to&lt;/span&gt; &lt;span class="n"&gt;change&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nc"&gt;Vulnerabilities&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;StateTransition&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

    &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;graphql_errors&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="kd"&gt;include&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;a_hash_including&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'message'&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="kd"&gt;include&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'too long'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vulnerability_1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;not_to&lt;/span&gt; &lt;span class="n"&gt;be_dismissed&lt;/span&gt;
    &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vulnerability_2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;not_to&lt;/span&gt; &lt;span class="n"&gt;be_dismissed&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;not_to change { ... count }&lt;/code&gt; block matters most. Returning an error isn't enough; the spec has to prove that nothing was written.&lt;/p&gt;

&lt;h2&gt;
  
  
  Round 1: my own CI mistakes
&lt;/h2&gt;

&lt;p&gt;The first pipeline failed two jobs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;rubocop&lt;/code&gt;&lt;/strong&gt;: I'd left extra blank lines at the end of four request specs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;graphql-verify&lt;/code&gt;&lt;/strong&gt;: I'd changed the &lt;code&gt;securityFindingRevertToDetected&lt;/code&gt; description, but not the committed introspection schemas (&lt;code&gt;public/-/graphql/introspection_result.json&lt;/code&gt; and its &lt;code&gt;_no_deprecated&lt;/code&gt; sibling).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That second one is worth knowing about. In GitLab, a GraphQL &lt;strong&gt;description&lt;/strong&gt; is part of the public schema. Changing it means regenerating the GraphQL API docs &lt;em&gt;and&lt;/em&gt; the introspection JSON files, and CI checks that they match. I fixed both in &lt;code&gt;74bc19de&lt;/code&gt;, and the next pipeline passed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Round 2: docs review
&lt;/h2&gt;

&lt;p&gt;Brendan Lynch added Isaac Durham as a reviewer for the vulnerability management group. Isaac reviewed from the docs side and suggested adding the limit to the &lt;code&gt;dismiss&lt;/code&gt; and &lt;code&gt;revert_to_detected&lt;/code&gt; descriptions, to match the other &lt;code&gt;comment&lt;/code&gt; arguments. Isaac also reminded me that I'd need to rerun &lt;code&gt;bundle exec rake gitlab:graphql:docs:compile&lt;/code&gt; afterwards.&lt;/p&gt;

&lt;p&gt;I applied both suggestions in &lt;code&gt;06b168dd&lt;/code&gt;, with regenerated docs and introspection JSON. The pipeline failed again, on &lt;code&gt;rubocop&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ee/app/graphql/mutations/security/finding/revert_to_detected.rb:25:121: C: Layout/LineLength: Line is too long. [122/120]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The new description string was two characters too long for one line.&lt;/p&gt;

&lt;h2&gt;
  
  
  Round 3: backend review, four suggestions
&lt;/h2&gt;

&lt;p&gt;Subashis then did a thorough review. It opened by summarizing why the change was safe:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The change is minimal and non breaking (it enforces the already-documented model limit at the GraphQL boundary)…&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Then came four suggestions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Magic number duplication.&lt;/strong&gt; &lt;code&gt;50_000&lt;/code&gt; appeared as a literal in 7 places (the model and 6 GraphQL arguments). Extract &lt;code&gt;COMMENT_MAX_LENGTH&lt;/code&gt; on the model so the model and the API can't drift apart.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A failure-only spec.&lt;/strong&gt; The new &lt;code&gt;securityFindingDismiss&lt;/code&gt; request spec only covered the over-limit case. Add a happy path, or trim the unused fields.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Description style.&lt;/strong&gt; &lt;code&gt;confirm.rb&lt;/code&gt; and &lt;code&gt;resolve.rb&lt;/code&gt; used "(maximum 50,000 characters)", while my updated &lt;code&gt;dismiss.rb&lt;/code&gt; and &lt;code&gt;revert_to_detected.rb&lt;/code&gt; used ". Maximum 50,000 characters." Pick one form.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Loose error matching.&lt;/strong&gt; &lt;code&gt;expect(graphql_errors.to_s).to include('comment is too long')&lt;/code&gt; turns the whole errors array into a string. Matching the message directly gives a more readable failure.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I applied all four in &lt;code&gt;2fd1e423&lt;/code&gt;. The constant lives on the model, and both the model validation and the API use it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="no"&gt;COMMENT_MAX_LENGTH&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;50_000&lt;/span&gt;

&lt;span class="c1"&gt;# ...&lt;/span&gt;
&lt;span class="n"&gt;validates&lt;/span&gt; &lt;span class="ss"&gt;:comment&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;length&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;maximum&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="no"&gt;COMMENT_MAX_LENGTH&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Suggestion 3 is interesting because it went the other way from Round 2. Isaac's suggestion used the sentence form; Subashis suggested the parenthetical form, which the shared concern and two other mutations already used. I went with the parenthetical form everywhere, so the schema reads the same across all nine mutations. That meant regenerating the docs and introspection JSON once more, and it also fixed the &lt;code&gt;Layout/LineLength&lt;/code&gt; offense from Round 2.&lt;/p&gt;

&lt;p&gt;Then CI failed one more time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Layout/LineEndStringConcatenationIndentation
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When I split the long description into two concatenated strings, the continuation line wasn't indented the way RuboCop wanted. One more small commit (&lt;code&gt;345bc6ff&lt;/code&gt;) fixed it, and the pipeline went green.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo8hsyttte1ndfyiy2ph3.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fo8hsyttte1ndfyiy2ph3.png" alt="Four rounds of CI and review" width="800" height="503"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Round 4: the Danger warning, and the merge
&lt;/h2&gt;

&lt;p&gt;After the constant extraction, Danger left a note on the model file:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Did you consider new validations can break existing records?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;It's a reasonable thing for a bot to ask, because the diff touches a &lt;code&gt;validates&lt;/code&gt; line. But it's not a new validation. I replied on the thread:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This isn't a new model validation: &lt;code&gt;Vulnerabilities::StateTransition&lt;/code&gt; already had &lt;code&gt;validates :comment, length: { maximum: 50_000 }&lt;/code&gt;. This MR only moves the literal into &lt;code&gt;COMMENT_MAX_LENGTH&lt;/code&gt; so the GraphQL &lt;code&gt;comment&lt;/code&gt; arguments can reuse the same limit. Existing records already satisfy it, so nothing can break.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Subashis confirmed it:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;It's a false positive.&lt;/strong&gt; The MR doesn't add a new model validation&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Subashis also pointed out that the database check constraint enforces the limit on its own, and that the genuinely new validations in this MR are GraphQL argument validators, which "only run on incoming API input, never against stored records."&lt;/p&gt;

&lt;p&gt;Isaac and Subashis both approved, and Subashis merged the MR. In total there were seven pipelines: three failed on my RuboCop and &lt;code&gt;graphql-verify&lt;/code&gt; mistakes, and the rest passed.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Validate at the boundary, before side effects.&lt;/strong&gt; If a service can write data before an error is raised, rescuing the error is too late. Reject bad input where it enters the system.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Watch out for &lt;code&gt;insert_all!&lt;/code&gt;.&lt;/strong&gt; It skips model validations. If a code path uses it, model validations can't protect that path, and the check has to happen somewhere else.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. In GitLab, GraphQL descriptions are schema.&lt;/strong&gt; Changing a description means regenerating &lt;code&gt;doc/api/graphql/reference&lt;/code&gt; and the introspection JSON. &lt;code&gt;graphql-verify&lt;/code&gt; will catch you if you forget.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Run RuboCop on the exact files you touched before pushing.&lt;/strong&gt; Three of my seven pipelines failed on RuboCop or generated files, all avoidable. Each extra pipeline costs reviewers time and CI minutes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. When suggestions conflict, choose consistency.&lt;/strong&gt; I went with the form the shared concern already used, which kept all nine descriptions the same.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Answer bot warnings with facts.&lt;/strong&gt; The Danger note was a fair question. A short, specific reply let the maintainer confirm it was a false positive quickly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Let maintainers improve the scope.&lt;/strong&gt; Lorenz added two mutations the issue had missed. A plan comment before writing code is what made that possible.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/issues/637728" rel="noopener noreferrer"&gt;gitlab-org/gitlab#637728&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Merge request: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/269616" rel="noopener noreferrer"&gt;gitlab-org/gitlab!269616&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The review comment that started it: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253678" rel="noopener noreferrer"&gt;gitlab-org/gitlab!253678&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks to Lorenz van Herwaarden for the issue and the extra scope, Subashis Chakraborty for a detailed and kind review, Isaac Durham for the docs review, and Brendan Lynch for routing it to the right people.&lt;/p&gt;

</description>
      <category>gitlab</category>
      <category>opensource</category>
      <category>graphql</category>
      <category>ruby</category>
    </item>
    <item>
      <title>Two queries too many: removing each_batch from GitLab's Maven snapshot cleanup</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Fri, 13 Nov 2026 09:52:23 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/two-queries-too-many-removing-eachbatch-from-gitlabs-maven-snapshot-cleanup-508k</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/two-queries-too-many-removing-eachbatch-from-gitlabs-maven-snapshot-cleanup-508k</guid>
      <description>&lt;p&gt;The final diff of &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/270142" rel="noopener noreferrer"&gt;gitlab-org/gitlab!270142&lt;/a&gt; is tiny: a handful of lines in the service and a few more in its spec. But getting it merged took a database review with real production query plans, two spec improvements from a reviewer who tested them by breaking the code on purpose, and a Danger failure I caused myself.&lt;/p&gt;

&lt;p&gt;This post covers what the issue was, why &lt;code&gt;each_batch&lt;/code&gt; was the wrong tool here, how the specs got stronger in review, and what database review looks like when you're a community contributor without access to GitLab's production database tools.&lt;/p&gt;

&lt;h2&gt;
  
  
  The issue: a batch loop over something already batched
&lt;/h2&gt;

&lt;p&gt;The issue was &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/issues/639257" rel="noopener noreferrer"&gt;gitlab-org/gitlab#639257&lt;/a&gt;, opened by Tim Rizzi. It came from a suggestion by Daniyal Arshad in the review of an earlier MR (&lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/266409" rel="noopener noreferrer"&gt;!266409&lt;/a&gt;), which added a Maven snapshot pass to the package cleanup policy.&lt;/p&gt;

&lt;p&gt;Some background. &lt;code&gt;Packages::Cleanup::ExecutePolicyService&lt;/code&gt; runs a project's package cleanup policy. For Maven snapshot packages, it works out which old snapshot builds to remove and marks their files as &lt;code&gt;pending_destruction&lt;/code&gt;. A background job deletes them later.&lt;/p&gt;

&lt;p&gt;The marking happened in slices of up to 200 ids. Each slice was handed to another service:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;old_ids_by_name&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flatten&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;each_slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;MARK_PACKAGE_FILES_FOR_DESTRUCTION_SERVICE_BATCH_SIZE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
  &lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Packages&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;MarkPackageFilesForDestructionService&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Packages&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;PackageFile&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;id_in&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch_deadline&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;batch_deadline&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;batch_size&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="no"&gt;MARK_PACKAGE_FILES_FOR_DESTRUCTION_SERVICE_BATCH_SIZE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="vi"&gt;@counts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="ss"&gt;:marked_package_files_total_count&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="ss"&gt;:marked_package_files_count&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

  &lt;span class="k"&gt;break&lt;/span&gt; &lt;span class="ss"&gt;:timeout&lt;/span&gt; &lt;span class="k"&gt;unless&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;success?&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And here's the core of that service:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="vi"&gt;@package_files&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;each_batch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;of&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;min_batch_size&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;batched_package_files&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;batch_deadline&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;past?&lt;/span&gt;
    &lt;span class="n"&gt;timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="k"&gt;break&lt;/span&gt;
  &lt;span class="k"&gt;end&lt;/span&gt;

  &lt;span class="n"&gt;updates_count&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="n"&gt;batched_package_files&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update_all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="ss"&gt;:pending_destruction&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;each_batch&lt;/code&gt; is GitLab's standard way to walk a large relation safely. It finds a start id, then a boundary id, then yields a batch. That's exactly right when you don't know how big the relation is.&lt;/p&gt;

&lt;p&gt;Here, though, the relation is &lt;code&gt;PackageFile.id_in(ids)&lt;/code&gt; with at most 200 ids, and the batch size is 200. There is only ever &lt;strong&gt;one batch&lt;/strong&gt;. So each slice paid for two lookups whose only job was to find out what we already knew.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4h90xt9bei6sk9crexy5.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4h90xt9bei6sk9crexy5.png" alt="Statements per slice, before and after" width="800" height="274"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The plan
&lt;/h2&gt;

&lt;p&gt;I commented on the issue with this plan:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;My plan is to replace &lt;code&gt;each_batch&lt;/code&gt; over the one-batch snapshot relation with &lt;code&gt;each_slice&lt;/code&gt; plus &lt;code&gt;Packages::PackageFile.id_in(ids).update_all(status: :pending_destruction)&lt;/code&gt;, matching &lt;code&gt;mark_packages_for_destruction&lt;/code&gt;, and to stub &lt;code&gt;batch_deadline&lt;/code&gt; in the snapshot timeout spec.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;code&gt;mark_packages_for_destruction&lt;/code&gt; is a method in the same service that already slices ids and updates them directly. Matching an existing pattern in the same file is the easiest kind of change to review. Tim replied that the plan "sounds right", assigned me, and asked me to tag Tim and Fiona McCawley on the MR.&lt;/p&gt;

&lt;h2&gt;
  
  
  The change
&lt;/h2&gt;

&lt;p&gt;The production change is the loop body, now one statement per slice:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;old_ids_by_name&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;values&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flatten&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;each_slice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;MARK_PACKAGE_FILES_FOR_DESTRUCTION_SERVICE_BATCH_SIZE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
  &lt;span class="k"&gt;break&lt;/span&gt; &lt;span class="ss"&gt;:timeout&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;batch_deadline&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;past?&lt;/span&gt;

  &lt;span class="vi"&gt;@counts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="ss"&gt;:marked_package_files_total_count&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt;
    &lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Packages&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;PackageFile&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;id_in&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update_all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="ss"&gt;:pending_destruction&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two behaviors had to stay the same:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The deadline.&lt;/strong&gt; The old service checked &lt;code&gt;batch_deadline&lt;/code&gt; before each batch. The new loop checks it before each slice, so a long cleanup still stops on time and reports &lt;code&gt;:timeout&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The count.&lt;/strong&gt; &lt;code&gt;update_all&lt;/code&gt; returns the number of updated rows, so &lt;code&gt;marked_package_files_total_count&lt;/code&gt; still adds up the same way.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The spec needed a change too. The old timeout example stubbed &lt;code&gt;MarkPackageFilesForDestructionService&lt;/code&gt; to return an error, and that service isn't called anymore.&lt;/p&gt;

&lt;h2&gt;
  
  
  CI lesson: Danger fails on long commit subjects
&lt;/h2&gt;

&lt;p&gt;My first push failed the &lt;code&gt;danger-review&lt;/code&gt; job, and it was entirely my fault. GitLab's Danger rules require commit subjects of &lt;strong&gt;72 characters or fewer&lt;/strong&gt;, and mine was longer. I reworded it to &lt;code&gt;Use update_all for Maven snapshot cleanup slices&lt;/code&gt; and pushed again. The next pipeline passed.&lt;/p&gt;

&lt;p&gt;The same Danger run also left a note on the &lt;code&gt;update_all&lt;/code&gt; line:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;When using any of these: … Bulk commands: &lt;code&gt;bulk_insert&lt;/code&gt;, &lt;code&gt;update_all&lt;/code&gt;, etc. … You must:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Include the &lt;strong&gt;full database query&lt;/strong&gt; in the merge request description (including scopes, pagination, and limits)&lt;/li&gt;
&lt;li&gt;Include the query execution plan&lt;/li&gt;
&lt;li&gt;Request a ~database review&lt;/li&gt;
&lt;/ol&gt;
&lt;/blockquote&gt;

&lt;p&gt;Adding the SQL was easy: a primary-key &lt;code&gt;UPDATE&lt;/code&gt; on &lt;code&gt;packages_package_files&lt;/code&gt; with &lt;code&gt;WHERE id IN (...)&lt;/code&gt; for up to 200 ids. The query plan was harder.&lt;/p&gt;

&lt;h2&gt;
  
  
  Database review without database access
&lt;/h2&gt;

&lt;p&gt;GitLab's query plans come from Database Lab, which runs queries against a clone of production. It needs GitLab team access, and as a community contributor I don't have it. I put the full SQL in the description, explained that it's the same statement &lt;code&gt;mark_packages_for_destruction&lt;/code&gt; already runs, and asked for a database review.&lt;/p&gt;

&lt;p&gt;Fiona stepped in:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Database Lab needs GitLab team access, so I've captured the plans: the old and new SQL, each followed by its query plan, are now in the &lt;strong&gt;Database query&lt;/strong&gt; section of the description.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Fiona went further than I expected and picked a &lt;strong&gt;real slice&lt;/strong&gt; from production data: the first 200 ids this pass would mark for one snapshot package in &lt;code&gt;gitlab-org/editor-extensions/gitlab-eclipse-plugin&lt;/code&gt;. That package has 601 installable files, and with &lt;code&gt;keep_n_duplicated_package_files: 1&lt;/code&gt; the pass marks 442 of them in slices of 200, 200 and 42.&lt;/p&gt;

&lt;p&gt;Fiona's table, summarized:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Statement&lt;/th&gt;
&lt;th&gt;Plan&lt;/th&gt;
&lt;th&gt;Time (cold clone)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;After&lt;/td&gt;
&lt;td&gt;&lt;code&gt;UPDATE ... WHERE id IN (...)&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Index Scan on the primary key, 200 rows&lt;/td&gt;
&lt;td&gt;968 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Before&lt;/td&gt;
&lt;td&gt;start &lt;code&gt;SELECT ... LIMIT 1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Index Only Scan on the primary key&lt;/td&gt;
&lt;td&gt;6 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;boundary &lt;code&gt;SELECT ... LIMIT 1 OFFSET 200&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Index Only Scan on the primary key&lt;/td&gt;
&lt;td&gt;92 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;UPDATE ... AND id &amp;gt;= &amp;lt;start&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Index Scan on the primary key, 200 rows&lt;/td&gt;
&lt;td&gt;596 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The headline: &lt;strong&gt;the &lt;code&gt;UPDATE&lt;/code&gt; plan is unchanged&lt;/strong&gt;, and the two lookups disappear. The &lt;code&gt;UPDATE&lt;/code&gt; times differ between runs mostly because of cold I/O on the clone. Fiona noted that a second run of the new &lt;code&gt;UPDATE&lt;/code&gt; on a warmed clone took 760 ms with the same plan, and every &lt;code&gt;UPDATE&lt;/code&gt; run touched about the same number of buffers.&lt;/p&gt;

&lt;p&gt;This is the part of contributing that doesn't show up in the diff. The reviewer filled a gap I couldn't fill, and did it with real data instead of a guess.&lt;/p&gt;

&lt;h2&gt;
  
  
  The review that made the specs stronger
&lt;/h2&gt;

&lt;p&gt;Fiona also left two spec suggestions. Both came with a "Why" section explaining how Fiona had tested them. I think this is the best kind of review feedback.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Guard the listing deadline check
&lt;/h3&gt;

&lt;p&gt;The service checks the deadline in two places: while &lt;em&gt;listing&lt;/em&gt; snapshot files, and before each &lt;em&gt;update slice&lt;/em&gt;. My spec for "deadline already passed" asserted that no file changed and that &lt;code&gt;timeout&lt;/code&gt; was true. Fiona found a hole:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I deleted &lt;code&gt;break :timeout if batch_deadline.past?&lt;/code&gt; from the listing loop (&lt;code&gt;execute_policy_service.rb:184&lt;/code&gt;) and ran this example locally: it still passed.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Without the listing check, the listing runs to the end, then the first slice's deadline check stops the marking, and no file changes. My assertions couldn't tell the two code paths apart. The suggested fix was one line on a counter that only grows &lt;em&gt;after&lt;/em&gt; listing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="ss"&gt;:counts&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="ss"&gt;:unique_package_id_and_file_name_total_count&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With this line, the example fails if the listing check is removed.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Don't rely on counting calls
&lt;/h3&gt;

&lt;p&gt;My "timeout between slices" spec stubbed the deadline with a sequence: &lt;code&gt;and_return(false, false, true)&lt;/code&gt;. Fiona pointed out that the sequence is shared by &lt;strong&gt;every&lt;/strong&gt; &lt;code&gt;batch_deadline.past?&lt;/code&gt; call in &lt;code&gt;execute&lt;/code&gt;, including other passes. So a new check anywhere earlier would shift it and break the test for the wrong reason.&lt;/p&gt;

&lt;p&gt;The suggestion flips the deadline right after the first update, by wrapping &lt;code&gt;PackageFile.id_in&lt;/code&gt;. That's the same approach an existing example uses for the generic rule:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="nf"&gt;allow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Packages&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;PackageFile&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;receive&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:id_in&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;and_wrap_original&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;
  &lt;span class="nf"&gt;allow&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;service&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;receive&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:batch_deadline&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;and_return&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;second&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ago&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="n"&gt;method&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With the batch size stubbed to 2, the example now asserts that exactly two files are marked before the timeout:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s1"&gt;'stops after the current batch and reports a timeout'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="n"&gt;expect&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;execute&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="n"&gt;change&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;installable_files&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;by&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="ss"&gt;:timeout&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;be&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="ss"&gt;:counts&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="ss"&gt;:marked_package_files_total_count&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="nf"&gt;eq&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I applied both in one commit (&lt;code&gt;028c6d2f&lt;/code&gt;), specs only, and credited Fiona in the commit message. Fiona then approved the backend side and asked Jean van der Walt for the database review.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiqx00ivtj6g3f32yz23p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fiqx00ivtj6g3f32yz23p.png" alt="From issue to merge" width="800" height="468"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Database approval, and a follow-up idea
&lt;/h2&gt;

&lt;p&gt;Jean reviewed the database side:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Database looks good: the &lt;code&gt;UPDATE&lt;/code&gt; plan is unchanged, and each slice now skips the two &lt;code&gt;each_batch&lt;/code&gt; lookups.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Jean also left a non-blocking note: the 200-row &lt;code&gt;UPDATE&lt;/code&gt; takes about 600–960 ms on a cold clone, and a follow-up could try a smaller &lt;code&gt;MARK_PACKAGE_FILES_FOR_DESTRUCTION_SERVICE_BATCH_SIZE&lt;/code&gt;. I agreed it's worth trying and kept it out of this MR. Changing the batch size changes the risk profile, and it deserves its own measurement.&lt;/p&gt;

&lt;p&gt;Abdul Wadood gave the database maintainer approval ("~database LGTM! 🚀"), and Abdul merged the MR. Five pipelines in total: one failure (my commit subject) and four green.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. &lt;code&gt;each_batch&lt;/code&gt; is for unknown sizes.&lt;/strong&gt; If you've already sliced the ids, a direct &lt;code&gt;update_all&lt;/code&gt; per slice is cheaper and easier to read. Look for an existing pattern in the same file before inventing one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Keep deadline semantics when you refactor a loop.&lt;/strong&gt; Check the deadline at the same granularity as before, and keep the counters adding up the same way.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Test that your test can fail.&lt;/strong&gt; Fiona's method was simple: delete the line the spec claims to protect and see if the spec still passes. If it does, the spec isn't guarding anything.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Prefer event-based stubs over call-count sequences.&lt;/strong&gt; &lt;code&gt;and_return(false, false, true)&lt;/code&gt; breaks when someone adds an unrelated check. Flipping the stub when the interesting thing happens doesn't.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Community contributors can still do database work.&lt;/strong&gt; Put the full SQL in the description, explain why it's safe, and ask. Reviewers with access can capture the plans. Just make their job easy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. Keep commit subjects within 72 characters.&lt;/strong&gt; Danger will fail the pipeline if you don't.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/issues/639257" rel="noopener noreferrer"&gt;gitlab-org/gitlab#639257&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Merge request: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/270142" rel="noopener noreferrer"&gt;gitlab-org/gitlab!270142&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Where the idea came from: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/266409" rel="noopener noreferrer"&gt;gitlab-org/gitlab!266409&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;GitLab's guide to preparing bulk updates for database review: &lt;a href="https://docs.gitlab.com/development/database_review/#preparation-when-using-bulk-update-operations" rel="noopener noreferrer"&gt;docs.gitlab.com/development/database_review&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks to Tim Rizzi for the issue, Daniyal Arshad for the original idea, Fiona McCawley for the plans and the spec reviews, Jean van der Walt for the database review, and Abdul Wadood for the final approval.&lt;/p&gt;

</description>
      <category>gitlab</category>
      <category>opensource</category>
      <category>ruby</category>
      <category>postgres</category>
    </item>
    <item>
      <title>The duplicate object that cost a query: fixing an N+1 in GitLab's labels preloader</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Fri, 13 Nov 2026 09:52:04 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/the-duplicate-object-that-cost-a-query-fixing-an-n1-in-gitlabs-labels-preloader-2c58</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/the-duplicate-object-that-cost-a-query-fixing-an-n1-in-gitlabs-labels-preloader-2c58</guid>
      <description>&lt;p&gt;Sometimes a performance bug is one missing line. Sometimes the fix for it has a second bug hiding inside, one only a careful reviewer would spot. This merge request had both.&lt;/p&gt;

&lt;p&gt;This is the story of &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/270160" rel="noopener noreferrer"&gt;gitlab-org/gitlab!270160&lt;/a&gt;, which fixed an N+1 query in GitLab's GraphQL group labels. I'll cover what the issue was, the plan I proposed, the first fix, the review comment that changed it, and what I'd tell anyone starting out with GitLab contributions.&lt;/p&gt;

&lt;h2&gt;
  
  
  The issue: two extra queries for every subgroup
&lt;/h2&gt;

&lt;p&gt;The issue was &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/issues/635535" rel="noopener noreferrer"&gt;gitlab-org/gitlab#635535&lt;/a&gt;, opened by Terri Chu. It was one of the best-written bug reports I've picked up. In short:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;Group.labels&lt;/code&gt; with &lt;code&gt;includeDescendantGroups: true&lt;/code&gt; ran &lt;strong&gt;2 queries per subgroup that owns a returned label&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;The queries came from &lt;code&gt;GroupPolicy&lt;/code&gt; conditions that read each label group's &lt;strong&gt;root ancestor&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;EE::Preloaders::LabelsPreloader&lt;/code&gt; already preloaded root ancestors for &lt;em&gt;project&lt;/em&gt; labels' groups, but not for &lt;em&gt;group&lt;/em&gt; labels' groups.&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;search_labels&lt;/code&gt; MCP tool always sets &lt;code&gt;includeDescendantGroups: true&lt;/code&gt;, so it hit this on every group call.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The issue even named the two queries and where they come from:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Query&lt;/th&gt;
&lt;th&gt;Condition&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;root ancestor lookup (&lt;code&gt;find_namespaces_by_id&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;user_banned_from_namespace&lt;/code&gt; → &lt;code&gt;@subject.root_ancestor&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;saml_providers&lt;/code&gt; (cached)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;needs_new_sso_session&lt;/code&gt; → &lt;code&gt;SsoEnforcer.access_restricted?&lt;/code&gt; → &lt;code&gt;root_saml_provider&lt;/code&gt; → &lt;code&gt;root_ancestor.saml_provider&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There was also an existing spec that &lt;em&gt;documented&lt;/em&gt; the problem instead of hiding it. The MCP &lt;code&gt;search_labels&lt;/code&gt; N+1 example carried a &lt;code&gt;threshold: 4&lt;/code&gt; allowance with a TODO pointing at the issue:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="c1"&gt;# TODO: https://gitlab.com/gitlab-org/gitlab/-/work_items/635535&lt;/span&gt;
&lt;span class="n"&gt;it_behaves_like&lt;/span&gt; &lt;span class="s1"&gt;'an MCP tool that avoids N+1 queries'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;threshold&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's a nice touch for a contributor. When the fix works, you can delete the allowance and the spec becomes the guard.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa2nj23txozib30q20s35.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa2nj23txozib30q20s35.png" alt="Where the extra queries came from, before the fix" width="800" height="1085"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The plan, before writing any code
&lt;/h2&gt;

&lt;p&gt;GitLab's community workflow asks you to comment on the issue with a plan and get assigned before you start. Here's what I wrote:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;My plan is to preload root ancestors for group labels' groups in &lt;code&gt;EE::Preloaders::LabelsPreloader&lt;/code&gt; (this is already done for project labels), add a QueryRecorder example on &lt;code&gt;Group.labels(include_descendant_groups: true)&lt;/code&gt;, and tighten the MCP &lt;code&gt;search_labels&lt;/code&gt; threshold.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Terri confirmed the plan and made the threshold part concrete: drop &lt;code&gt;threshold: 4&lt;/code&gt; completely, "so the example fails if the per-subgroup queries come back."&lt;/p&gt;

&lt;p&gt;So the plan had three parts:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Production change&lt;/strong&gt;: run the existing root-ancestor preload over group labels' groups too.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A new regression spec&lt;/strong&gt;: a &lt;code&gt;QueryRecorder&lt;/code&gt; example on the group labels GraphQL query.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Remove the allowance&lt;/strong&gt;: delete &lt;code&gt;threshold: 4&lt;/code&gt; so the MCP spec guards the fix.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Version 1: one combined list
&lt;/h2&gt;

&lt;p&gt;The preloader code was short. Before the fix, only project labels' groups got their root ancestors preloaded:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="c1"&gt;# preloading the root ancestors for the project labels for the authorizations checks&lt;/span&gt;
&lt;span class="n"&gt;project_groups&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;project_labels&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter_map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;:&lt;/span&gt;&lt;span class="n"&gt;project&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter_map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;:&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;project_groups&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;empty?&lt;/span&gt;

&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Namespaces&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Preloaders&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;GroupRootAncestorPreloader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;project_groups&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;My first commit (&lt;code&gt;ee096b74&lt;/code&gt;) followed the fix suggested in the issue: build one list of groups from both kinds of labels and preload it once.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="c1"&gt;# preloading the root ancestors for the authorizations checks&lt;/span&gt;
&lt;span class="n"&gt;groups&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;group_labels&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter_map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;:&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;project_labels&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter_map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;:&lt;/span&gt;&lt;span class="n"&gt;project&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter_map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;:&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;empty?&lt;/span&gt;

&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Namespaces&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Preloaders&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;GroupRootAncestorPreloader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;saml_provider&lt;/code&gt; preload that follows then ran over &lt;code&gt;groups.map(&amp;amp;:root_ancestor)&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;For the regression spec, I added an example to &lt;code&gt;ee/spec/requests/api/graphql/group/labels_query_spec.rb&lt;/code&gt;. It records a control run, adds one more private subgroup with its own label, and checks the query count doesn't grow:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s1"&gt;'does not add queries for additional subgroup labels'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="n"&gt;run_query&lt;/span&gt;
  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;graphql_errors&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="n"&gt;be_blank&lt;/span&gt;

  &lt;span class="n"&gt;control&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ActiveRecord&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;QueryRecorder&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;skip_cached&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;run_query&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="n"&gt;extra_group&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:group&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="ss"&gt;:private&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;parent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nf"&gt;create&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="ss"&gt;:group_label&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;extra_group&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;title&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'fff'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

  &lt;span class="n"&gt;expect&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;run_query&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;not_to&lt;/span&gt; &lt;span class="nf"&gt;exceed_all_query_limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;control&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;graphql_errors&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="n"&gt;be_blank&lt;/span&gt;
  &lt;span class="nf"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;graphql_data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'group'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'labels'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'nodes'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;pluck&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'title'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt; &lt;span class="kd"&gt;include&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'fff'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two details matter here:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;skip_cached: false&lt;/code&gt; with &lt;code&gt;exceed_all_query_limit&lt;/code&gt; counts &lt;strong&gt;cached queries too&lt;/strong&gt;. One of the two queries in the issue (&lt;code&gt;saml_providers&lt;/code&gt;) was cached, so a normal count could miss it.&lt;/li&gt;
&lt;li&gt;The last line checks that the new label really comes back. A query-count spec that passes because the data vanished doesn't prove anything.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I removed &lt;code&gt;threshold: 4&lt;/code&gt; from the MCP spec, opened the MR, and asked Terri for review. The pipeline passed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The review: "the preloader drops the second copy"
&lt;/h2&gt;

&lt;p&gt;Terri asked Alex Filatov for an initial review. Alex ran both validation steps locally and confirmed they passed. Alex also confirmed that with the production change reverted, both new examples fail, and then left this:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;suggestion:&lt;/strong&gt; A subgroup that owns both a group label and a project label still costs one extra query, because the root ancestor preloader drops the second copy of that group.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;The explanation was precise. &lt;code&gt;group_labels.filter_map(&amp;amp;:group)&lt;/code&gt; and &lt;code&gt;project_labels...filter_map(&amp;amp;:group)&lt;/code&gt; come from &lt;strong&gt;separate &lt;code&gt;Preloader&lt;/code&gt; calls&lt;/strong&gt;. So when one subgroup owns both a group label and a project with a label, that subgroup exists as &lt;strong&gt;two different Ruby objects&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Here's the start of the shared preloader:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;initialize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;namespaces&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;root_ancestor_preloads&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="vi"&gt;@namespaces&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;namespaces&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;uniq&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;compact&lt;/span&gt;
  &lt;span class="vi"&gt;@root_ancestor_preloads&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;root_ancestor_preloads&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;uniq&lt;/code&gt; uses ActiveRecord equality, and ActiveRecord compares records by id. So the two objects count as duplicates, and only the first one keeps its place in &lt;code&gt;@namespaces&lt;/code&gt;. Later, the preloader sets &lt;code&gt;namespace.root_ancestor = root_ancestor&lt;/code&gt; only on the objects in that list. The project-side copy never gets one, so the first time policy code asks for its root ancestor, it runs a query of its own.&lt;/p&gt;

&lt;p&gt;In other words, my version 1 fixed the group-label side and quietly broke something that used to work. Before my change, the project-side groups &lt;em&gt;were&lt;/em&gt; preloaded, because they were the only list.&lt;/p&gt;

&lt;p&gt;Alex also brought proof: a small spec that fails on version 1 (&lt;code&gt;Expected a maximum of 21 queries, got 23&lt;/code&gt;, where the extra query is &lt;code&gt;namespaces ... LIMIT 1&lt;/code&gt;) and passes with the proposed fix.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnmyq98w0ugpmvotyesxl.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnmyq98w0ugpmvotyesxl.png" alt="The duplicate object trap, and the per-list fix" width="800" height="1518"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Version 2: preload once per list
&lt;/h2&gt;

&lt;p&gt;The fix is to run the preloader once for each list, so every Ruby object that policy code will touch gets its own &lt;code&gt;root_ancestor&lt;/code&gt;. This is commit &lt;code&gt;26d2a3d3&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="c1"&gt;# Preload the root ancestors for the authorization checks. The two lists hold different&lt;/span&gt;
&lt;span class="c1"&gt;# instances of a shared group, and the preloader drops duplicates by id, so run it per list.&lt;/span&gt;
&lt;span class="n"&gt;group_lists&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;group_labels&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter_map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;:&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;project_labels&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter_map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;:&lt;/span&gt;&lt;span class="n"&gt;project&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter_map&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;:&lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reject&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;:&lt;/span&gt;&lt;span class="n"&gt;empty?&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;group_lists&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;empty?&lt;/span&gt;

&lt;span class="n"&gt;group_lists&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;each&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;groups&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Namespaces&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;Preloaders&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="nc"&gt;GroupRootAncestorPreloader&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execute&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;groups&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;group_lists&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flatten&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This costs one extra query that doesn't grow with the number of groups. A constant query beats a per-group query.&lt;/p&gt;

&lt;p&gt;Before applying it, I opened &lt;code&gt;NamespaceRootAncestorPreloader&lt;/code&gt; and confirmed the &lt;code&gt;namespaces.uniq&lt;/code&gt; behavior myself. A reviewer's suggestion is a strong signal, but I still want to understand &lt;em&gt;why&lt;/em&gt; before I push it under my name. I also added Alex's example as a new file, &lt;code&gt;ee/spec/models/ee/preloaders/labels_preloader_spec.rb&lt;/code&gt;, so this exact case stays covered:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ruby"&gt;&lt;code&gt;&lt;span class="n"&gt;it&lt;/span&gt; &lt;span class="s1"&gt;'does not query root ancestors per group that owns both group and project labels'&lt;/span&gt; &lt;span class="k"&gt;do&lt;/span&gt;
  &lt;span class="n"&gt;labels&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;create_group_with_both_labels&lt;/span&gt;
  &lt;span class="n"&gt;control&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;preload&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

  &lt;span class="n"&gt;labels&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nc"&gt;Array&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;create_group_with_both_labels&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;flatten&lt;/span&gt;

  &lt;span class="n"&gt;expect&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nf"&gt;preload&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;not_to&lt;/span&gt; &lt;span class="nf"&gt;exceed_query_limit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;control&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The commit message explains the root cause and credits the reviewer:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Preload root ancestors per label list in LabelsPreloader

Group labels and project labels are loaded by separate preloaders, so a
group that owns both kinds of labels is two Ruby objects.
GroupRootAncestorPreloader drops duplicates by id, which left the
project-side copy without a root ancestor and cost one query per such
group. Run the root ancestor preloader once per list instead.

Suggested by @alexfilatov.gitlab in review.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Alex re-checked the new version: 25 examples passing across the four related spec files, the new preloader spec failing as expected against the old code, and RuboCop clean. Alex then asked a maintainer to merge.&lt;/p&gt;

&lt;h2&gt;
  
  
  Merge
&lt;/h2&gt;

&lt;p&gt;Fred de Gier did the maintainer review. Beyond the specs, Fred checked that permissions behaved the same after the change:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I checked locally that the new examples fail without your change and pass with it, and that users with different access to the subgroups still see exactly the same labels as before, so this looks good to me.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's a check I want to remember. A preloading change in an authorization path should never change &lt;em&gt;what&lt;/em&gt; a user can see, only how many queries it takes to decide. The MR merged on the same day it got its first review. All four of its pipelines passed.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I took away
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. ActiveRecord equality is not object identity.&lt;/strong&gt; Two &lt;code&gt;Group&lt;/code&gt; objects with the same id are &lt;code&gt;==&lt;/code&gt;, so &lt;code&gt;uniq&lt;/code&gt; treats them as duplicates. But a value cached on one object (like &lt;code&gt;root_ancestor&lt;/code&gt;) does not appear on the other. When a preloader dedupes by id and writes to objects, every &lt;em&gt;copy&lt;/em&gt; you'll use later has to be in the list it writes to.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Count cached queries in N+1 specs.&lt;/strong&gt; &lt;code&gt;QueryRecorder.new(skip_cached: false)&lt;/code&gt; with &lt;code&gt;exceed_all_query_limit&lt;/code&gt; caught a query that a normal count could miss.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Delete the allowance; don't just loosen it.&lt;/strong&gt; Removing &lt;code&gt;threshold: 4&lt;/code&gt; turned a spec that documented a bug into one that guards against it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Credit reviewers in the commit.&lt;/strong&gt; "Suggested by … in review" in the commit body keeps the history honest and makes the next reader's job easier.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Stay assigned until the issue is closed.&lt;/strong&gt; After the merge I unassigned myself, because the MR had merged. Terri reassigned me and explained why: the issue should move with the MR (&lt;code&gt;In dev&lt;/code&gt;, &lt;code&gt;In review&lt;/code&gt;, then &lt;code&gt;Verification&lt;/code&gt; after the merge) and only close once the change is verified in production. I didn't know that, and it's useful for any GitLab contributor to hear.&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Issue: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/issues/635535" rel="noopener noreferrer"&gt;gitlab-org/gitlab#635535&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Merge request: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/270160" rel="noopener noreferrer"&gt;gitlab-org/gitlab!270160&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The spec that introduced &lt;code&gt;threshold: 4&lt;/code&gt;: &lt;a href="https://gitlab.com/gitlab-org/gitlab/-/merge_requests/264415" rel="noopener noreferrer"&gt;gitlab-org/gitlab!264415&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks to Terri Chu for a bug report that made the work easy to start, Alex Filatov for a review that made the fix correct, and Fred de Gier for the careful final check.&lt;/p&gt;

&lt;p&gt;If you've been thinking about contributing to GitLab, the community issues are a great place to start: &lt;a href="https://contributors.gitlab.com" rel="noopener noreferrer"&gt;contributors.gitlab.com&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>gitlab</category>
      <category>opensource</category>
      <category>rails</category>
      <category>performance</category>
    </item>
    <item>
      <title>Zwischenzug: your next Lichess game unlocks when your pulse settles</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Mon, 09 Nov 2026 17:05:40 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/zwischenzug-your-next-lichess-game-unlocks-when-your-pulse-settles-4d44</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/zwischenzug-your-next-lichess-game-unlocks-when-your-pulse-settles-4d44</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/presage-2026-11-04"&gt;Presage SmartSpectra SDK Challenge&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Signal&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Live pulse rate (Presage SmartSpectra, on-device)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Platform&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Electron + strict TypeScript · macOS (Apple Silicon) and Windows x64&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Chess&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Lichess, embedded in the app, via its public API&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Privacy&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Camera only between games, video never leaves the machine, telemetry off, one-click delete&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Code&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;a href="https://github.com/CodewithJha/zwischenzug" rel="noopener noreferrer"&gt;github.com/CodewithJha/zwischenzug&lt;/a&gt; (MIT)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The problem: tilt
&lt;/h3&gt;

&lt;p&gt;Online chess puts the next game one click away. After a painful loss, the impulsive "one more to win it back" is where losing streaks start.&lt;/p&gt;

&lt;p&gt;An analysis of &lt;strong&gt;11.3 million Lichess games&lt;/strong&gt; with player fixed effects (&lt;a href="https://github.com/romanbreyten/chess-tilt" rel="noopener noreferrer"&gt;romanbreyten/chess-tilt&lt;/a&gt;) found:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;After…&lt;/th&gt;
&lt;th&gt;Effect&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1 loss&lt;/td&gt;
&lt;td&gt;about &lt;strong&gt;7 rating points&lt;/strong&gt; below your usual level in the next game&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5 losses in a row&lt;/td&gt;
&lt;td&gt;about &lt;strong&gt;24 rating points&lt;/strong&gt; below&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;a 30-minute break&lt;/td&gt;
&lt;td&gt;the effect is gone&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5 straight losses (blitz)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;56%&lt;/strong&gt; of next games start &lt;strong&gt;within a minute&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The author notes the data can't separate emotional tilt from simply having an off day. Neither Lichess nor chess.com has a built-in cooldown. Existing extensions count losses and block play for a fixed time, whether or not you've actually calmed down.&lt;/p&gt;

&lt;h3&gt;
  
  
  The idea
&lt;/h3&gt;

&lt;p&gt;A &lt;em&gt;zwischenzug&lt;/em&gt; is the chess term for an "in-between move": the move you play before the recapture everyone expects. Zwischenzug puts an in-between move after a loss: &lt;strong&gt;a pause that ends when your body has settled, not when a fixed timer runs out.&lt;/strong&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  How a cooldown works
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzrr2pmgydxdxwahoccpu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fzrr2pmgydxdxwahoccpu.png" alt="Cooldown state machine" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Consent.&lt;/strong&gt; Choose "Allow camera between games" or "Timer only". Nothing runs before this.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One-minute check-in.&lt;/strong&gt; The webcam reads your pulse; the median of the stable readings becomes &lt;strong&gt;your&lt;/strong&gt; baseline.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Play.&lt;/strong&gt; Lichess runs inside the app. &lt;strong&gt;The camera is off while you play.&lt;/strong&gt; Games are only allowed while the embedded Lichess session is signed in as the account you chose, so a second account or anonymous play inside the app doesn't get around the lock.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Loss detected&lt;/strong&gt; through the Lichess public API within seconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Lock.&lt;/strong&gt; The Lichess view is hidden, and seek, challenge and rematch requests are cancelled at the network layer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Unlock&lt;/strong&gt;, whichever comes first:&lt;/li&gt;
&lt;/ol&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Path&lt;/th&gt;
&lt;th&gt;Rule&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Pulse&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;at or below baseline + max(5 bpm, 7%) for &lt;strong&gt;15 s&lt;/strong&gt;, after at least &lt;strong&gt;60 s&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Timer&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;no valid reading for &lt;strong&gt;180 s&lt;/strong&gt; (timer-only mode, no face, poor light, SDK error)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Override&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;type a confirmation sentence (logged as an override)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Cap&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;always unlocks &lt;strong&gt;15 minutes&lt;/strong&gt; after the loss&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Escalation.&lt;/strong&gt; After 5 losses in a row, it suggests a 30-minute break, the point at which the study saw the effect disappear.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/qnOiJMkBo1s" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One real session, start to finish:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;check-in (baseline &lt;strong&gt;77.5 bpm&lt;/strong&gt; from 59 readings)&lt;/li&gt;
&lt;li&gt;Lichess sign-in&lt;/li&gt;
&lt;li&gt;a real bullet loss&lt;/li&gt;
&lt;li&gt;lock, and live pulse against the baseline (&lt;strong&gt;79.4 bpm&lt;/strong&gt; right after the loss)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;unlocked by pulse after 64 s&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The face is blurred and sped-up parts are labelled on screen. The narration is AI-generated (Kokoro TTS); the footage and every number are real.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Check-in: live pulse&lt;/th&gt;
&lt;th&gt;Loss, then lock&lt;/th&gt;
&lt;th&gt;Unlocked by pulse&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fm9zr2ksszum6w0tgouin.gif" alt="Check-in" width="759" height="477"&gt;&lt;/td&gt;
&lt;td&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fjzz2v76cbt1ca6to47w1.gif" alt="Loss to lock" width="759" height="477"&gt;&lt;/td&gt;
&lt;td&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fegb8hwt56qbcwx471bct.gif" alt="Unlock by pulse" width="759" height="477"&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/CodewithJha" rel="noopener noreferrer"&gt;
        CodewithJha
      &lt;/a&gt; / &lt;a href="https://github.com/CodewithJha/zwischenzug" rel="noopener noreferrer"&gt;
        zwischenzug
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      Privacy-first chess tilt cooldown: after a Lichess loss, your next game unlocks when your pulse is back near baseline.
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div&gt;
&lt;a rel="noopener noreferrer" href="https://github.com/CodewithJha/zwischenzug/build/icon.png"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fraw.githubusercontent.com%2FCodewithJha%2Fzwischenzug%2FHEAD%2Fbuild%2Ficon.png" alt="Zwischenzug logo" width="128" height="128"&gt;&lt;/a&gt;
&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;Zwischenzug&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;After a Lichess loss, your next game unlocks when your pulse is back near your baseline.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;A privacy-first tilt cooldown for online chess: a desktop app for macOS (and Windows) built with Electron
TypeScript, the Presage SmartSpectra SDK and the Lichess open API.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://github.com/CodewithJha/zwischenzug/actions/workflows/ci.yml" rel="noopener noreferrer"&gt;&lt;img src="https://github.com/CodewithJha/zwischenzug/actions/workflows/ci.yml/badge.svg" alt="CI"&gt;&lt;/a&gt;
&lt;a href="https://github.com/CodewithJha/zwischenzug/LICENSE" rel="noopener noreferrer"&gt;&lt;img src="https://camo.githubusercontent.com/5e0abb27bc77af909f6e0c7cd00eee168f348ba6cbde1d3b1bcb5789c20477fd/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4d49542d626c61636b" alt="License: MIT"&gt;&lt;/a&gt;
&lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/7d8f1a101c48c517c9a637be22d45fd18c48589ee21d44293bb0653b7d3aee60/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f706c6174666f726d2d6d61634f5325323061726d363425323025374325323057696e646f77732532307836342d626c61636b"&gt;&lt;img src="https://camo.githubusercontent.com/7d8f1a101c48c517c9a637be22d45fd18c48589ee21d44293bb0653b7d3aee60/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f706c6174666f726d2d6d61634f5325323061726d363425323025374325323057696e646f77732532307836342d626c61636b" alt="Platform: macOS Apple Silicon | Windows x64"&gt;&lt;/a&gt;
&lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/cbad6cc3929b7bca3cc242ad9b458767582526c8183e4c402202a3e4cce509ad/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f456c656374726f6e2d34332d626c61636b"&gt;&lt;img src="https://camo.githubusercontent.com/cbad6cc3929b7bca3cc242ad9b458767582526c8183e4c402202a3e4cce509ad/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f456c656374726f6e2d34332d626c61636b" alt="Electron 43"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;You play Lichess inside Zwischenzug. When you lose, new games are locked. With your consent
the app reads your pulse from the webcam, on your own computer, and unlocks once it has stayed
near your personal baseline. No usable camera signal? A timer unlocks instead. You can always
skip a cooldown by typing a confirmation sentence.&lt;/p&gt;
&lt;p&gt;The camera is &lt;strong&gt;never on while you play&lt;/strong&gt;: only during a one-minute check-in and on the
cooldown screen after a loss.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;Zwischenzug is a habit tool, not a medical device. It does not diagnose, treat or monitor
any condition, and it does not claim to detect "tilt". It only…&lt;/p&gt;
&lt;/blockquote&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/CodewithJha/zwischenzug" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;&lt;strong&gt;Try it:&lt;/strong&gt; prebuilt installers are on the &lt;a href="https://github.com/CodewithJha/zwischenzug/releases/tag/v0.1.0" rel="noopener noreferrer"&gt;v0.1.0 release&lt;/a&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Platform&lt;/th&gt;
&lt;th&gt;File&lt;/th&gt;
&lt;th&gt;Status&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;macOS (Apple Silicon)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Zwischenzug-0.1.0-arm64.dmg&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Main platform, tested with a real webcam and real games&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Windows 10/11 (x64)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Zwischenzug-0.1.0-win-x64-setup.exe&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Built and launch-checked on GitHub's Windows runner (the app starts and the SDK loads); camera use on Windows not yet tested&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Both are unsigned (a solo hackathon project with no paid certificates); the README explains how to open them past Gatekeeper and SmartScreen.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;For judges:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You need a free Lichess account and a free Presage API key. There are no shared credentials: the key is tied to a personal free trial, and the Lichess login happens on lichess.org inside the app.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Live Presage signal in about a minute:&lt;/strong&gt; do the one-minute check-in, then open &lt;strong&gt;Settings › Advanced › Pulse diagnostics&lt;/strong&gt; for the raw readings.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Without a key:&lt;/strong&gt; choose "Timer only".&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Full steps are in the README's &lt;a href="https://github.com/CodewithJha/zwischenzug#how-judges-can-test" rel="noopener noreferrer"&gt;"How judges can test"&lt;/a&gt; section.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Engineering:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Electron with context isolation, sandbox and a strict CSP. The Lichess page runs in a separate, permission-less view with no preload.&lt;/li&gt;
&lt;li&gt;Hardened Electron fuses: no &lt;code&gt;ELECTRON_RUN_AS_NODE&lt;/code&gt;, no &lt;code&gt;--inspect&lt;/code&gt;, asar integrity on.&lt;/li&gt;
&lt;li&gt;The pure logic (engine, parsing, persistence) is in &lt;code&gt;src/core&lt;/code&gt;, with &lt;strong&gt;189 unit tests&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;CI runs on macOS, and a separate workflow builds and checks the Windows installer.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  How It Responds to Human Signals
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmw5cm0b5rog1figp3qcd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmw5cm0b5rog1figp3qcd.png" alt="How Zwischenzug senses pulse on device" width="800" height="525"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Signal used: pulse rate, live.&lt;/strong&gt; The pulse decides when you can play again; it isn't a decorative readout.&lt;/p&gt;

&lt;h3&gt;
  
  
  SDK integration
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fptpxxsaesufojkwzvc9p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fptpxxsaesufojkwzvc9p.png" alt="Architecture" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Setup:&lt;/strong&gt; &lt;code&gt;@smartspectra/node-sdk&lt;/code&gt; 3.4, integrated following Presage's Electron quickstart.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Where it runs:&lt;/strong&gt; the native runtime loads in the Electron &lt;strong&gt;main process&lt;/strong&gt; and is bound to the sandboxed app page through a narrow preload bridge. The app page captures the camera frames.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What it asks for:&lt;/strong&gt; Zwischenzug requests only &lt;strong&gt;&lt;code&gt;PULSE_RATE&lt;/code&gt;&lt;/strong&gt; and uses the SDK's &lt;strong&gt;stability and validation&lt;/strong&gt; signals:

&lt;ul&gt;
&lt;li&gt;unstable readings never settle a cooldown;&lt;/li&gt;
&lt;li&gt;a reading older than 5 s counts as no reading;&lt;/li&gt;
&lt;li&gt;values outside 30–220 bpm are ignored at the boundary to the main process.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Calibration:&lt;/strong&gt; the median of stable readings over 60 s, with at least 10 needed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hints:&lt;/strong&gt; SDK hints such as "too dark" or "no face in view" are shown to the user and switch the screen to the timer path. After a retryable failure, the measurement restarts (up to 5 times, 15 s apart), so the pulse path resumes when you're back in view.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The unlock rule, in code
&lt;/h3&gt;

&lt;p&gt;The band is relative to &lt;strong&gt;your&lt;/strong&gt; baseline, not a population number:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// src/core/cooldown.ts&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;upperBand&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;baselineBpm&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;CooldownRules&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="kr"&gt;number&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;baselineBpm&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;toleranceBpm&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;baselineBpm&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nx"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;tolerancePct&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With a 77.5 bpm baseline, the band tops out at about 83 bpm. The next game unlocks after the pulse has stayed at or below that for 15 s, once at least 60 s have passed since the loss.&lt;/p&gt;

&lt;h3&gt;
  
  
  What the screen shows, in real time
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;A big live bpm number, and a gauge with your baseline band and your pulse at the loss.&lt;/li&gt;
&lt;li&gt;A progress bar that fills while you stay in the band (15 s).&lt;/li&gt;
&lt;li&gt;The active path, always spelled out: &lt;strong&gt;"Live pulse"&lt;/strong&gt; or &lt;strong&gt;"Timer"&lt;/strong&gt;, with the reason.&lt;/li&gt;
&lt;li&gt;A calm breathing pace (clearly labelled as a guide, not a measurement) and a quiet &lt;strong&gt;Skip cooldown&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Data flow
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftr06f71f0t04t9bmjk2h.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ftr06f71f0t04t9bmjk2h.png" alt="Data flow" width="800" height="500"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  What I verified for real
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;With a real webcam, a real Presage key and real Lichess games (Nov 7–9):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;three check-ins (baselines 80.7, 77.3 and 77.5 bpm)&lt;/li&gt;
&lt;li&gt;three real losses unlocked &lt;strong&gt;by pulse&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;the timer fallback (camera covered, "too dark")&lt;/li&gt;
&lt;li&gt;the typed override&lt;/li&gt;
&lt;li&gt;restarting the app mid-cooldown (the lock and its start time were kept)&lt;/li&gt;
&lt;li&gt;sign-in and sign-out gating&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Automated:&lt;/strong&gt; 189 unit tests.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One honest finding:&lt;/strong&gt; in my casual bullet games, my pulse right after a loss stayed inside my band, so the 60 s minimum decided the unlock rather than the pulse. Longer sessions and loss streaks are where the pulse condition should matter most, and that's the next thing I want to measure.&lt;/p&gt;




&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Best Thoughtful Real-World Use&lt;/strong&gt;, and &lt;strong&gt;Most Creative or Delightful Use&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Best Thoughtful Real-World Use&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The need:&lt;/strong&gt; tilt-driven requeueing in online chess, measured across 11.3 million games.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Who it's for:&lt;/strong&gt; Lichess players who know they spiral after losses and want a pause that adapts to them instead of a fixed penalty.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Usability:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;consent comes first;&lt;/li&gt;
&lt;li&gt;a timer fallback and a typed escape hatch, so you're never trapped by a bad camera reading;&lt;/li&gt;
&lt;li&gt;works with no key ("Timer only");&lt;/li&gt;
&lt;li&gt;plain status text explains why it's waiting.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Responsible data handling:&lt;/strong&gt; everything stays on your machine (details below).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Most Creative or Delightful Use:&lt;/strong&gt; a chess clock for your nervous system. The camera doesn't watch your game; it waits with you after it, and lets you back in when you're ready.&lt;/p&gt;




&lt;h2&gt;
  
  
  Consent and Data Handling
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Camera consent comes first.&lt;/strong&gt; Nothing uses the camera until you choose "Allow camera between games". "Timer only" never touches it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The camera is on only between games:&lt;/strong&gt; during the one-minute check-in, on the cooldown screen after a loss, and on the diagnostics screen if you start a measurement. &lt;strong&gt;Never while you play.&lt;/strong&gt; Only the app's own page can request it; the embedded Lichess page gets no device permissions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;On-device processing.&lt;/strong&gt; The SmartSpectra SDK estimates pulse on your computer. &lt;strong&gt;No video or frames are uploaded or saved.&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What reaches Presage:&lt;/strong&gt; its servers are contacted only to validate the API key and record usage, never with video.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;SDK telemetry is turned off&lt;/strong&gt; (&lt;code&gt;enableTelemetry: false&lt;/code&gt;). Per Presage's &lt;a href="https://smartspectra.presagetech.com/docs/telemetry-and-privacy" rel="noopener noreferrer"&gt;Telemetry &amp;amp; Privacy doc&lt;/a&gt;, the SDK then starts no telemetry session and transmits nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Stored locally only:&lt;/strong&gt;

&lt;ul&gt;
&lt;li&gt;consent choice&lt;/li&gt;
&lt;li&gt;Lichess username&lt;/li&gt;
&lt;li&gt;baseline pulse&lt;/li&gt;
&lt;li&gt;a game log: result, losses in a row, baseline, first pulse after the loss, time to unlock, unlock reason&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Individual pulse readings are not stored. Diagnostic logs contain app events (for example game ids and outcomes), never pulse values or keys.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;No sharing.&lt;/strong&gt; Nothing is sent to me or anyone else. The only network traffic is lichess.org and Presage's key and usage checks:

&lt;ul&gt;
&lt;li&gt;lichess.org means the site, its public API, and a sign-in check using the embedded session's own cookies;&lt;/li&gt;
&lt;li&gt;the app never sees your Lichess password.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Retention and deletion:&lt;/strong&gt; data is kept until you delete it; nothing expires on its own. &lt;strong&gt;Delete all my data&lt;/strong&gt; removes:

&lt;ul&gt;
&lt;li&gt;the game log, baseline, username and consent&lt;/li&gt;
&lt;li&gt;backups and damaged copies of those files&lt;/li&gt;
&lt;li&gt;the saved Presage key&lt;/li&gt;
&lt;li&gt;the log files&lt;/li&gt;
&lt;li&gt;the embedded Lichess session&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It keeps only your own optional config file. Export (CSV/JSON) happens only when you ask, to a file you choose.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;No AI services&lt;/strong&gt; are used by the app.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not a medical device.&lt;/strong&gt; Zwischenzug does not diagnose, treat or monitor any condition and does not claim to detect "tilt"; it only compares a pulse estimate with your own baseline. There's no covert monitoring: the camera only runs with consent, between games.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;strong&gt;Thanks&lt;/strong&gt; to Presage for the SmartSpectra SDK, Lichess for its open API, and &lt;a href="https://github.com/romanbreyten/chess-tilt" rel="noopener noreferrer"&gt;romanbreyten/chess-tilt&lt;/a&gt; for the research behind the time floors.&lt;/p&gt;

&lt;p&gt;If this is useful to you, a ❤️ or a comment helps, and I'd love feedback from chess players on the thresholds.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>presagechallenge</category>
      <category>chess</category>
      <category>electron</category>
    </item>
    <item>
      <title>I stopped re-running flaky tests: a deterministic verdict, and a Gemma that's only allowed to guess out loud</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Mon, 26 Oct 2026 06:32:55 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/i-stopped-re-running-flaky-tests-a-deterministic-verdict-and-a-gemma-thats-only-allowed-to-guess-1le9</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/i-stopped-re-running-flaky-tests-a-deterministic-verdict-and-a-gemma-thats-only-allowed-to-guess-1le9</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-week3-2026-10-19"&gt;Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A test fails in CI. You press "re-run". It passes. You merge, and a week later it fails again on someone else's pull request.&lt;/p&gt;

&lt;p&gt;I have done this far more often than I'd like to admit. Re-running is the lazy move, and it's lazy in the bad way: it throws away the evidence. The first failure, the environment, the commit and the attempt that followed it are exactly what you'd need to decide whether the test is flaky or whether you just broke something. FlakePilot is my attempt at being lazy in the good way: file the evidence once, let a fixed set of rules read it, and stop guessing.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;FlakePilot&lt;/strong&gt; is an open-source (Apache-2.0), read-only control plane for investigating flaky tests.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;CI uploads a JUnit XML report. FlakePilot stores it with who, where and when: run id, attempt, job, commit SHA, ref, environment.&lt;/li&gt;
&lt;li&gt;It rebuilds each test's &lt;strong&gt;attempt chain&lt;/strong&gt; (&lt;code&gt;failed → passed&lt;/code&gt;, &lt;code&gt;failed → failed&lt;/code&gt;, and so on).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft5le1xzxlbj1e4hu0gyo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Ft5le1xzxlbj1e4hu0gyo.png" alt="Illustration of how attempt chains map to verdicts (illustration, not real data)" width="800" height="350"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It classifies every failure with a versioned, deterministic rule set, &lt;code&gt;ti-classification/1&lt;/code&gt;. The result is &lt;code&gt;flaky&lt;/code&gt;, &lt;code&gt;deterministic_regression&lt;/code&gt;, or an honest &lt;code&gt;inconclusive&lt;/code&gt; such as &lt;code&gt;insufficient_history&lt;/code&gt;. There are no confidence scores, because the data doesn't support them.&lt;/li&gt;
&lt;li&gt;Every verdict is stored together with the evidence it was computed from, so you can see &lt;em&gt;why&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;On request, an investigation can draft a &lt;strong&gt;hypothesis&lt;/strong&gt; about the cause with an open-weight model (Gemma). Code then checks that every claim cites stored evidence.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The effort it saves is the rerun-and-wonder loop. The user is whoever triages flaky CI at a small team or in an open-source project: the person who is currently reading three logs to decide whether to press "re-run".&lt;/p&gt;

&lt;p&gt;I want to be upfront about the weight of the AI here, because the challenge asks for open-source AI at the core. In FlakePilot the &lt;strong&gt;verdict is never a model's output&lt;/strong&gt;. Gemma is an optional, off-by-default advisory stage. I'll explain below why I think that's the right design, and where it leaves this submission.&lt;/p&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Deployed control plane: &lt;a href="https://app.flakepilot.tech" rel="noopener noreferrer"&gt;app.flakepilot.tech&lt;/a&gt;. It uses GitHub sign-in, so the signed-in pages need a login. The public landing page is open.&lt;/li&gt;
&lt;li&gt;I have no screen recording to show, so I'm not going to pretend there is one. What you can open without logging in is the landing page and the repository. The signed-in run view is what the diagrams below describe.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Please note: the execution plane is closed. FlakePilot never runs repository code, ever. It only reads reports.&lt;/p&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/CodewithJha" rel="noopener noreferrer"&gt;
        CodewithJha
      &lt;/a&gt; / &lt;a href="https://github.com/CodewithJha/FlakePilot" rel="noopener noreferrer"&gt;
        FlakePilot
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      Evidence-backed flaky-test investigation: ingest CI results, classify failures, propose and verify minimal patches with human approval.
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;FlakePilot&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;Evidence-based investigation and verification of recurring test failures. FlakePilot ingests your CI test reports, reconstructs each run's attempt chains, and classifies every failure with a deterministic, versioned rule set — never a model. Since 2026-10-25 the control plane is deployed on Render at &lt;strong&gt;&lt;a href="https://app.flakepilot.tech" rel="nofollow noopener noreferrer"&gt;app.flakepilot.tech&lt;/a&gt;&lt;/strong&gt;; the execution plane stays closed.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Pre-alpha. Not production-ready.&lt;/em&gt; See &lt;a href="https://github.com/CodewithJha/FlakePilot#status-and-verification" rel="noopener noreferrer"&gt;Status and verification&lt;/a&gt; for what is verified and what is not.&lt;/p&gt;
&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;Contents&lt;/h2&gt;
&lt;/div&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#what-flakepilot-does" rel="noopener noreferrer"&gt;What FlakePilot does&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#ci-reports-attempts-evidence-and-classification" rel="noopener noreferrer"&gt;CI reports, attempts, evidence and classification&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#the-investigation-workflow" rel="noopener noreferrer"&gt;The investigation workflow&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#architecture" rel="noopener noreferrer"&gt;Architecture&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#technology-stack" rel="noopener noreferrer"&gt;Technology stack&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/CodewithJha/FlakePilot#routes-and-user-workflows" rel="noopener noreferrer"&gt;Routes and user workflows&lt;/a&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#github-authentication" rel="noopener noreferrer"&gt;GitHub authentication&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#ci-ingest-endpoint" rel="noopener noreferrer"&gt;CI ingest endpoint&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#signed-in-pages" rel="noopener noreferrer"&gt;Signed-in pages&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#reproductions" rel="noopener noreferrer"&gt;Reproductions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#agent-route-auth0-opt-in" rel="noopener noreferrer"&gt;Agent route (Auth0, opt-in)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#security-model-and-limitations" rel="noopener noreferrer"&gt;Security model and limitations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#production-deployment" rel="noopener noreferrer"&gt;Production deployment&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/CodewithJha/FlakePilot#configuration" rel="noopener noreferrer"&gt;Configuration&lt;/a&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#github-app-sign-in-setup" rel="noopener noreferrer"&gt;GitHub App sign-in setup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#hosted-providers-adr-017" rel="noopener noreferrer"&gt;Hosted providers (ADR-017)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#operator-only-tools" rel="noopener noreferrer"&gt;Operator-only tools&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/CodewithJha/FlakePilot#local-development" rel="noopener noreferrer"&gt;Local development&lt;/a&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#prerequisites" rel="noopener noreferrer"&gt;Prerequisites&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#quality-gates" rel="noopener noreferrer"&gt;Quality gates&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#demo-journey" rel="noopener noreferrer"&gt;Demo journey&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#status-and-verification" rel="noopener noreferrer"&gt;Status and verification&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#documentation-map" rel="noopener noreferrer"&gt;Documentation map&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#contributing" rel="noopener noreferrer"&gt;Contributing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/FlakePilot#security-licence" rel="noopener noreferrer"&gt;Security, licence&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;What FlakePilot does&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;Flaky tests are treated like incidents, not opinions: every verdict has to come from stored, auditable CI evidence…&lt;/p&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/CodewithJha/FlakePilot" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;The repository is public and Apache-2.0. The first commit is from 20 October 2026 (19:32 IST), inside the challenge window, and the history is in the repository. If any commits land after the deadline, they are noted in the README.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The shape of the system
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb7o703s3seqa6og2zxqd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fb7o703s3seqa6og2zxqd.png" alt="FlakePilot architecture: a solid deterministic path from CI report to verdict, and a dashed advisory path through Gemma that never changes a verdict" width="800" height="450"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The stack is Python 3.12, FastAPI and PostgreSQL (SQLAlchemy, Alembic, asyncpg), managed with uv, and checked with Ruff, strict mypy, import-linter and pytest. It is a modular monolith with bounded contexts, and the import boundaries between them are enforced by 21 import-linter contracts in CI. The UI is a small React + TypeScript app built with Vite. Python still does every authorization check and only then embeds the run's data in the page.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fail-closed on purpose
&lt;/h3&gt;

&lt;p&gt;Most of the decisions were about what the system refuses to do:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Missing configuration disables a feature instead of guessing a default.&lt;/li&gt;
&lt;li&gt;A resource another tenant can see is always &lt;code&gt;404 not_found&lt;/code&gt;, never &lt;code&gt;403&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A commit SHA is stored exactly as CI reported it. Its revision status stays &lt;code&gt;unchecked&lt;/code&gt;, because FlakePilot does not verify Git ancestry.&lt;/li&gt;
&lt;li&gt;A test with too little history is &lt;code&gt;inconclusive&lt;/code&gt;, not a coin flip.&lt;/li&gt;
&lt;li&gt;Nothing executes repository code.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Where Gemma fits
&lt;/h3&gt;

&lt;p&gt;The investigation walks bounded stages: collect, research, browse, hypothesize, prioritize, brief. Each stage is &lt;code&gt;skipped&lt;/code&gt; with a stated reason when its provider isn't configured. The &lt;strong&gt;hypothesize&lt;/strong&gt; stage asks a model to suggest candidate causes from the stored evidence. It is off unless an organisation admin turns on an AI policy, and its output is stored as advisory data. It never changes a verdict and never grants anything.&lt;/p&gt;

&lt;p&gt;The part I'm happiest with is the checking step. The model's answer has to parse as a fixed JSON shape, and each claim has to cite an evidence id that actually exists for that run. A claim that cites nothing, or cites something that isn't there, is rejected.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;Honestly, for this project the reason is &lt;strong&gt;trust and swappability&lt;/strong&gt;, not capability.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The verdict doesn't depend on a model at all.&lt;/strong&gt; That's deliberate. If the model is wrong, slow, or retired, the classification is the same. An open-weight model fits an advisory role: I can change it without touching anything that decides a verdict.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The model is a configuration choice.&lt;/strong&gt; The hypothesize stage talks to either an operator-provided local endpoint or hosted Gemma. An operator who doesn't want failure logs leaving their network can point it at their own server.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Open code means the rules can be audited.&lt;/strong&gt; &lt;code&gt;ti-classification/1&lt;/code&gt; is a plain, versioned rule set in a public repository. If you disagree with a verdict, you can read exactly why it was produced.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;What I verified, and what I didn't:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Verified live (24–25 Oct 2026):&lt;/strong&gt; hosted Gemma 4 (&lt;code&gt;gemma-4-31b-it&lt;/code&gt;) through the Gemini API free tier. One investigation of a seeded flaky run, on a throwaway PostgreSQL database, produced an evidence-linked hypothesis with zero unverified claims.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Not verified:&lt;/strong&gt; I haven't recorded a run against a &lt;em&gt;local&lt;/em&gt; Gemma endpoint, so the "your logs never leave the machine" option is supported by the code and unit tests but not by a live run. Also, the hosted free tier may use prompts, which is a real trade-off.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So: open-weight AI is a deliberate, replaceable part of this project, but it is not the thing that makes the project work. If the challenge reads "open-source AI at the core" strictly, this is a borderline entry and I'd rather say so than dress it up.&lt;/p&gt;

&lt;h2&gt;
  
  
  What did not work
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Gemma's first answers all failed validation.&lt;/strong&gt; My prompt never said that &lt;code&gt;cause&lt;/code&gt; had to be a snake_case code or that citations had to be &lt;code&gt;{"evidence_id": ...}&lt;/code&gt; objects, so real answers came back as &lt;code&gt;invalid_hypothesis&lt;/code&gt;. The fix was to spell out every key and end the prompt with an example that the validator accepts. A unit test checks that example against the validator so the two can't drift apart. (Commit &lt;code&gt;fd71d89&lt;/code&gt;.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A hosted call can simply fail.&lt;/strong&gt; In one earlier end-to-end run the hypothesize stage was skipped with &lt;code&gt;provider_http_error&lt;/code&gt; after the bounded retries. The system recorded that honestly and carried on, which is the behaviour I wanted, but it is also a reminder that a hosted free tier is not a dependency you want in the critical path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Several sponsor integrations are not in the product.&lt;/strong&gt; Some were evaluated and deliberately not adopted, or exercised only against synthetic data or a local dev server. The Prize Categories section says which is which.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Current limitations
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Pre-alpha. The README says "not production-ready", and I mean it.&lt;/li&gt;
&lt;li&gt;The execution plane is closed, so FlakePilot can't reproduce a failure itself. It only reads what CI already recorded.&lt;/li&gt;
&lt;li&gt;Raw report bodies are stored as plaintext files. There is no encryption at rest yet.&lt;/li&gt;
&lt;li&gt;The signed-in demo needs GitHub sign-in, so a quick look from outside is limited to the landing page and the code.&lt;/li&gt;
&lt;li&gt;The classifier needs history. With one run, most things are honestly &lt;code&gt;inconclusive&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The Lazy Payoff
&lt;/h2&gt;

&lt;p&gt;I haven't measured time saved, and I'm not going to invent a number. The qualitative payoff is concrete, though: instead of pressing "re-run" and hoping, you get a stored attempt chain and a verdict that names the evidence behind it. If you want me to measure it, the honest experiment is to count how often a triage question gets answered without opening a raw log. I haven't run that yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I learned
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A model that proposes while code verifies is easier to trust than a model that decides. Rejecting an unsupported claim is cheap and visible.&lt;/li&gt;
&lt;li&gt;"Inconclusive" is a feature. Making the system say it out loud forced me to design for missing history instead of hiding it.&lt;/li&gt;
&lt;li&gt;Writing the prompt's expected output down, with a validated example, solved more than any prompt tweaking did.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Next I'd like to run the hypothesize stage against a local Gemma endpoint end to end and record it, add a no-login demo, and measure the triage claim above.&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;p&gt;FlakePilot uses each of these partners for one specific job. Here is what each one does in the project.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Best Use of Gemma:&lt;/strong&gt; Gemma 4, served through the Gemini API, powers the optional hypothesize stage of an investigation. It reads the stored evidence for a failing test and proposes candidate causes. Every claim it makes must cite an evidence id that exists for that run, and code rejects anything that does not. The output is stored as advisory data and never changes a verdict.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;.tech domain:&lt;/strong&gt; &lt;code&gt;flakepilot.tech&lt;/code&gt; is the project's domain, and &lt;code&gt;app.flakepilot.tech&lt;/code&gt; serves the control plane. GitHub sign-in, the run view and the API all run on it with a valid certificate.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Render:&lt;/strong&gt; the API, the background worker and the PostgreSQL database run on Render. A digest-pinned image is deployed after CI passes, database migrations run before each release, and the database has no public connections.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;SerpApi:&lt;/strong&gt; the research stage of an investigation runs one Google search built from the failing test names. It keeps deduplicated https sources only, within a fixed budget, and records them as evidence for the investigation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ElevenLabs:&lt;/strong&gt; on explicit request, FlakePilot turns an investigation into a short spoken brief. The brief is made of counts and the report link only, with no repository text, and is returned as an MP3 that is not stored.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Auth0 for AI Agents:&lt;/strong&gt; an agent can investigate a run on someone's behalf through a delegated token. FlakePilot verifies the token against the Auth0 tenant's keys, and a token can only narrow what the person can do, never widen it. An org admin binds and unbinds agents, bindings expire within eight hours, and every action is attributed to the agent.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MongoDB Atlas:&lt;/strong&gt; classified run reports can be exported to Atlas. The export is write-only and idempotent, and each document holds ids, counts and verdicts with their evidence, with no test names or artifact keys. PostgreSQL stays the system of record.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tiger Data:&lt;/strong&gt; the reliability analytics run on Tiger Cloud's PostgreSQL service. They compute per-test failure history, per-environment counts, failure clusters, and day and week trends of failures and flakiness compared with the previous window.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;TabPFN:&lt;/strong&gt; TabPFN predicts which failing tests to look at first. FlakePilot sends it only numeric history rows with labels, and an evaluation harness checks it against the deterministic baseline with a temporal split and past-only features.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Temporal:&lt;/strong&gt; an investigation can run as a Temporal workflow. Retries happen only for transient failures, cancellation propagates, and if a worker dies mid-run, a replacement finishes the job and exactly one investigation is recorded.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sentry:&lt;/strong&gt; an error reporter sends allow-listed events (exception class, frames, a message template, request id and route) from the API process. It never sends repository text or secrets.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mastra:&lt;/strong&gt; the investigation runs as a bounded workflow of collect, research, browse, hypothesize and prioritize stages, and records one provenance row at the end. A Mastra workflow built on &lt;code&gt;@mastra/core&lt;/code&gt; calls the same agent route as an authority-free client.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Browserbase:&lt;/strong&gt; the browse stage captures allowlisted research pages through a Browserbase adapter. Only exact public hosts are allowed, redirects are refused, and each capture is stored as hashed evidence whose text never reaches the model.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
    </item>
    <item>
      <title>Your Hindi support bot said "done". Did it cancel the right order?</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Mon, 19 Oct 2026 06:43:38 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/your-hindi-support-bot-said-done-did-it-cancel-the-right-order-585o</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/your-hindi-support-bot-said-done-did-it-cancel-the-right-order-585o</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-week2-2026-10-12"&gt;Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/cQz-CUc158w" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://bolibench-demo-1b3o.onrender.com" rel="noopener noreferrer"&gt;https://bolibench-demo-1b3o.onrender.com&lt;/a&gt; · &lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/CodewithJha/bolibench" rel="noopener noreferrer"&gt;https://github.com/CodewithJha/bolibench&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The reply was perfect. The order was wrong.
&lt;/h2&gt;

&lt;p&gt;A customer types: &lt;strong&gt;"कुर्ती वाले order की quantity 2 कर दो।"&lt;/strong&gt; ("Make the kurti order quantity 2.")&lt;/p&gt;

&lt;p&gt;The agent answers in fluent Hindi: &lt;em&gt;"कुर्ती वाले ऑर्डर (ORD-701) की क्वांटिटी सफलतापूर्वक 2 कर दी गई है।"&lt;/em&gt; ("The kurti order (ORD-701) has been updated to 2.")&lt;/p&gt;

&lt;p&gt;It reads perfectly, and it's wrong. ORD-701 is the customer's &lt;strong&gt;kurta&lt;/strong&gt;. The kurti is ORD-702, and nobody touched it. Grade that conversation by reading it, or by asking another LLM whether the reply sounds right, and it passes. The customer gets two kurtas.&lt;/p&gt;

&lt;p&gt;That is the problem I built &lt;strong&gt;BoliBench&lt;/strong&gt; to catch.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;BoliBench is an open-source behavioural benchmark for tool-using conversational agents. It does not believe the reply.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It runs your agent against a sandboxed order system, records every tool call the agent actually makes, and judges the sandbox's &lt;strong&gt;final state&lt;/strong&gt;: which order got cancelled, which quantity changed, whether the agent asked before acting, whether it retried a flaky tool, whether it broke store policy. What the agent says it did is stored — and never used as evidence.&lt;/p&gt;

&lt;h3&gt;
  
  
  At a glance
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;What it judges&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Recorded tool calls + final sandbox state, never reply text&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Suite&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;12 hand-written order-management cases&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Languages in the cases&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Hindi, Hinglish (Hindi-English code-switching), Awadhi-influenced Hindi&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Models tested&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;qwen3:8b and gemma3:4b locally via Ollama; Gemma 4 (&lt;code&gt;gemma-4-31b-it&lt;/code&gt;) hosted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Agent frameworks tested&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;A small Python example agent, and a Mastra TypeScript agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Judge&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Plain deterministic checks — no hidden LLM grader&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Licence&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Apache-2.0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  The 12 cases are the language people actually use
&lt;/h3&gt;

&lt;p&gt;The suite is 12 hand-written order-management cases in the language people actually use in north India:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Hindi-English code-switching: &lt;em&gt;"मेरा Kurta वाला order cancel कर दो please"&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;Awadhi-influenced Hindi: &lt;em&gt;"ऊ वाला order जउन कल मँगाए रहेन, ओका रद्द कइ दें"&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;Vague references: &lt;em&gt;"वो लाल वाला"&lt;/em&gt; ("the red one")&lt;/li&gt;
&lt;li&gt;Mid-sentence corrections: &lt;em&gt;"2 कर दो… नहीं नहीं, 3 कर दो"&lt;/em&gt; ("make it 2… no no, make it 3")&lt;/li&gt;
&lt;li&gt;Look-alike items (Kurta and Kurti) and look-alike sellers (Mohan Stores and Sohan Stores)&lt;/li&gt;
&lt;li&gt;A forbidden bulk cancel, and injected tool failures the agent has to recover from&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Who would pay for this — the side hustle
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Teams shipping Hindi, Hinglish or Awadhi support and voice agents&lt;/strong&gt; for Indian commerce, D2C brands and small sellers. Their question isn't "is the reply fluent?" It's "did the agent cancel the right order, and only that one?" A wrong cancellation costs a refund, a sale and an angry call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Voice-agent builders and agencies&lt;/strong&gt; who swap models and prompts every week and need a regression run that says which behaviour broke, case by case, before customers find out.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;What they'd pay for isn't the 12 public cases. It's private suites built from their own order flows, native-speaker review of those cases, and the same regression run wired into their CI.&lt;/p&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://bolibench-demo-1b3o.onrender.com" rel="noopener noreferrer"&gt;https://bolibench-demo-1b3o.onrender.com&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/cQz-CUc158w" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fho7cn0aowwqww52kx3hv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fho7cn0aowwqww52kx3hv.png" alt="BoliBench live demo home — the 12 cases and five real runs side by side" width="799" height="562"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The page explains what BoliBench checks, then puts the 12 cases against five real runs side by side: every verdict links to its full artifact (transcript, every tool call, final state, each check). An integrations panel says, for each technology, how far it has actually been verified: deployed, hosted, local, or only against stubs.&lt;/p&gt;

&lt;p&gt;Anyone can replay a scripted agent against the real evaluator for free, and a token holder can run a case live against the hosted open-weight model (Gemma 4, &lt;code&gt;gemma-4-31b-it&lt;/code&gt;); one hosted live run of &lt;code&gt;cancel-single-order&lt;/code&gt; passed in 35 seconds.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fowejmum15n4uzx675rul.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fowejmum15n4uzx675rul.png" alt="Evidence view — transcript, tool calls, final state and every check for one case" width="799" height="562"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/CodewithJha" rel="noopener noreferrer"&gt;
        CodewithJha
      &lt;/a&gt; / &lt;a href="https://github.com/CodewithJha/bolibench" rel="noopener noreferrer"&gt;
        bolibench
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      Open-source benchmark for conversational AI agents: task correctness, tool calls, and state verification on Hindi-English/Awadhi cases
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;BoliBench&lt;/h1&gt;
&lt;/div&gt;
&lt;p&gt;&lt;strong&gt;An open-source behavioural benchmark for tool-using conversational AI agents.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;BoliBench evaluates what an agent actually did—not what it said it did. It places an agent in a sandboxed order-management environment, records every tool call, and deterministically judges the final state and actions against each case's expected behaviour.&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;A fluent reply is not evidence that a task succeeded. BoliBench judges the recorded actions and the resulting state, never the wording of the agent's reply.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://bolibench-demo-1b3o.onrender.com" rel="nofollow noopener noreferrer"&gt;https://bolibench-demo-1b3o.onrender.com&lt;/a&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Repository:&lt;/strong&gt; &lt;a href="https://github.com/CodewithJha/bolibench" rel="noopener noreferrer"&gt;https://github.com/CodewithJha/bolibench&lt;/a&gt;
&lt;strong&gt;License:&lt;/strong&gt; Apache-2.0&lt;/p&gt;

&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;Table of contents&lt;/h2&gt;
&lt;/div&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#why-bolibench-exists" rel="noopener noreferrer"&gt;Why BoliBench exists&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#what-it-evaluates" rel="noopener noreferrer"&gt;What it evaluates&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#how-it-works" rel="noopener noreferrer"&gt;How it works&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#quick-start" rel="noopener noreferrer"&gt;Quick start&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#running-an-evaluation" rel="noopener noreferrer"&gt;Running an evaluation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#cases-and-review-status" rel="noopener noreferrer"&gt;Cases and review status&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#evaluation-rules" rel="noopener noreferrer"&gt;Evaluation rules&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#project-structure" rel="noopener noreferrer"&gt;Project structure&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#integrations-and-verification-status" rel="noopener noreferrer"&gt;Integrations and verification status&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#security-and-limitations" rel="noopener noreferrer"&gt;Security and limitations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/CodewithJha/bolibench#documentation" rel="noopener noreferrer"&gt;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;Why BoliBench exists&lt;/h2&gt;
&lt;/div&gt;
&lt;p&gt;Consider the request:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;कुर्ती वाले order की quantity 2 कर दो।&lt;/strong&gt;
“Make the quantity of the kurti order 2.”&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;An agent might reply, fluently, that the kurti order was updated—while actually changing the…&lt;/p&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/CodewithJha/bolibench" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;Apache-2.0.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;uv &lt;span class="nb"&gt;sync
&lt;/span&gt;uv run bolibench validate suites/orders        &lt;span class="c"&gt;# schema-check the cases&lt;/span&gt;
uv run bolibench review-sheet suites/orders    &lt;span class="c"&gt;# a blank sheet for a native speaker to review the cases&lt;/span&gt;
uv run bolibench run suites/orders &lt;span class="nt"&gt;--agent-url&lt;/span&gt; http://127.0.0.1:8000/turn &lt;span class="nt"&gt;--timeout&lt;/span&gt; 180 &lt;span class="nt"&gt;--output&lt;/span&gt; runs/
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Any agent can take part: BoliBench POSTs each user turn to your agent, hands it a per-turn tool URL, and records whatever it calls there. Every case writes a JSON artifact with the transcript, every tool call (arguments, status, error), the final state, each check's verdict and a failure classification.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. The judge: state, not sentences
&lt;/h3&gt;

&lt;p&gt;A Python sandbox holds the orders and exposes five tools (&lt;code&gt;list_orders&lt;/code&gt;, &lt;code&gt;get_order&lt;/code&gt;, &lt;code&gt;cancel_order&lt;/code&gt;, &lt;code&gt;update_quantity&lt;/code&gt;, &lt;code&gt;ask_user&lt;/code&gt;). Every attempt is logged as &lt;code&gt;ok&lt;/code&gt;, &lt;code&gt;rejected&lt;/code&gt; or &lt;code&gt;injected_failure&lt;/code&gt;, and only &lt;code&gt;ok&lt;/code&gt; calls change state.&lt;/p&gt;

&lt;p&gt;Each case is judged by plain, deterministic checks:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Final state&lt;/strong&gt; — is the sandbox in the state the case requires?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Required actions&lt;/strong&gt; — did a successful call actually happen?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Forbidden actions&lt;/strong&gt; — any attempt fails the case, even a rejected one&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ask-before-acting&lt;/strong&gt; — did the agent confirm where the case requires it?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Evidence present&lt;/strong&gt; — missing evidence is &lt;code&gt;inconclusive&lt;/code&gt;, never &lt;code&gt;pass&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The agent never sees the case id or the expected outcome.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. The agents: all on open-weight models
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;examples/ollama_agent.py&lt;/code&gt;, a small stdlib agent using Ollama's tool calling, running locally.&lt;/li&gt;
&lt;li&gt;The same agent with a &lt;strong&gt;JSON tool mode&lt;/strong&gt;, for models that have no native tool calling in Ollama. That's how Gemma runs: &lt;code&gt;gemma3:4b&lt;/code&gt; doesn't advertise &lt;code&gt;tools&lt;/code&gt; in Ollama, so the agent describes the tools in the prompt and accepts a call only if the whole reply is exactly &lt;code&gt;{"tool_calls": [...]}&lt;/code&gt;. Nothing is repaired or guessed.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;Mastra&lt;/strong&gt; TypeScript agent (&lt;code&gt;examples/mastra-agent&lt;/code&gt;) on &lt;code&gt;qwen3:8b&lt;/code&gt;, to check that BoliBench judges a framework-built agent the same way.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. What the runs showed
&lt;/h3&gt;

&lt;p&gt;Every row is one unseeded run of the same 12 cases; caveats follow the table.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Run&lt;/th&gt;
&lt;th&gt;Pass / fail / error (of 12)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;qwen3:8b, example agent v1&lt;/td&gt;
&lt;td&gt;3 / 9 / 0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3:8b, example agent v2 (prompt + loop fixes)&lt;/td&gt;
&lt;td&gt;6 / 6 / 0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3:8b, example agent v3 (native tools)&lt;/td&gt;
&lt;td&gt;6 / 6 / 0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemma3:4b, same v3 agent file, JSON tool mode&lt;/td&gt;
&lt;td&gt;2 / 10 / 0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3:8b, Mastra agent&lt;/td&gt;
&lt;td&gt;6 / 6 / 0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h4&gt;
  
  
  v1 failed on behaviour, not language
&lt;/h4&gt;

&lt;p&gt;The v1 agent mostly failed on behaviour, not language. It invented order ids (&lt;code&gt;123456&lt;/code&gt;, &lt;code&gt;789012&lt;/code&gt;) or passed words like &lt;code&gt;लाल वाला&lt;/code&gt; ("the red one") as an id, asked the customer a question and then acted without waiting for the answer, and passed an injected "503 … please retry" error straight to the customer. In the bulk-cancel case it said &lt;em&gt;"सभी आदेश रद्द कर दिए गए हैं"&lt;/em&gt; ("all orders have been cancelled") and did exactly that, against a store policy it had been given.&lt;/p&gt;

&lt;h4&gt;
  
  
  v2 fixed the agent — and BoliBench caught the regression too
&lt;/h4&gt;

&lt;p&gt;v2 changed only the agent: look orders up instead of guessing, a question ends the turn, retry a temporary failure once. Rejected tool calls went from 11 to 0, and both injected failures were retried.&lt;/p&gt;

&lt;p&gt;But two cases that passed before now failed: the agent became over-cautious and asked "which order?" when the Awadhi and Mohan/Sohan requests were already specific. BoliBench flagged the regression as plainly as the improvement — which is the whole point.&lt;/p&gt;

&lt;h4&gt;
  
  
  Gemma's 2/12 is mostly a tool-format result, not a language result
&lt;/h4&gt;

&lt;p&gt;In 7 of its 12 cases Gemma wrote a tool call in the right shape that the strict parser refused: a trailing full stop, a missing closing brace, or a sentence before the JSON. No tool ran, and the raw JSON became the reply.&lt;/p&gt;

&lt;p&gt;In the kurta/kurti case its rejected call was &lt;em&gt;exactly&lt;/em&gt; right (ORD-702, quantity 2), where qwen picked the wrong order. In the bulk-cancel case Gemma declined outright, which follows the policy more directly than qwen's "do you want to cancel all of them?".&lt;/p&gt;

&lt;p&gt;So this run compares model &lt;em&gt;plus&lt;/em&gt; tool interface, and the parser stays strict on purpose: in production, an almost-JSON call does nothing.&lt;/p&gt;

&lt;h4&gt;
  
  
  Mastra matched — with one honest caveat
&lt;/h4&gt;

&lt;p&gt;The framework agent's per-case verdicts matched the Python v2 run exactly. With one run each, that's a coincidence worth noting, not proof the two behave the same.&lt;/p&gt;

&lt;h4&gt;
  
  
  Local Gemma vs hosted Gemma 4 — two different runs, labelled as such
&lt;/h4&gt;

&lt;p&gt;The local Gemma evaluation is &lt;code&gt;gemma3:4b&lt;/code&gt; (2/12, JSON tool mode). Gemma 4 (&lt;code&gt;gemma-4-31b-it&lt;/code&gt;) also ran hosted through Google's Gemini API (an OpenAI-compatible endpoint) with native tool calls: one full-suite run scored 8 pass / 3 fail / 1 error, the error being provider latency past the turn deadline. That hosted run is Gemma evidence; it is not a DigitalOcean run, and I don't claim it as one.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Can the runs be repeated?
&lt;/h3&gt;

&lt;p&gt;A day later I reran three of Gemma's cases and two of Mastra's, with the same model digests and agent files. Every verdict, every tool call (tool, arguments, status) and every final state came out the same as in the bundled runs; only the wording of the replies changed.&lt;/p&gt;

&lt;p&gt;That's five cases, not a reproducibility study, but it means the bundled artifacts are what these agents actually do, not a lucky draw.&lt;/p&gt;

&lt;p&gt;Full per-case audits are in the repo: &lt;a href="https://github.com/CodewithJha/bolibench/blob/master/docs/audits/qwen3-8b-baseline-2026-10-17.md" rel="noopener noreferrer"&gt;v1 baseline&lt;/a&gt;, &lt;a href="https://github.com/CodewithJha/bolibench/blob/master/docs/audits/qwen3-8b-agent-v2-2026-10-17.md" rel="noopener noreferrer"&gt;v2&lt;/a&gt;, &lt;a href="https://github.com/CodewithJha/bolibench/blob/master/docs/audits/gemma-gemma3-4b-2026-10-17.md" rel="noopener noreferrer"&gt;Gemma&lt;/a&gt;, &lt;a href="https://github.com/CodewithJha/bolibench/blob/master/docs/integrations/mastra.md" rel="noopener noreferrer"&gt;Mastra&lt;/a&gt;, and every integration check, with commands and results, in the &lt;a href="https://github.com/CodewithJha/bolibench/blob/master/docs/live-verification-report.md" rel="noopener noreferrer"&gt;verification report&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Caveats, from the audits
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Single unseeded runs at each model's default temperature (qwen 0.6, Gemma 1). Per-case differences may be sampling noise. None of this measures qwen3:8b or Gemma in general.&lt;/li&gt;
&lt;li&gt;The v2 fixes were written after seeing v1 fail on the same 12 cases, so v2's score is not an unbiased estimate even for that agent.&lt;/li&gt;
&lt;li&gt;Gemma ran in prompt-based JSON tool mode and qwen in native mode. The JSON prompt was written once and deliberately not tuned on the suite.&lt;/li&gt;
&lt;li&gt;No case has been reviewed by a native speaker yet, so every language-related diagnosis is low confidence.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  5. Sponsor integrations — and how far each is actually verified
&lt;/h3&gt;

&lt;p&gt;I held the integrations to the same rule as the agents: a claim needs a recorded run. Each one below says whether it ran against the real hosted service, against the sponsor's software on my laptop, or only against stubs.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Integration&lt;/th&gt;
&lt;th&gt;How it was verified&lt;/th&gt;
&lt;th&gt;Entered as a prize category?&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Gemma&lt;/td&gt;
&lt;td&gt;Local &lt;code&gt;gemma3:4b&lt;/code&gt; full suite + hosted Gemma 4 full suite, both audited&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Yes&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Render&lt;/td&gt;
&lt;td&gt;Live demo deployed, token-gated live mode running in the container&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Yes&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mastra&lt;/td&gt;
&lt;td&gt;TypeScript agent, full-suite run&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Yes&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ElevenLabs&lt;/td&gt;
&lt;td&gt;Verified live on the free tier, speech round-trip end to end&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Yes&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Temporal&lt;/td&gt;
&lt;td&gt;Local dev server; workflow resumed after &lt;code&gt;kill -9&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Yes&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Backboard&lt;/td&gt;
&lt;td&gt;Real API, partial — comparison blocked by free-credit billing&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sentry Agent Tracing&lt;/td&gt;
&lt;td&gt;Real SDK, offline only — no trace ever reached Sentry&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MongoDB&lt;/td&gt;
&lt;td&gt;Local MongoDB 7 only — not Atlas&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tiger Data technology&lt;/td&gt;
&lt;td&gt;Local TimescaleDB + pgvector only — not Tiger Cloud&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DigitalOcean&lt;/td&gt;
&lt;td&gt;Code and configuration only — its API was never called&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h4&gt;
  
  
  Entered categories, with their evidence
&lt;/h4&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mastra:&lt;/strong&gt; &lt;code&gt;examples/mastra-agent&lt;/code&gt;, a TypeScript agent built on &lt;code&gt;@mastra/core&lt;/code&gt;, orchestrates &lt;code&gt;qwen3:8b&lt;/code&gt; through BoliBench's tool gateway. Full suite above; it also rejects malformed requests (400, 413, 404, 409) without falling over.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Temporal:&lt;/strong&gt; &lt;code&gt;python -m bolibench.temporal_run&lt;/code&gt; runs a suite as a Temporal workflow, one activity per case. On the local dev server, its summary was byte-identical to a plain &lt;code&gt;bolibench run&lt;/code&gt;. I &lt;code&gt;kill -9&lt;/code&gt;'d the worker in the middle of a case and restarted it: Temporal re-ran that case (attempt 2) and the run completed. Starting the same workflow again re-ran nothing. This makes the &lt;em&gt;evaluation&lt;/em&gt; durable. The agent itself isn't a workflow, and the crash test used a deliberately slow stub agent so the kill would land mid-case.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ElevenLabs:&lt;/strong&gt; &lt;code&gt;bolibench run --speech elevenlabs&lt;/code&gt; speaks each utterance with text-to-speech, transcribes it back with speech-to-text, and gives the agent only the transcript (stored as &lt;code&gt;asr_text&lt;/code&gt;). Verified live on the free tier: on three cases run end to end with local &lt;code&gt;qwen3:8b&lt;/code&gt;, the speech round-trip scored 1 pass / 2 fail against 0 / 3 for the same cases as plain text. Synthetic voices, not real speakers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Render:&lt;/strong&gt; the live demo above is one Docker container on Render (free plan), running as &lt;code&gt;nobody&lt;/code&gt; on a read-only filesystem. Live mode is on: the same container runs the example agent against the hosted Gemma 4 model, gated by a demo token, with a daily cap.&lt;/li&gt;
&lt;/ul&gt;

&lt;h4&gt;
  
  
  Also genuinely used, with their limits stated plainly
&lt;/h4&gt;

&lt;p&gt;Documented here, but &lt;strong&gt;not entered as prize categories&lt;/strong&gt;, because the evidence below is local, partial, or offline rather than a verified hosted use:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Backboard (real API, partial):&lt;/strong&gt; &lt;code&gt;examples/backboard_agent.py&lt;/code&gt; bridges BoliBench to Backboard's API. Verified live against the real service: authentication, model listing, an assistant/thread round-trip, tool calls, and memory isolation between cases (two bridge bugs were found this way and fixed). What I could not do: run the open-weight models I wanted to compare, because the account's free credit does not pay for LLM chat. A zero-priced preview model ran 4 cases (3 pass / 1 fail). So: the bridge is real and tested; a multi-model comparison through one Backboard key is &lt;strong&gt;not&lt;/strong&gt; claimed, because it never ran.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sentry Agent Tracing (SDK, offline only):&lt;/strong&gt; the example agent emits &lt;code&gt;gen_ai.invoke_agent&lt;/code&gt;, &lt;code&gt;gen_ai.chat&lt;/code&gt; and &lt;code&gt;gen_ai.execute_tool&lt;/code&gt; spans with token counts, grouped per case. This was tested with the real SDK against an in-memory transport; no DSN was ever configured, so no trace reached Sentry and I claim no hosted tracing. The offline work did catch a real leak: with the SDK's default local-variable capture, an error event carried the customer's utterance and an &lt;code&gt;Authorization&lt;/code&gt; header; local variables are now off, and a test keeps them off.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MongoDB (local only, not Atlas):&lt;/strong&gt; &lt;code&gt;bolibench export --to mongo&lt;/code&gt; exports run history with idempotent upserts and indexes. Verified against a local MongoDB 7 server, including the report command. No Atlas cluster or credentials were available, so this is a local MongoDB integration — not an Atlas deployment, and not entered as one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tiger Data technology (local only, not Tiger Cloud):&lt;/strong&gt; &lt;code&gt;bolibench export --to tiger&lt;/code&gt; stores run history in a TimescaleDB hypertable and embeds every failure with EmbeddingGemma into pgvector for hybrid keyword + vector "have we seen this failure before?" search. Verified locally on TimescaleDB 2.30.2 + pgvector 0.8.7 (22 failures embedded; the top matches for a failure were the same case in other runs). No Tiger Cloud service or credentials were available, so no cloud use is claimed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DigitalOcean (code and configuration only — never called):&lt;/strong&gt; an OpenAI-compatible backend option and an App Platform spec exist in the repo, and the same client code ran live against Ollama locally and Google's Gemini API hosted. DigitalOcean's own API was never called: no model access key was available. I list it here only to be explicit that it is implemented-but-unused, not an integration I ran.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;No sponsor outreach is claimed anywhere in this post: there is no record of contact with any partner, and appearing in this list means only that the code above exists and was exercised as described.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Because the data is the customer's
&lt;/h3&gt;

&lt;p&gt;Order histories, names, addresses and complaints are exactly what a support team can't paste into someone else's API. With an open-weight model in Ollama, the agent under test, the sandbox and every artifact stay on one laptop.&lt;/p&gt;

&lt;h3&gt;
  
  
  Because regression runs have to be cheap enough to run every time
&lt;/h3&gt;

&lt;p&gt;A team tuning prompts wants the suite after every change. Locally, the Gemma run took 82 seconds and the qwen v3 run 3 minutes 15, on a 16 GB laptop, with no bill. Metered runs get run less often, and the regressions slip through.&lt;/p&gt;

&lt;h3&gt;
  
  
  Because the judge has to be inspectable
&lt;/h3&gt;

&lt;p&gt;A benchmark that tells you "your agent cancelled the wrong order" must be something you can read, rerun and argue with. BoliBench's checks are plain deterministic code, with no hidden LLM grader. When we found a bug in our own failure classifier, the fix and the reasoning went into a public audit.&lt;/p&gt;

&lt;h3&gt;
  
  
  Because pinned weights make results attributable
&lt;/h3&gt;

&lt;p&gt;Each audit records the exact model digest and the agent file's hash. When the score moved from 3/12 to 6/12, the weights were byte-for-byte the same, so the change comes from the agent code (or, with one run each, from sampling), never from a provider quietly updating the model behind an API.&lt;/p&gt;

&lt;h3&gt;
  
  
  Because we control the whole stack
&lt;/h3&gt;

&lt;p&gt;With the weights on our own machine we could put Gemma behind a different tool interface than qwen, run both on the same agent file, and read every raw reply. That's how we learned Gemma's failures were mostly formatting, not understanding: in the kurta/kurti case its rejected call was right where the other model's accepted call was wrong.&lt;/p&gt;

&lt;h3&gt;
  
  
  Because these languages need their speakers
&lt;/h3&gt;

&lt;p&gt;Awadhi and everyday Hinglish are where generic tools are weakest. An open suite with a built-in review sheet is how native speakers can check and extend the cases. That review hasn't happened yet, and it's the next thing BoliBench needs.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limitations
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;No native-speaker review yet.&lt;/strong&gt; All 12 cases are synthetic and marked &lt;code&gt;proposed&lt;/code&gt;. Nothing in BoliBench understands Hindi or Awadhi; case correctness rests on the people who wrote them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Transcript-level&lt;/strong&gt; unless the ElevenLabs speech round-trip is used, and that uses synthetic voices.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;12 cases, single unseeded runs.&lt;/strong&gt; The counts are not model measurements.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Integrations are verified only as far as stated.&lt;/strong&gt; Where a line above says local or stubs, the hosted service hasn't been exercised.&lt;/li&gt;
&lt;li&gt;The agent under test must run next to BoliBench (localhost only). That's a dev safeguard, not a security boundary.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What I would do next
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Get the 12 cases reviewed by native speakers, using the review sheet already in the repo — this is the highest-value next step.&lt;/li&gt;
&lt;li&gt;Grow private-suite support for a first paying team: their own order flows, their own policies, the same deterministic judge.&lt;/li&gt;
&lt;li&gt;Wire the regression run into CI so a prompt or model change shows its case-by-case diff before it ships.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Try it
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Live demo:&lt;/strong&gt; &lt;a href="https://bolibench-demo-1b3o.onrender.com" rel="noopener noreferrer"&gt;https://bolibench-demo-1b3o.onrender.com&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Code (Apache-2.0):&lt;/strong&gt; &lt;a href="https://github.com/CodewithJha/bolibench" rel="noopener noreferrer"&gt;https://github.com/CodewithJha/bolibench&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you ship a Hindi, Hinglish or Awadhi agent, I would genuinely like to know: how do you check today that it did what it said it did?&lt;/p&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Best Use of Gemma: &lt;code&gt;gemma3:4b&lt;/code&gt; locally (2/12, JSON tool mode) and &lt;code&gt;gemma-4-31b-it&lt;/code&gt; hosted (8/3/1), both audited above, plus EmbeddingGemma for failure search.&lt;/li&gt;
&lt;li&gt;Best Use of Render: the live demo, with token-gated live mode running the agent in the container.&lt;/li&gt;
&lt;li&gt;Best Use of Mastra: the Mastra agent's full-suite run above.&lt;/li&gt;
&lt;li&gt;Best Use of ElevenLabs: the verified live speech round-trip above.&lt;/li&gt;
&lt;li&gt;Best Use of Temporal: the suite run as a durable workflow that resumed after &lt;code&gt;kill -9&lt;/code&gt; (a durable evaluation run, stated as such).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Documented above but &lt;strong&gt;not entered&lt;/strong&gt; as prize categories: Backboard (real API, billing-blocked comparison), Sentry Agent Tracing (SDK, offline only), MongoDB Atlas (local MongoDB only), Tiger Data (local TimescaleDB only), DigitalOcean (code/config only, never called). Not used at all: .tech Domain, Entire, GitHub Copilot, Tinker, TabPFN, SerpApi, Arduino.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
      <category>opensource</category>
      <category>ai</category>
    </item>
    <item>
      <title>ShadeLoop: an open-model field journal that gets you outside for 20 minutes</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Mon, 12 Oct 2026 06:54:54 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/shadeloop-an-open-model-field-journal-that-gets-you-outside-for-20-minutes-1o78</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/shadeloop-an-open-model-field-journal-that-gets-you-outside-for-20-minutes-1o78</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges"&gt;DEV Hacktoberfest 2026 "Touch Grass" challenge&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;You have 20 minutes. ShadeLoop finds you a shady bench, walks you there, and turns what you notice into evidence for the next person.&lt;/strong&gt;&lt;br&gt;
An open-weight model drafts the plan. A deterministic rule engine has the final word. Every step is logged, timed and shown to you.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg61dkowm9u07f9yrnaaj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg61dkowm9u07f9yrnaaj.png" alt="ShadeLoop Discover screen on a laptop: describe a 20 minute outing and get rule-checked missions near you" width="800" height="500"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Discover on a laptop: one sentence in, rule-checked missions out.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;ShadeLoop&lt;/strong&gt; is a field journal that turns "I have 20 minutes" into a short, safe outdoor mission near you, and then turns what you notice outside into shared evidence for the next person.&lt;/p&gt;

&lt;p&gt;You describe the outing you want ("a quiet 20 minute break somewhere shady with a bench"), and ShadeLoop:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Reads your request&lt;/strong&gt; with an open-weight model (Gemma 4), and remembers your earlier requests only if you opt in (Backboard).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Checks the real world&lt;/strong&gt;: current weather (Open-Meteo, with MET Norway as an automatic fallback) and mapped parks and gardens near your &lt;em&gt;rounded&lt;/em&gt; location (OpenStreetMap plus SerpApi).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Drafts missions&lt;/strong&gt; with Gemma, then runs every draft through a &lt;strong&gt;deterministic rule engine&lt;/strong&gt; that checks your time budget, walking limits, accessibility needs and past field reports. The engine has the final word: it can shorten or refuse a draft, and then template missions are used and labelled as such.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Field Mode&lt;/strong&gt; walks you through each step, with read-aloud steps (ElevenLabs, or your browser's voice offline), quick observation taps (shade, noise, bench, crowding, path, step-free) and dictated notes that Gemma turns into suggested observations you confirm.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Closes the loop&lt;/strong&gt;: you get a report you can listen to, and your observations become evidence: time-of-day patterns and "places with similar field reports" (Tiger Data, TimescaleDB plus pgvector).&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It's an installable PWA that works offline (an IndexedDB queue that syncs idempotently), and every plan shows a &lt;strong&gt;process log&lt;/strong&gt;: what ran, how long it took, what failed and what covered for it. Nothing is presented as more certain than it is.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Femtberq2pesjbsqxy31b.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Femtberq2pesjbsqxy31b.png" alt="ShadeLoop on a phone: the same Discover flow as an installable, offline-capable PWA" width="500" height="1700"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;The same flow on a phone. It installs as a PWA and keeps working offline.&lt;/em&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  Try it in 60 seconds
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Open &lt;strong&gt;&lt;a href="https://shadeloop.onrender.com" rel="noopener noreferrer"&gt;https://shadeloop.onrender.com&lt;/a&gt;&lt;/strong&gt; (give the free instance up to a minute to wake).&lt;/li&gt;
&lt;li&gt;Allow location; it's rounded to ~1 km before it leaves your device. No account needed.&lt;/li&gt;
&lt;li&gt;Type something like &lt;em&gt;"a quiet 20 minute break somewhere shady with a bench"&lt;/em&gt; and hit plan.&lt;/li&gt;
&lt;li&gt;Open the &lt;strong&gt;process log&lt;/strong&gt; under the plan to see every step, its timing and what covered for any failure.&lt;/li&gt;
&lt;li&gt;Start &lt;strong&gt;Field Mode&lt;/strong&gt;, tap what you notice, and finish to get your report.&lt;/li&gt;
&lt;li&gt;Done? &lt;strong&gt;Settings → Delete my data&lt;/strong&gt; wipes everything in one call.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Live app:&lt;/strong&gt; &lt;a href="https://shadeloop.onrender.com" rel="noopener noreferrer"&gt;https://shadeloop.onrender.com&lt;/a&gt; (free Render instance, so the first load can take ~30–60 s to wake up)&lt;/p&gt;
&lt;h2&gt;
  
  
  Code
&lt;/h2&gt;

&lt;p&gt;Repo: &lt;a href="https://github.com/CodewithJha/shadeloop" rel="noopener noreferrer"&gt;github.com/CodewithJha/shadeloop&lt;/a&gt; (MIT)&lt;/p&gt;
&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Stack:&lt;/strong&gt; npm workspaces (&lt;code&gt;shared&lt;/code&gt;, &lt;code&gt;server&lt;/code&gt;, &lt;code&gt;web&lt;/code&gt;) · React 19 + Vite 8 PWA (vite-plugin-pwa, IndexedDB offline queue) · Express 5 + Zod · MongoDB Atlas · Tiger Data (TimescaleDB hypertable + 15-min continuous aggregate, pgvector) · Gemma 4 &lt;code&gt;gemma-4-26b-a4b-it&lt;/code&gt; via Google AI Studio (or &lt;code&gt;gemma3:4b&lt;/code&gt; locally via Ollama) · Backboard · ElevenLabs · SerpApi · OpenStreetMap Overpass (with mirror failover) · Open-Meteo → MET Norway · Sentry agent tracing · Docker on Render · Vitest (with mongodb-memory-server) · GitHub Actions CI.&lt;/p&gt;
&lt;h3&gt;
  
  
  Architecture at a glance
&lt;/h3&gt;


&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TD
    R[Your request + rounded location] --&amp;gt; M[Backboard: recall earlier requests, opt-in]
    R --&amp;gt; W[Weather: Open-Meteo]
    W -- 429 / failure --&amp;gt; MET[MET Norway fallback]
    R --&amp;gt; P[Places: OpenStreetMap Overpass + SerpApi]
    M --&amp;gt; G[Gemma 4: parse + draft missions]
    W --&amp;gt; G
    MET --&amp;gt; G
    P --&amp;gt; G
    G --&amp;gt; E{Deterministic rule engine}
    E -- accepted --&amp;gt; MS[Missions]
    E -- refused --&amp;gt; T[Template missions, labelled]
    MS --&amp;gt; F[Field Mode: steps read aloud by ElevenLabs]
    T --&amp;gt; F
    F --&amp;gt; O[Observations: MongoDB + Tiger Data hypertable + pgvector fingerprints]
    O --&amp;gt; RP[Report: read aloud, time-of-day, similar places]
    S[[Sentry agent tracing: plan run, tool spans, model spans]] -.-&amp;gt; M &amp;amp; W &amp;amp; P &amp;amp; G &amp;amp; E &amp;amp; F &amp;amp; RP&lt;/code&gt;&lt;/pre&gt;


&lt;h3&gt;
  
  
  How a mission travels from your phone to the server and back
&lt;/h3&gt;


&lt;pre data-lang="mermaid"&gt;&lt;code&gt;sequenceDiagram
    participant U as You (PWA)
    participant W as Web app (React + IndexedDB)
    participant A as Express API
    participant X as Gemma / Weather / Places / Backboard
    participant D as MongoDB + Tiger Data
    U-&amp;gt;&amp;gt;W: Describe the outing (20 min, shade, bench)
    W-&amp;gt;&amp;gt;A: POST /sessions (anonymous token)
    W-&amp;gt;&amp;gt;A: POST /missions/plan
    A-&amp;gt;&amp;gt;X: recall memory, weather, places, Gemma draft
    A-&amp;gt;&amp;gt;A: rule engine accepts or refuses every candidate
    A-&amp;gt;&amp;gt;D: save missions + process log
    A--&amp;gt;&amp;gt;W: missions + timed process log
    U-&amp;gt;&amp;gt;W: Start mission (Field Mode)
    W-&amp;gt;&amp;gt;A: GET /missions/:id/steps/:n/speech (ElevenLabs)
    U-&amp;gt;&amp;gt;W: Tap or dictate observations (works offline)
    W-&amp;gt;&amp;gt;W: queue in IndexedDB when offline
    W-&amp;gt;&amp;gt;A: POST /missions/:id/observations or POST /sync
    A-&amp;gt;&amp;gt;D: store once (idempotent), hypertable row + pgvector fingerprint
    W-&amp;gt;&amp;gt;A: GET /reports/:id, /locations/:id/hourly, /locations/:id/similar
    A--&amp;gt;&amp;gt;U: report read aloud, time-of-day pattern, similar places&lt;/code&gt;&lt;/pre&gt;


&lt;h3&gt;
  
  
  Frontend vs backend: who does what
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Layer&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;th&gt;Where in the repo&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Web (React 19 + Vite PWA)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Discover form, missions list, Field Mode with voice and observation taps, History, Settings &amp;gt; Integrations with live status, offline queue in IndexedDB, installable PWA with offline fonts&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;web/src/App.tsx&lt;/code&gt;, &lt;code&gt;Field.tsx&lt;/code&gt;, &lt;code&gt;Settings.tsx&lt;/code&gt;, &lt;code&gt;Integrations.tsx&lt;/code&gt;, &lt;code&gt;store.ts&lt;/code&gt;, &lt;code&gt;api.ts&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;API (Express 5 + Zod)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Validates every request, anonymous sessions, rate limits on expensive routes, OpenAPI spec at &lt;code&gt;/openapi.json&lt;/code&gt;, health and dependency checks&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;server/src/app.ts&lt;/code&gt;, &lt;code&gt;openapi.ts&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Agent pipeline&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Reads the request with Gemma, recalls Backboard memory, fetches weather and places, drafts missions, logs every step with timings&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;server/src/missions.ts&lt;/code&gt;, &lt;code&gt;ai.ts&lt;/code&gt;, &lt;code&gt;providers.ts&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Rule engine&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Deterministic veto on walking time, limits and past field reports; refused plans fall back to labelled templates&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;server/src/domain.ts&lt;/code&gt;, &lt;code&gt;fallback.ts&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Data&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;MongoDB for missions, observations and reports; Tiger Data hypertable for time-of-day patterns and pgvector for similar places&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;server/src/db.ts&lt;/code&gt;, &lt;code&gt;tiger.ts&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Privacy&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Rounded location, export everything (&lt;code&gt;GET /me/export&lt;/code&gt;), delete everything (&lt;code&gt;DELETE /me&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;server/src/account.ts&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Observability&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Sentry spans for the plan run, each tool call and each model call&lt;/td&gt;
&lt;td&gt;&lt;code&gt;server/src/instrument.ts&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;
&lt;h3&gt;
  
  
  The model proposes, the rules decide
&lt;/h3&gt;

&lt;p&gt;This is the actual veto, from &lt;code&gt;server/src/domain.ts&lt;/code&gt;. Every candidate, from Gemma or a template, goes through it, and no model text can override it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="cm"&gt;/** Deterministic checks every candidate (AI or template) must pass. AI text can't override these. */&lt;/span&gt;
&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;validateCandidate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;Candidate&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;PlanContext&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;errors&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;[]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[];&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;unsafe&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;findUnsafeText&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;title&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;description&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;objective&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;whyItMatches&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="nx"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;s&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)].&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;unsafe&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;errors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`unsafe text: "&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;unsafe&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="c1"&gt;// ...&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;maxMinutes&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;Math&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;min&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;availableMinutes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;RULES&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;maxMissionMinutes&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;estimatedMinutes&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;maxMinutes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;errors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`estimated &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;estimatedMinutes&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; min exceeds limit &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;maxMinutes&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; min`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;walkingMinutes&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;prefs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;maxWalkMinutes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;errors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`walking &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;walkingMinutes&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; min exceeds your limit of &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;prefs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;maxWalkMinutes&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; min`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;chk&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nf"&gt;conditionChecksFor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;loc&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt; &lt;span class="o"&gt;??&lt;/span&gt; &lt;span class="kc"&gt;null&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;chk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;required&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nx"&gt;chk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;contradicted&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="nx"&gt;errors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`required condition &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;chk&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;condition&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; is contradicted by reports`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;errors&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;length&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="kd"&gt;const&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;errors&lt;/span&gt; &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="c1"&gt;// ...unverified accessibility conditions are surfaced as explicit uncertainty, never as facts&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That &lt;code&gt;walking … exceeds your limit&lt;/code&gt; line is exactly the refusal you'll see in the live process log below. Even when a draft passes, an accessibility condition backed only by anonymous reports is shown as &lt;strong&gt;unverified&lt;/strong&gt;, with a note to check access yourself.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Real numbers from today's live runs on &lt;a href="https://shadeloop.onrender.com" rel="noopener noreferrer"&gt;https://shadeloop.onrender.com&lt;/a&gt; (12 Oct 2026, a throwaway session in Bengaluru, deleted afterwards via &lt;code&gt;DELETE /me&lt;/code&gt;):&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Process log of a full plan (28.1 s total):&lt;/strong&gt; Gemma read the request in 10.6 s → Backboard recall 63 ms → places 3.5 s (OpenStreetMap: 45 places, SerpApi: &lt;strong&gt;20 places&lt;/strong&gt;) → Gemma drafted 2 candidates in 11.7 s → rule engine accepted both → Backboard save 1.9 s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Second plan:&lt;/strong&gt; Backboard &lt;strong&gt;recalled 1&lt;/strong&gt; earlier request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rule engine refusal&lt;/strong&gt; (in an earlier live run today): Gemma proposed a walk of &lt;strong&gt;23 min against a 20 min limit&lt;/strong&gt;, the engine refused it ("walking 23 min exceeds your limit of 20 min") and fell back to labelled templates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Weather:&lt;/strong&gt; Open-Meteo returned &lt;strong&gt;HTTP 429&lt;/strong&gt; from Render's shared IP; the process log read &lt;code&gt;Open-Meteo failed (Open-Meteo HTTP 429) → MET Norway: 28.5°C&lt;/code&gt; (1.3 s).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ElevenLabs:&lt;/strong&gt; step audio 17.6 KB in 1.5 s; report audio 61 KB in 1.6 s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tiger Data pgvector:&lt;/strong&gt; two parks with different field reports matched at &lt;strong&gt;similarity 0.4&lt;/strong&gt; (not a fake 1.0), and time-of-day patterns were read back from the hypertable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Offline replay idempotency:&lt;/strong&gt; sending the same observation twice returned &lt;strong&gt;201 then 200&lt;/strong&gt; and stored it once.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Delete my data:&lt;/strong&gt; removed missions, observations, reports, Tiger rows and the Backboard memory; the old token then got 401.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tests:&lt;/strong&gt; &lt;strong&gt;129 server&lt;/strong&gt; and &lt;strong&gt;43 web&lt;/strong&gt; tests (Vitest), plus lint, typecheck, PWA build and Docker build in CI.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  A real process log from the live app
&lt;/h3&gt;

&lt;p&gt;Copied from live plans on 12 Oct 2026 (Bengaluru, throwaway session). The refusal line comes from an earlier live run the same day; the other lines are the 11:44 IST and 11:14 IST runs.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;PROCESS LOG
+0.1s   ✓ READ REQUEST   gemma-4-26b-a4b-it read the request (1 call) · 10.6s
+10.7s  ✓ MEMORY         Backboard recalled 0 earlier requests · 0.06s
+2.4s   ✓ WEATHER        Open-Meteo failed (Open-Meteo HTTP 429) → MET Norway: 28.5°C · 1.3s
+10.8s  ✓ PLACES         openstreetmap: 45 places · serpapi: 20 places · 3.5s
+14.4s  ✓ MODEL DRAFTS   gemma-4-26b-a4b-it drafted 2 candidates (1 call) · 11.7s
+26.1s  ✓ RULE ENGINE    accepted "Park Stillness"
+26.1s  ✓ RULE ENGINE    accepted "Garden Observation"
+26.2s  ✓ MEMORY SAVE    Backboard saved this request · 1.9s
        ✗ RULE ENGINE    refused a draft: walking 23 min exceeds your limit of 20 min → labelled templates
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  What happens when things fail
&lt;/h2&gt;

&lt;p&gt;Free public APIs fail. ShadeLoop is built so a failure is a labelled step in the log, not a blank screen:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;What fails&lt;/th&gt;
&lt;th&gt;What covers for it&lt;/th&gt;
&lt;th&gt;What you see&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Open-Meteo (e.g. HTTP 429 on Render's shared IP)&lt;/td&gt;
&lt;td&gt;MET Norway forecast&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Open-Meteo failed (HTTP 429) → MET Norway: 28.5°C&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OSM Overpass main endpoint&lt;/td&gt;
&lt;td&gt;Mirror failover (&lt;code&gt;overpass.kumi.systems&lt;/code&gt;, &lt;code&gt;overpass.private.coffee&lt;/code&gt;), then SerpApi places&lt;/td&gt;
&lt;td&gt;Amber "covered by SerpApi" in Integrations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemma unavailable or every draft refused&lt;/td&gt;
&lt;td&gt;Labelled template missions, still checked by the same rule engine&lt;/td&gt;
&lt;td&gt;"template" label on the mission&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;No network at all&lt;/td&gt;
&lt;td&gt;IndexedDB queue in the PWA, synced idempotently later&lt;/td&gt;
&lt;td&gt;Replay returns 201 then 200, stored once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ElevenLabs unavailable&lt;/td&gt;
&lt;td&gt;Your browser's built-in voice&lt;/td&gt;
&lt;td&gt;Steps are still read aloud&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2n7ekbqxns88r5983jxj.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2n7ekbqxns88r5983jxj.png" alt="Settings &gt; Integrations: all 10 sponsors with live status, verified only after a real call succeeded" width="800" height="778"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Settings → Integrations. "Verified" only appears after a real call succeeded; covered failures show in amber.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Privacy and offline by design
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Coarse location:&lt;/strong&gt; rounded to ~1 km before it leaves the device.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No accounts:&lt;/strong&gt; an anonymous session token, nothing else.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Opt-in memory:&lt;/strong&gt; Backboard only remembers requests if you turn it on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One-tap delete:&lt;/strong&gt; &lt;code&gt;DELETE /me&lt;/code&gt; removes missions, observations, reports, Tiger Data rows and the Backboard memory; the old token then gets 401.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Offline-first PWA:&lt;/strong&gt; observations queue in IndexedDB and sync idempotently, so a flaky park Wi-Fi never duplicates a report.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3m4mboc2nrbgcoe39kqo.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3m4mboc2nrbgcoe39kqo.png" alt="History: past missions and field reports, with time-of-day patterns and similar places" width="800" height="500"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;History: your past missions and reports become evidence: time-of-day patterns and pgvector "similar places".&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What makes it different
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The model proposes, the rules decide.&lt;/strong&gt; Gemma drafts; a tested, deterministic engine accepts, shortens or refuses every draft.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nothing is hidden.&lt;/strong&gt; A per-plan process log with real timings, and an integrations panel that says "verified" only after a real call succeeded.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Failures are covered, and named.&lt;/strong&gt; Open-Meteo 429 → MET Norway; OSM down → SerpApi, shown as amber "covered by …".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It closes the loop.&lt;/strong&gt; Your field notes become shared evidence: time-of-day patterns and pgvector "similar places".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Private and offline-first.&lt;/strong&gt; Location rounded to ~1 km, no accounts, opt-in memory, one-tap delete, and an offline PWA with idempotent sync.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why Does Open Innovation Matter?
&lt;/h2&gt;

&lt;p&gt;Getting someone outside for 20 minutes sounds small, but the tool that does it knows where you are, when you're free and what you struggle with. That's exactly the kind of software that should be open.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Open weights.&lt;/strong&gt; The planner is Gemma, an open-weight model. Anyone can inspect what it is, run the same model, and reproduce the behaviour. There's no black box between you and the suggestion to walk somewhere.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Runs locally.&lt;/strong&gt; Set &lt;code&gt;AI_PROVIDER=ollama&lt;/code&gt; and &lt;code&gt;gemma3:4b&lt;/code&gt; runs on your own laptop (about 20–30 s per plan on my Mac). Your request never has to leave your machine, and the app still works with no AI at all: it falls back to labelled templates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The model is not the authority.&lt;/strong&gt; A deterministic, tested rule engine checks every draft against your time, walking limit and accessibility needs. Open code means anyone can read the exact rules that keep a suggestion safe.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A transparent process log.&lt;/strong&gt; Every plan shows each step, its timing, what failed and what covered for it ("Open-Meteo 429 → MET Norway"). The integrations panel only says "verified" after a real call succeeded; a key that's merely set says "waiting for first use".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Open data, given back.&lt;/strong&gt; Places come from OpenStreetMap (credited on every mission), weather from Open-Meteo and MET Norway. Your field reports become evidence that helps the next person find shade or a free bench.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Privacy by design.&lt;/strong&gt; Location is rounded to about 1 km before it leaves your device, memory is opt-in, there are no accounts (just an anonymous session), and &lt;strong&gt;Delete my data&lt;/strong&gt; wipes MongoDB, Tiger Data and Backboard in one call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;MIT licence.&lt;/strong&gt; Fork it for your campus, your city or your accessibility needs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  My Agent Session
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Entire session:&lt;/strong&gt; &lt;a href="https://entire.io/gh/CodewithJha/shadeloop/commit/89b8eada487181a88134341db161312060d1bce4" rel="noopener noreferrer"&gt;https://entire.io/gh/CodewithJha/shadeloop/commit/89b8eada487181a88134341db161312060d1bce4&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GitHub Copilot CLI&lt;/strong&gt; picked up &lt;a href="https://github.com/CodewithJha/shadeloop/issues/1" rel="noopener noreferrer"&gt;issue #1&lt;/a&gt; and fixed it in commit &lt;code&gt;89b8ead&lt;/code&gt;, which is the session linked above.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Sponsor&lt;/th&gt;
&lt;th&gt;What it does in ShadeLoop&lt;/th&gt;
&lt;th&gt;Live evidence (12 Oct 2026)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Gemma&lt;/td&gt;
&lt;td&gt;Parses requests and field notes, drafts missions; &lt;code&gt;gemma3:4b&lt;/code&gt; runs locally via Ollama&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;gemma-4-26b-a4b-it&lt;/code&gt; drafted 2 missions, rule engine accepted them&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sentry Agent Tracing&lt;/td&gt;
&lt;td&gt;Each plan traced as an agent run with tool and model spans&lt;/td&gt;
&lt;td&gt;Traces sent as agent runs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MongoDB Atlas&lt;/td&gt;
&lt;td&gt;Missions, observations, reports, place evidence, idempotent sync&lt;/td&gt;
&lt;td&gt;Replay returned 201 then 200, stored once&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ElevenLabs&lt;/td&gt;
&lt;td&gt;Reads Field Mode steps and the report aloud&lt;/td&gt;
&lt;td&gt;Step audio 17.6 KB in 1.5 s, report 61 KB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Backboard&lt;/td&gt;
&lt;td&gt;Opt-in memory of your earlier requests&lt;/td&gt;
&lt;td&gt;Recalled 1 on the second plan; deleted with your data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tiger Data&lt;/td&gt;
&lt;td&gt;Time-of-day hypertable + pgvector similar places&lt;/td&gt;
&lt;td&gt;Similarity 0.4 between two parks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SerpApi&lt;/td&gt;
&lt;td&gt;Google Maps places alongside OpenStreetMap&lt;/td&gt;
&lt;td&gt;20 places&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GitHub Copilot&lt;/td&gt;
&lt;td&gt;Copilot CLI fixed issue #1; CI runs 129 + 43 tests&lt;/td&gt;
&lt;td&gt;Commit &lt;code&gt;89b8ead&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Render&lt;/td&gt;
&lt;td&gt;Hosts the API and PWA as one Docker service&lt;/td&gt;
&lt;td&gt;&lt;a href="https://shadeloop.onrender.com" rel="noopener noreferrer"&gt;https://shadeloop.onrender.com&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Entire&lt;/td&gt;
&lt;td&gt;Publishes the agent session behind the Copilot fix&lt;/td&gt;
&lt;td&gt;Session link above&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Limitations and honesty
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cold start:&lt;/strong&gt; the free Render instance sleeps when idle, so the first request can take 30–60 s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Weather:&lt;/strong&gt; Open-Meteo rate-limits Render's shared IP (HTTP 429). MET Norway now covers it automatically, but MET Norway has no "feels like" temperature, so heat and cold notes based on it don't appear on those plans.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Places:&lt;/strong&gt; public OpenStreetMap Overpass sometimes fails from Render (its mirrors too); SerpApi still supplies places and the panel marks OSM "covered by SerpApi".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sentry:&lt;/strong&gt; traces are sent, but the app can't confirm delivery itself, and says so.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Evidence is young:&lt;/strong&gt; field-report patterns and similarity need real reports to be meaningful; with few reports the app says "insufficient data" rather than guessing.&lt;/li&gt;
&lt;li&gt;Search results show that a place is mapped, not that it is shaded, safe, accessible or open.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Challenges I hit
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Open-Meteo rate limits.&lt;/strong&gt; Render's shared IP started getting HTTP 429. Instead of hiding it, I added MET Norway as an automatic fallback and made the log name the switch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Overpass from the cloud.&lt;/strong&gt; The public OSM endpoint fails intermittently from Render, so it fails over across mirrors and SerpApi fills the gap.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keeping the model honest.&lt;/strong&gt; Gemma happily proposed a 23 min walk against a 20 min limit. That's why the rule engine, not the model, has the final word, and why its refusals are logged.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Verified" has to mean something.&lt;/strong&gt; A key that's only set says "waiting for first use"; the panel turns green only after a real call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Similarity that isn't fake.&lt;/strong&gt; pgvector place fingerprints only mean something with real reports, so the app shows real scores (0.4, not 1.0) and says "insufficient data" when it should.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;More field reports, so time-of-day patterns and "similar places" get genuinely useful.&lt;/li&gt;
&lt;li&gt;Local-first by default: make the Ollama &lt;code&gt;gemma3:4b&lt;/code&gt; path the easy one-command setup.&lt;/li&gt;
&lt;li&gt;A "feels like" estimate when MET Norway is the weather source.&lt;/li&gt;
&lt;li&gt;Forks for campuses and cities, with their own accessibility rules. It's MIT, so go for it.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you try it, tell me what mission you got and what you noticed. That's the whole point: get outside for 20 minutes. 🌳&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
      <category>opensource</category>
      <category>ai</category>
    </item>
    <item>
      <title>My friend runs her supplements shop from WhatsApp voice notes, so I built her a shop helper</title>
      <dc:creator>Priyanshu Jha</dc:creator>
      <pubDate>Mon, 05 Oct 2026 06:32:07 +0000</pubDate>
      <link>https://dev.to/priyanshu_jha_eee132b17e5/my-friend-runs-her-supplements-shop-from-whatsapp-voice-notes-so-i-built-her-a-shop-helper-5785</link>
      <guid>https://dev.to/priyanshu_jha_eee132b17e5/my-friend-runs-her-supplements-shop-from-whatsapp-voice-notes-so-i-built-her-a-shop-helper-5785</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-weekend-2026-10-01"&gt;Hacktoberfest Weekend Challenge: Build for a Friend&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Her day, before Nivara
&lt;/h2&gt;

&lt;p&gt;My friend runs a supplements shop on her own. No staff, no counter, no software. Just her phone.&lt;/p&gt;

&lt;p&gt;Orders come in as WhatsApp messages and voice notes, mostly in Hindi or Hinglish: "Rahul bhai ko 2 MB biozyme whey bhej dena kal tak." She keeps the stock count in her head. Most weeks that works. Then a best-seller runs out, and she finds out when a customer asks for it.&lt;/p&gt;

&lt;p&gt;When I asked what she actually wanted, it wasn't a chatbot. It was three answers every morning:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;What do I deliver today?&lt;/li&gt;
&lt;li&gt;What do I reorder before it runs out?&lt;/li&gt;
&lt;li&gt;Am I paying a supplier more than I need to?&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So that's what I built.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Nivara does
&lt;/h2&gt;

&lt;p&gt;Nivara is a small web app she opens on her phone between chats.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Today&lt;/strong&gt; answers the morning question first. The top of the screen is a loaded barbell: every job is a plate, heaviest first, coloured like competition bumper plates. Red is "do this now", yellow is "this week", green is "when you have a minute". The heaviest job gets the full width of the screen and one button.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1icfl7dtocp19nmujc1j.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1icfl7dtocp19nmujc1j.png" alt="Today on a phone" width="780" height="1688"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Orders&lt;/strong&gt; turns a voice note into a draft order. She can record one or upload a forwarded WhatsApp note. I uploaded a short Hinglish clip to the live site. It showed what it heard, "राहुल भाई को 2 MB Biozyme भेज देना कल तक", and drafted Rahul Verma, 2 × MB biozyme whey at ₹2,599, ₹5,198 in total, delivery on 6 Oct worked out from "कल तक". Nothing is saved until she taps "Confirm and save". Pasting a chat message works the same way.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flh6mrzdak70dt6tbuper.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flh6mrzdak70dt6tbuper.png" alt="A voice note read into a draft order" width="780" height="1240"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ask&lt;/strong&gt; takes typed or spoken questions in English or Hinglish. I said "मुझे क्या restock करना चाहिए?" and got the same two products, with the same order quantities, that Today and Forecast show. Answers can be read aloud.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F32hn1zmcp8dxkwf4rjeb.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F32hn1zmcp8dxkwf4rjeb.png" alt="Ask, answering a spoken Hinglish question" width="780" height="1688"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stock&lt;/strong&gt;, &lt;strong&gt;Forecast&lt;/strong&gt; and &lt;strong&gt;Suppliers&lt;/strong&gt; cover the rest: what's on the shelf, what sells in the next 7 days against how long a delivery takes, and which quotes are cheaper. A cheaper quote only counts if it saves at least ₹10 and 5% a unit, so she isn't nagged to switch suppliers over ₹3. A morning brief is ready at 8 every day.&lt;/p&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;  &lt;iframe src="https://www.youtube.com/embed/8ziwm3fX8K0" width="710" height="399"&gt;
  &lt;/iframe&gt;
&lt;/p&gt;

&lt;p&gt;Music: "Wallpaper" Kevin MacLeod (incompetech.com), CC BY 4.0&lt;/p&gt;

&lt;p&gt;Live: &lt;a href="https://nivara-x9iv.onrender.com" rel="noopener noreferrer"&gt;https://nivara-x9iv.onrender.com&lt;/a&gt; (free Render plan, so the first open can take up to a minute).&lt;/p&gt;

&lt;h2&gt;
  
  
  The one rule: Gemma does language, code does facts
&lt;/h2&gt;

&lt;p&gt;A shop helper that confidently gives the wrong stock number is worse than none. So I drew a hard line early.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Gemma reads and writes words.&lt;/strong&gt; It turns her messages and voice-note transcripts into JSON, picks which lookup a question needs, and writes the one-line summary at the top of the morning brief.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Code owns every fact.&lt;/strong&gt; Stock, held units, days of stock, risk, how many to order, supplier savings and dates are plain TypeScript over the database, with tests. "Kal", "कल" and "parson" become dates in code, not in the model. Gemma never writes to the database.&lt;/p&gt;

&lt;p&gt;In practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Gemma's order JSON is checked with zod. If it's broken, Gemma gets one retry with the error, then the request fails cleanly.&lt;/li&gt;
&lt;li&gt;Code matches the draft to real customers and products ("bhai" is ignored in the name), then shows it to her. Only the confirm button saves, and it validates again.&lt;/li&gt;
&lt;li&gt;Standard questions are answered from templates over the tool output. I first let a small Gemma write those answers, and it dropped list items, swapped supplier names and mixed up available and total stock. Templates fixed that.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Why an open model
&lt;/h2&gt;

&lt;p&gt;Her customers' names, their chat orders and her margins are not data I want to send to a closed API by default.&lt;/p&gt;

&lt;p&gt;On the live site Nivara uses Gemma 4 (&lt;code&gt;gemma-4-26b-a4b-it&lt;/code&gt;) through Google AI Studio, because a free Render instance can't run a model. The same app runs against Gemma in Ollama with one environment variable, and I built most of it against &lt;code&gt;gemma3:4b&lt;/code&gt; on my laptop. If she wants her data to stay on her own laptop, it can. It costs nothing per message and doesn't depend on a vendor's pricing.&lt;/p&gt;

&lt;p&gt;A small open model also kept me honest. It can't be trusted with numbers, so the design never asks it for any.&lt;/p&gt;

&lt;h2&gt;
  
  
  The data, and what's not real
&lt;/h2&gt;

&lt;p&gt;I don't have her sales history yet, and there is no open dataset of Indian supplement sales. So I built the most honest stand-in I could:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Catalogue:&lt;/strong&gt; 212 real products from Open Food Facts India (ODbL). Real brands, and real mess: a few Chyawanprash jars made it in.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prices:&lt;/strong&gt; Open Prices had observed INR prices for only 4 of them. The rest are estimates from pack size, product type and brand tier. My first pass priced whey tubs as sachets (₹139 for Biozyme Performance Whey); after fixing the pack-size rules it's ₹2,749.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Demand:&lt;/strong&gt; Google Trends search interest for India, scaled to what a solo shop might sell. It's a stand-in, tagged &lt;code&gt;proxy&lt;/code&gt; in the database, and the Forecast page says so in one line.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When she confirms and delivers orders, those real sales join the same history.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making it hers
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Shop words, not database IDs: Order 1, 2, 3, product names, "Rahul Verma", India time.&lt;/li&gt;
&lt;li&gt;No partner names on her pages. Those live on one Health page for me.&lt;/li&gt;
&lt;li&gt;When something fails she sees a plain sentence and a Retry button, not a stack trace.&lt;/li&gt;
&lt;li&gt;On her phone the menu folds into one button, and every tap target is big enough for a thumb.&lt;/li&gt;
&lt;li&gt;The example questions only read data. Tapping one can never change an order or the stock.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Checking it the way she'd use it
&lt;/h2&gt;

&lt;p&gt;Before handing it over I clicked every page and button on the live site, on 360 and 390 px phones and a laptop, and checked each change in the databases rather than trusting the screen. Saving an order had to create it in Atlas. Marking it delivered had to take two tubs off the shelf (21 to 19), record the sale, and add a row to the Tiger sales history. That pass found real bugs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Today said to reorder 11 of a product while Ask said 7. One path used the forecast, the other used last week's sales. Now every page uses the forecast.&lt;/li&gt;
&lt;li&gt;"Cheapest online" picked a ₹197 sachet for a ₹6,949 tub of whey. An online price now has to share a brand word and sit within half to double her price.&lt;/li&gt;
&lt;li&gt;Orders had numbers like "Order 27" made from database ids. They're now Order 1, 2, 3.&lt;/li&gt;
&lt;li&gt;Times showed "2:38 am" because the server runs on UTC. Everything is in India time now, and delivered sales land on the shop's day.&lt;/li&gt;
&lt;li&gt;Voice notes with background music came back with "[outro jingle]" in the transcript. Sound tags are off now, and a note that is only music counts as silence.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What each partner does for her
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Tiger Data: her sales history and product search.&lt;/strong&gt; Daily units per product live in a Timescale hypertable, &lt;code&gt;sales_daily&lt;/code&gt;. Two continuous aggregates, &lt;code&gt;sales_demand_7d&lt;/code&gt; and &lt;code&gt;sales_demand_28d&lt;/code&gt;, keep rolling 7- and 28-day totals ready, and the brief and Ask read her realised sales from them while the forecast handles next week. When she marks an order delivered in Atlas, a change stream writes each line to an &lt;code&gt;order_sales&lt;/code&gt; table keyed by order and product, so a repeated sync can't count a sale twice, and adds the units to &lt;code&gt;sales_daily&lt;/code&gt; on her day in India time. Product search is hybrid: Postgres full-text search plus pgvector over 768-dimension &lt;code&gt;gemini-embedding-001&lt;/code&gt; embeddings, ranked by text rank plus twice the vector similarity. Hard filters like "under 3000" apply to both sides, so a close vector match can't slip past her price limit. "MuscleBlaze whey under 3000" returns matches cheapest first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;MongoDB Atlas: the source of truth.&lt;/strong&gt; Every order she confirms, her stock, customers, saved rules and briefs. The only way to save an order is the confirm path, which checks every product and customer exists.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Gemma: reading her messages.&lt;/strong&gt; Order extraction from text and voice transcripts, choosing the tool, the brief's summary line.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;ElevenLabs: her voice notes.&lt;/strong&gt; Scribe turns the voice note or spoken question into text, then it goes through the same Gemma and code checks as a typed message. Answers can be read aloud.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TabPFN: what sells next week.&lt;/strong&gt; TabPFN forecasts 7 days per product from 60 days of history. Render has no Python, so I ran it on my laptop over all 212 products (317 seconds on CPU) and stored the predictions in Atlas. Days of stock, risk and how many to order all come from that one forecast, and Health shows the day it was worked out.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;SerpApi: online prices and demand.&lt;/strong&gt; Google Shopping prices for products about to run out, and the Google Trends data behind the demand estimate.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Render: where it lives.&lt;/strong&gt; One free web service in Singapore, health-checked on &lt;code&gt;/api/health&lt;/code&gt;. A GitHub Actions job pings it every 10 minutes until 15 Oct so you don't land on a cold start. With no background worker on the free plan, the app writes the 8 AM brief itself.&lt;/p&gt;

&lt;p&gt;The stack also uses Backboard to keep her rules in her own words, Mastra for the agent's tool definitions, Sentry to trace each question with prompts and customer details redacted, and Temporal for the scheduled jobs when a worker runs (on Render they run inside the app). There are 137 tests plus 15 recorded Keploy API cases.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy45mv33ptpmga0j8y4s2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fy45mv33ptpmga0j8y4s2.png" alt="Health, behind the scenes" width="800" height="873"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd do next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Read orders straight from WhatsApp Business instead of forwarding voice notes.&lt;/li&gt;
&lt;li&gt;One tap from "order 11 more" to a message for the supplier.&lt;/li&gt;
&lt;li&gt;Rerun TabPFN on her real sales once there are a few weeks of them.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Prize Categories
&lt;/h2&gt;

&lt;p&gt;Best Use of Tiger Data, Best Use of SerpApi, Best Use of TabPFN, Best Use of ElevenLabs, Best Use of MongoDB Atlas, Best Use of Gemma, Best Use of Render&lt;/p&gt;

&lt;h2&gt;
  
  
  Links
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Repo: &lt;a href="https://github.com/CodewithJha/nivara" rel="noopener noreferrer"&gt;https://github.com/CodewithJha/nivara&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Live app: &lt;a href="https://nivara-x9iv.onrender.com" rel="noopener noreferrer"&gt;https://nivara-x9iv.onrender.com&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Built by Priyanshu Jha (&lt;a href="https://github.com/CodewithJha" rel="noopener noreferrer"&gt;@CodewithJha&lt;/a&gt;) for a friend who deserves to stop counting tubs in her head.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>weekendchallenge</category>
      <category>hf26challenge</category>
      <category>ai</category>
    </item>
  </channel>
</rss>
