<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: אחיה כהן</title>
    <description>The latest articles on DEV Community by אחיה כהן (@achiya-automation).</description>
    <link>https://dev.to/achiya-automation</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3810102%2Fefb43e59-992c-4f8b-91df-ee602c7c853f.jpg</url>
      <title>DEV Community: אחיה כהן</title>
      <link>https://dev.to/achiya-automation</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/achiya-automation"/>
    <language>en</language>
    <item>
      <title>Safari gave my blank tab no URL. My security check read that as "not yours."</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Sat, 10 Oct 2026 19:28:10 +0000</pubDate>
      <link>https://dev.to/achiya-automation/safari-gave-my-blank-tab-no-url-my-security-check-read-that-as-not-yours-p4d</link>
      <guid>https://dev.to/achiya-automation/safari-gave-my-blank-tab-no-url-my-security-check-read-that-as-not-yours-p4d</guid>
      <description>&lt;p&gt;I maintain &lt;a href="https://github.com/achiya-automation/safari-mcp" rel="noopener noreferrer"&gt;safari-mcp&lt;/a&gt;, an MCP server that lets AI agents drive the real Safari on a Mac, logged-in sessions included. On October 7 an agent opened a blank tab, and its very next command on that tab was refused:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;receipt is not valid for this origin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The agent had done everything right. It called &lt;code&gt;safari_new_tab()&lt;/code&gt; without a URL, kept the receipt the tool returned, and passed it to &lt;code&gt;safari_navigate&lt;/code&gt;. The tab could not be navigated, the receipt could not be rotated, and the tab could not be closed. It sat there in a named Safari profile, and nothing the agent was allowed to do could touch it.&lt;/p&gt;

&lt;p&gt;I shipped two releases that day, and each changed one line of code. Both lines were in a guard I had built to be strict on purpose.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a receipt is
&lt;/h2&gt;

&lt;p&gt;When the safari-mcp Safari extension opens a tab for an agent, it mints an opaque receipt for that tab. Every later command carries the receipt, and before it runs anything the extension checks two things:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;the receipt belongs to this tab, and&lt;/li&gt;
&lt;li&gt;the tab is still on the origin the receipt was minted on.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The second check is the one that matters. An agent works in a browser that also holds the user's email, bank and admin panels. If a page redirects the agent's tab somewhere else, I want the agent's next &lt;code&gt;evaluate&lt;/code&gt; refused, not run against whatever is loaded there now. To keep working, the agent has to ask for a fresh receipt on purpose (&lt;code&gt;getReceipt&lt;/code&gt;), and the extension issues one only when it can read an http(s) origin, or &lt;code&gt;about:blank&lt;/code&gt;, for the tab.&lt;/p&gt;

&lt;p&gt;Fail closed. On paper, that is exactly the behavior you want.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug was one missing value
&lt;/h2&gt;

&lt;p&gt;This is how the extension derived a tab's origin before the fix:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;_receiptOrigin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;rawUrl&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;rawUrl&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="dl"&gt;""&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;about:blank&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;parsed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sr"&gt;/^https&lt;/span&gt;&lt;span class="se"&gt;?&lt;/span&gt;&lt;span class="sr"&gt;:$/&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;parsed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;protocol&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;?&lt;/span&gt; &lt;span class="nx"&gt;parsed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;origin&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;""&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;""&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A blank tab's receipt is minted on &lt;code&gt;about:blank&lt;/code&gt;. I assumed Safari would report a blank tab's URL as &lt;code&gt;about:blank&lt;/code&gt;. It doesn't. It leaves &lt;code&gt;url&lt;/code&gt; off the tab entirely.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;String(undefined || "")&lt;/code&gt; is &lt;code&gt;""&lt;/code&gt;. &lt;code&gt;new URL("")&lt;/code&gt; throws. The function returns &lt;code&gt;""&lt;/code&gt;, which in this code means "no origin at all". &lt;code&gt;"about:blank" !== ""&lt;/code&gt;, so the check refused the tab's own receipt. The escape hatch called the same function, got the same &lt;code&gt;""&lt;/code&gt;, and refused as well: &lt;code&gt;Cannot issue a receipt for this tab URL&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The guard did what I built it to do. It just could not tell "this tab moved somewhere else" from "Safari didn't tell me where this tab is", and it treated both as an attack.&lt;/p&gt;

&lt;h2&gt;
  
  
  The sibling bug, one day earlier
&lt;/h2&gt;

&lt;p&gt;On October 6 a different run left two tabs open in a named profile, both on claude.ai/new. In both cases the page itself had moved the tab, through a redirect or a login bounce. Rotation only works when the new page has an http(s) origin the extension can read; a tab that lands on a page without one, or on a page whose URL Safari doesn't show the extension, can't be rotated.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;close_tab&lt;/code&gt; was held to the receipt's origin like every other command. The refusal suggested rotating the receipt, which was exactly what these two tabs couldn't do. Nothing could close them.&lt;/p&gt;

&lt;p&gt;Same shape as the blank tab: a rule that is right for commands that run code in a page, applied to a command that doesn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  Two one-line fixes
&lt;/h2&gt;

&lt;p&gt;v2.22.12, the moved tab:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;allowReceiptOriginChange&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;type&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;get_tab_receipt&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;type&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;close_tab&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Closing a tab runs nothing in the page, so a close now finds its tab the way &lt;code&gt;getReceipt&lt;/code&gt; does, by the receipt rather than by the origin. Every other command stays bound to the receipt's origin, and a receipt the extension never issued still closes nothing.&lt;/p&gt;

&lt;p&gt;v2.22.13, the blank tab:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;raw&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;about:blank&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;about:blank&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A tab Safari shows no URL for now counts as blank. A receipt minted on a web origin still takes no page command there, so a tab that went from a real site to a blank page is still refused.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I got wrong
&lt;/h2&gt;

&lt;p&gt;The origin binding was fine. Two assumptions inside it were not.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I treated "unknown" as "mismatch".&lt;/strong&gt; A fail-closed check has at least three outcomes: it matches, it doesn't match, or the platform didn't give you enough to decide. My code folded the third into the second, because the function that read the URL returned the same empty string for "no URL reported" and for "an origin I won't trust". Once those two share a value, the guard will sooner or later lock out the owner. The fix was to decide, in code, what a missing URL means for a tab the extension created: it is blank.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;I held every command to the same rule.&lt;/strong&gt; The origin check exists to stop code from running in the wrong page. &lt;code&gt;close_tab&lt;/code&gt; runs no code in any page. Binding it to the origin added no protection and took away the only way out. A guard should be as narrow as the harm it prevents, and every guard needs an exit that doesn't depend on the thing it guards.&lt;/p&gt;

&lt;p&gt;A test that mocks the extension API with a tidy &lt;code&gt;url: "about:blank"&lt;/code&gt; could never have caught the first bug. Both showed up the first time a real Safari profile did something slightly unusual, which is the reason this project drives the real browser in the first place.&lt;/p&gt;

&lt;p&gt;If you use safari-mcp's Safari extension, both fixes live in the extension, so they take effect once you rebuild the extension app from 2.22.13 or later (the README's "Installing the Extension" section has the steps).&lt;/p&gt;

&lt;p&gt;How do your own guards represent "I couldn't tell"? Does it get its own value, or does it quietly share one with "no"?&lt;/p&gt;

</description>
      <category>security</category>
      <category>javascript</category>
      <category>opensource</category>
      <category>ai</category>
    </item>
    <item>
      <title>Cupom Chatwoot 2026: o único código real dá 5% (e onde colocar)</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Tue, 06 Oct 2026 08:13:33 +0000</pubDate>
      <link>https://dev.to/achiya-automation/cupom-chatwoot-2026-o-unico-codigo-real-da-5-e-onde-colocar-41i4</link>
      <guid>https://dev.to/achiya-automation/cupom-chatwoot-2026-o-unico-codigo-real-da-5-e-onde-colocar-41i4</guid>
      <description>&lt;p&gt;Aviso antes de tudo: &lt;code&gt;ACHIYAVS&lt;/code&gt; é o meu código de afiliado do Chatwoot. Se você usar, paga 5% a menos no Chatwoot Cloud e o Chatwoot me paga uma comissão. O acordo é só esse, e é por isso que este texto existe.&lt;/p&gt;

&lt;p&gt;Escrevo em português porque o Chatwoot é muito usado no Brasil e, quando procurei "cupom chatwoot", não achei nenhuma página em português com um código que funcione no Chatwoot Cloud. Achei o guia oficial que mostra onde digitar um cupom, uma oferta de parceiro para outro produto e listas de códigos que não existem.&lt;/p&gt;

&lt;h2&gt;
  
  
  O resumo
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Código:&lt;/strong&gt; &lt;code&gt;ACHIYAVS&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Desconto:&lt;/strong&gt; 5% em todos os pagamentos da assinatura, mensal ou anual, segundo as regras do &lt;a href="https://www.chatwoot.com/affiliate-program" rel="noopener noreferrer"&gt;programa de afiliados do Chatwoot&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Vale para:&lt;/strong&gt; os planos pagos do Chatwoot Cloud (Startups, Business e Enterprise)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Não vale para:&lt;/strong&gt; o plano gratuito Hacker e o Chatwoot auto-hospedado&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Onde colocar:&lt;/strong&gt; dentro da conta, no portal de faturamento. No cadastro não existe campo de cupom&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Quanto isso dá em reais
&lt;/h2&gt;

&lt;p&gt;Os preços são os da &lt;a href="https://www.chatwoot.com/pt-BR/pricing/" rel="noopener noreferrer"&gt;página de preços do Chatwoot em português&lt;/a&gt;, por agente, por mês:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Plano&lt;/th&gt;
&lt;th&gt;Preço&lt;/th&gt;
&lt;th&gt;Com ACHIYAVS&lt;/th&gt;
&lt;th&gt;Diferença por agente&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Startups&lt;/td&gt;
&lt;td&gt;R$ 119&lt;/td&gt;
&lt;td&gt;R$ 113,05&lt;/td&gt;
&lt;td&gt;R$ 5,95 por mês&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Business&lt;/td&gt;
&lt;td&gt;R$ 249&lt;/td&gt;
&lt;td&gt;R$ 236,55&lt;/td&gt;
&lt;td&gt;R$ 12,45 por mês&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enterprise&lt;/td&gt;
&lt;td&gt;R$ 639&lt;/td&gt;
&lt;td&gt;R$ 607,05&lt;/td&gt;
&lt;td&gt;R$ 31,95 por mês&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;É pouco. Numa equipe de cinco atendentes no Business, a diferença fica em R$ 62,25 por mês. Prefiro mostrar esse número pequeno e verdadeiro a prometer um desconto grande que não existe.&lt;/p&gt;

&lt;h2&gt;
  
  
  Onde digitar o código
&lt;/h2&gt;

&lt;p&gt;O cupom entra depois que a conta já existe. O teste grátis de 15 dias vem primeiro:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Entre no Chatwoot Cloud e abra &lt;strong&gt;Configurações → Faturamento&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Clique em &lt;strong&gt;Ir para o portal de faturamento&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Escolha o plano, clique em &lt;strong&gt;Adicionar código promocional&lt;/strong&gt;, digite &lt;code&gt;ACHIYAVS&lt;/code&gt; e clique em &lt;strong&gt;Aplicar&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Confira o resumo do pedido antes de clicar em &lt;strong&gt;Enviar&lt;/strong&gt;. O valor precisa aparecer com o desconto; se não aparecer, o código não foi aplicado.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;O próprio Chatwoot tem um &lt;a href="https://www.chatwoot.com/hc/user-guide/articles/1777049615-como-usar-seu-codigo-de-cupom-no-chatwoot" rel="noopener noreferrer"&gt;guia em português sobre como usar um cupom&lt;/a&gt;, com as telas.&lt;/p&gt;

&lt;p&gt;Vale saber também: um link de afiliado (aquele com &lt;code&gt;?via=&lt;/code&gt; no final) não dá desconto nenhum sozinho. O Chatwoot registra a indicação só pelo cupom. Sem digitar o código, você paga o preço cheio.&lt;/p&gt;

&lt;h2&gt;
  
  
  E os cupons de 50%, 60%, 70%?
&lt;/h2&gt;

&lt;p&gt;Quem procura cupom do Chatwoot encontra SINTU, PINTU, START, KUMAR, MALIK, SAVE55, GET50 e outros, prometendo de 20% a 70% de desconto. Nenhum corresponde a um desconto que o Chatwoot oferece. O programa de afiliados dá os mesmos 5% para todos (&lt;a href="https://www.chatwoot.com/affiliate-program" rel="noopener noreferrer"&gt;as regras estão publicadas&lt;/a&gt;), então um código que promete mais do que isso não é de afiliado: foi inventado, é de outro produto ou expirou há muito tempo.&lt;/p&gt;

&lt;h2&gt;
  
  
  E os 15% da fazer.ai?
&lt;/h2&gt;

&lt;p&gt;Esse desconto existe, mas é para outro produto. A &lt;a href="https://fazer.ai/parceria-chatwoot" rel="noopener noreferrer"&gt;fazer.ai&lt;/a&gt; é a parceira oficial do Chatwoot no Brasil e oferece 15% na &lt;strong&gt;licença Enterprise do Chatwoot auto-hospedado&lt;/strong&gt;. O cupom é aplicado no painel de Super Admin da sua própria instalação e vale só para operações no Brasil. Se você roda o Chatwoot no seu servidor e precisa do Enterprise, é a melhor oferta que conheço. No Chatwoot Cloud, o desconto disponível é o de afiliado, de 5%.&lt;/p&gt;

&lt;h2&gt;
  
  
  O que pesa mais do que o cupom
&lt;/h2&gt;

&lt;p&gt;Cada plano do Chatwoot Cloud guarda o histórico de conversas por um tempo limitado e apaga o que passa disso: 30 dias no Hacker, 1 ano no Startups, 2 anos no Business e 3 anos no Enterprise. Se a sua equipe precisa consultar conversas de mais de um ano atrás, o plano certo é o Business, e a diferença de R$ 119 para R$ 249 por agente pesa muito mais do que qualquer cupom. O Business também traz voz, times e regras de automação, que o Startups não tem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Se você prefere auto-hospedar
&lt;/h2&gt;

&lt;p&gt;O Chatwoot Community Edition é gratuito e roda no seu servidor sem limite de agentes, então ali o cupom não se aplica, e nem precisa. Eu rodo Chatwoot auto-hospedado para clientes, ligado ao WhatsApp, e sei que cuidar do servidor dá trabalho. O Cloud faz sentido para quem não quer esse trabalho, e é aí que o código vale.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Última verificação: 6 de outubro de 2026, no painel de afiliados do Chatwoot, onde o &lt;code&gt;ACHIYAVS&lt;/code&gt; aparece como "5% off on all payments". A versão em inglês desta página fica em &lt;a href="https://achiya-automation.com/en/blog/chatwoot-promo-code/?utm_source=devto&amp;amp;utm_medium=article" rel="noopener noreferrer"&gt;achiya-automation.com&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>chatwoot</category>
      <category>braziliandevs</category>
      <category>portugues</category>
      <category>saas</category>
    </item>
    <item>
      <title>n8n's Postgres Node Doesn't Send Parameters. My Dry-Run Did.</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Mon, 05 Oct 2026 05:49:09 +0000</pubDate>
      <link>https://dev.to/achiya-automation/n8ns-postgres-node-doesnt-send-parameters-my-dry-run-did-2j5b</link>
      <guid>https://dev.to/achiya-automation/n8ns-postgres-node-doesnt-send-parameters-my-dry-run-did-2j5b</guid>
      <description>&lt;p&gt;The n8n docs for the &lt;a href="https://docs.n8n.io/integrations/builtin/app-nodes/n8n-nodes-base.postgres/" rel="noopener noreferrer"&gt;Postgres node&lt;/a&gt; say you can use &lt;code&gt;$1&lt;/code&gt;, &lt;code&gt;$2&lt;/code&gt; and &lt;code&gt;$3&lt;/code&gt; in a query "to build prepared statements to use with query parameters." On September 18 I found out what that means in practice, from a failure that my own dry-run had told me couldn't happen.&lt;/p&gt;

&lt;h2&gt;
  
  
  The query that passed in psql
&lt;/h2&gt;

&lt;p&gt;I was running an end-to-end test of a workflow I maintain that books a callback. One Postgres node stores when the callback is due, and the number of minutes comes from earlier in the flow. Sometimes there is no number, so the value is an empty string. Simplified, the node's query looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;CASE&lt;/span&gt;
  &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
  &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="n"&gt;make_interval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mins&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;END&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before deploying, I had tested it in psql the way I'd test any parameterized query:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;PREPARE&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;CASE&lt;/span&gt; &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
         &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="n"&gt;make_interval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mins&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;END&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;EXECUTE&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;   &lt;span class="c1"&gt;-- returns NULL&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;NULL, as expected. The CASE sees the empty string and never reaches the cast.&lt;/p&gt;

&lt;p&gt;In n8n, the same node failed with &lt;code&gt;invalid input syntax for type integer: ""&lt;/code&gt;. I caught it only because I was watching that test step by step. The node had &lt;strong&gt;On Error&lt;/strong&gt; set to &lt;em&gt;Continue (using regular output)&lt;/em&gt;, so the execution finished green. No retry, no alert, and no callback saved.&lt;/p&gt;

&lt;h2&gt;
  
  
  What n8n actually sends
&lt;/h2&gt;

&lt;p&gt;The Postgres node runs on &lt;a href="https://github.com/vitaly-t/pg-promise/tree/11.9.1#query-formatting" rel="noopener noreferrer"&gt;pg-promise&lt;/a&gt;, pinned at 11.9.1 in n8n 2.41.6, the latest stable release as I write this. pg-promise has its own formatting engine, and its README says the engine is "used by default with all query methods, unless you opt out of it entirely via option &lt;code&gt;pgFormatting&lt;/code&gt;." n8n &lt;a href="https://github.com/n8n-io/n8n/blob/n8n%402.41.6/packages/nodes-base/nodes/Postgres/transport/index.ts#L122-L129" rel="noopener noreferrer"&gt;creates its pg-promise instance&lt;/a&gt; without that option.&lt;/p&gt;

&lt;p&gt;So the values never travel as bind parameters. pg-promise escapes them and writes them into the SQL text on the client, and Postgres receives one plain string. I reproduced it this morning against a local Postgres 16.15 with the same pg-promise version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;pgp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;pg-promise&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)();&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;sql&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;SELECT CASE WHEN $1 = '' THEN NULL ELSE make_interval(mins =&amp;gt; $1::int) END AS delay&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;pgp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;''&lt;/span&gt;&lt;span class="p"&gt;]));&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;CASE&lt;/span&gt; &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="n"&gt;make_interval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mins&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;END&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run that against the database and you get error &lt;code&gt;22P02&lt;/code&gt;, &lt;code&gt;invalid input syntax for type integer: ""&lt;/code&gt;. Send the same SQL with the same value as a real parameter (pg-promise's &lt;code&gt;ParameterizedQuery&lt;/code&gt;) and it returns &lt;code&gt;{"delay": null}&lt;/code&gt;, exactly like my psql test.&lt;/p&gt;

&lt;p&gt;To be fair to n8n, the part of the docs that says it "sanitizes data in query parameters, which prevents SQL injection" holds up. The escaping is real. What changes is how Postgres types the value.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the CASE didn't help
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;$1::int&lt;/code&gt; on a real parameter is a run-time cast. It runs when that branch runs, and in my test that branch never ran.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;''::int&lt;/code&gt; is something else. The Postgres docs on &lt;a href="https://www.postgresql.org/docs/current/sql-expressions.html#SQL-SYNTAX-TYPE-CASTS" rel="noopener noreferrer"&gt;type casts&lt;/a&gt; put it this way: "A cast applied to an unadorned string literal represents the initial assignment of a type to a literal constant value." Postgres hands the string to the integer input routine while it is still analyzing the statement, before it evaluates anything. An empty string isn't valid integer input, so the statement fails before the CASE gets a say. psql points the caret right at it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="n"&gt;ERROR&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;  &lt;span class="n"&gt;invalid&lt;/span&gt; &lt;span class="k"&gt;input&lt;/span&gt; &lt;span class="n"&gt;syntax&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="k"&gt;type&lt;/span&gt; &lt;span class="nb"&gt;integer&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nv"&gt;""&lt;/span&gt;
&lt;span class="n"&gt;LINE&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="n"&gt;make_interval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mins&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;E&lt;/span&gt;&lt;span class="p"&gt;...&lt;/span&gt;
                                                             &lt;span class="o"&gt;^&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The section on &lt;a href="https://www.postgresql.org/docs/current/sql-expressions.html#SYNTAX-EXPRESS-EVAL" rel="noopener noreferrer"&gt;expression evaluation rules&lt;/a&gt; says it plainly: CASE "is not a cure-all," and it "does not prevent early evaluation of constant subexpressions." Once your parameters are pasted in as literals, a lot more of your query is constant than it looks.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first half of the lesson: commas
&lt;/h2&gt;

&lt;p&gt;That workflow had already burned me once through the same Query Parameters field. Earlier, a free-text value with a comma in it shifted every parameter after it. The row was saved without an error, and every value after the comma had moved one column over.&lt;/p&gt;

&lt;p&gt;The reason is in &lt;a href="https://github.com/n8n-io/n8n/blob/n8n%402.41.6/packages/nodes-base/nodes/Postgres/v2/actions/database/executeQuery.operation.ts#L88-L114" rel="noopener noreferrer"&gt;executeQuery.operation.ts&lt;/a&gt;. For node version 2.5 and up (new nodes get 2.7), n8n evaluates each &lt;code&gt;{{ }}&lt;/code&gt; in the Query Parameters field separately. If the result is an array, it takes the items one by one. Anything else becomes a string, and unless that string happens to parse as JSON, it goes through &lt;a href="https://github.com/n8n-io/n8n/blob/n8n%402.41.6/packages/nodes-base/nodes/Postgres/v2/helpers/utils.ts#L46-L52" rel="noopener noreferrer"&gt;&lt;code&gt;stringToArray&lt;/code&gt;&lt;/a&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;String&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;,&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;map&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;entry&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;trim&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Split on commas, drop the empty pieces, trim the rest. The docs do describe the field as "a comma-separated list of values." They don't mention that the split happens after your expressions are evaluated, inside your data.&lt;/p&gt;

&lt;p&gt;Here is what that logic does to one item. I ran a copy of those exact functions:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Query Parameters: {{ $json.minutes }},{{ $json.note }},{{ $json.id }}&lt;/span&gt;
&lt;span class="c1"&gt;// item: { minutes: '', note: 'call back, after 5pm', id: 7 }&lt;/span&gt;
&lt;span class="c1"&gt;// values -&amp;gt; ["call back", "after 5pm", "7"]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The empty minutes disappeared, and the note became two values. One value vanished and one appeared, so the count still matched three placeholders and nothing failed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="s1"&gt;'call back'&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;text&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;minutes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'after 5pm'&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;text&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;note&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'7'&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
&lt;span class="c1"&gt;-- {"minutes":"call back","note":"after 5pm","id":7}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is the version that worries me more. The &lt;code&gt;''::int&lt;/code&gt; error at least stops the node. A shifted row looks fine until somebody reads it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Query Parameters is always one array expression.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;{{ [ $json.minutes, $json.note, $json.id ] }}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the whole field is one expression that returns an array, n8n uses that array as the parameter list and nothing gets split. Commas stay inside their value, and empty strings keep their position. It's also the form the n8n docs themselves use for &lt;code&gt;IN (...)&lt;/code&gt; lists.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Every cast on a value that can be empty goes through NULLIF.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;make_interval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mins&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;NULLIF&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;''&lt;/span&gt;&lt;span class="p"&gt;)::&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After formatting, the empty case becomes &lt;code&gt;make_interval(mins =&amp;gt; NULLIF('', '')::int)&lt;/code&gt;, which returns NULL, and with &lt;code&gt;'15'&lt;/code&gt; it returns a 15-minute interval. Both ran clean in the reproduction. NULLIF turns the empty string into NULL before the cast, and NULL cast to an integer is just NULL.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Dry-runs test the SQL that n8n will send, not a PREPARE.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I format the query with pg-promise first and run the output inside a transaction that I roll back:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;pgp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;pg-promise&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)();&lt;/span&gt;
&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;pgp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;format&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;sql&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;values&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt; &lt;span class="c1"&gt;// run it in psql between BEGIN; and ROLLBACK;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;My &lt;code&gt;PREPARE&lt;/code&gt; test was correct about a query that n8n never runs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. On Error → Continue stays off the node that writes the thing that matters.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The node that saves the callback now fails loudly, with Retry On Fail set to three tries. Continue is fine on a step whose failure can't hide anything downstream. On that node, it turned a failure into a quiet success.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;p&gt;If you write SQL in n8n's Postgres node, assume your values get pasted into the text as quoted literals, because they do. A cast on one of those literals happens when Postgres analyzes the statement, whatever branch it sits in. And a comma inside a value is a parameter boundary unless you pass an array.&lt;/p&gt;

&lt;p&gt;I build and maintain n8n integrations for businesses at &lt;a href="https://achiya-automation.com/en/services/integrations/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=n8n-postgres-query-parameters-literals" rel="noopener noreferrer"&gt;Achiya Automation&lt;/a&gt;, and these four rules now get checked before any Postgres node goes live.&lt;/p&gt;

&lt;p&gt;How do you fill Query Parameters in your n8n workflows, comma string or array? If it's the comma string, have you checked what happens when one of the values contains a comma?&lt;/p&gt;

</description>
      <category>n8n</category>
      <category>postgres</category>
      <category>automation</category>
      <category>discuss</category>
    </item>
    <item>
      <title>I filed it as a flaky test. It was a crash that try/catch can't reach.</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Sun, 04 Oct 2026 08:46:12 +0000</pubDate>
      <link>https://dev.to/achiya-automation/i-filed-it-as-a-flaky-test-it-was-a-crash-that-trycatch-cant-reach-5k5</link>
      <guid>https://dev.to/achiya-automation/i-filed-it-as-a-flaky-test-it-was-a-crash-that-trycatch-cant-reach-5k5</guid>
      <description>&lt;p&gt;&lt;a href="https://github.com/achiya-automation/safari-mcp" rel="noopener noreferrer"&gt;Safari MCP&lt;/a&gt;, the open-source server I maintain, runs one process that owns a local bridge port. Any other instance becomes a secondary: it proxies through that primary, and if the primary goes away, it takes the port over. One test covers exactly that handoff. Start a primary, start a secondary, stop the primary, wait for the secondary to say it's listening.&lt;/p&gt;

&lt;p&gt;On October 1 that test failed once on the Node 24 macOS runner, in a pull request run. The rerun passed, Node 20 and 22 passed, and I filed &lt;a href="https://github.com/achiya-automation/safari-mcp/issues/140" rel="noopener noreferrer"&gt;issue #140&lt;/a&gt; as a CI flake. The last line I wrote there was: "Filed to track the flake and the Node version it needs; no code change in this issue yet."&lt;/p&gt;

&lt;p&gt;Two days later the same test failed again, this time on main, a few minutes after the pull request with the same code had gone green on the same Node version.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the secondary died of
&lt;/h2&gt;

&lt;p&gt;The secondary never printed its listening line. Its stderr ended like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Uncaught: Error: setTypeOfService EINVAL
    at Socket.setTypeOfService (node:net:911:13)
    at writeH1 (node:internal/deps/undici/undici:8022:16)
    at Object.write (node:internal/deps/undici/undici:7758:18)
    at _resume (node:internal/deps/undici/undici:9656:54)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The code that was supposed to handle a missing primary looks like every health check you have written:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`http://127.0.0.1:&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/proxy-check`&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="na"&gt;signal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;AbortSignal&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;timeout&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="c1"&gt;// ...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="c1"&gt;// Nothing answers on the bridge port: the primary is gone. Take it over.&lt;/span&gt;
  &lt;span class="nf"&gt;takeOverThePort&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the primary is gone, &lt;code&gt;fetch()&lt;/code&gt; should reject and the catch should run. Instead the process ended.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the catch never ran
&lt;/h2&gt;

&lt;p&gt;The undici that ships inside Node 24 calls &lt;code&gt;socket.setTypeOfService()&lt;/code&gt; on every HTTP/1.1 request it writes. It's a quality-of-service hint and harmless almost everywhere. On macOS, though, the kernel refuses that option on a socket whose peer has just reset the connection. The call fails with &lt;code&gt;EINVAL&lt;/code&gt;, and Node turns that into a thrown error.&lt;/p&gt;

&lt;p&gt;The throw happens inside undici's write path, which runs from a socket I/O callback. Nothing there routes the error into the promise that &lt;code&gt;fetch()&lt;/code&gt; returned, so the &lt;code&gt;try/catch&lt;/code&gt; around &lt;code&gt;await fetch()&lt;/code&gt; never sees it. It becomes an uncaught exception, and an uncaught exception ends the process.&lt;/p&gt;

&lt;p&gt;So the crash needs three things at once: Node 24, macOS, and a peer that resets at the wrong moment. The test stops the primary a moment after the secondary comes up, close to the secondary's first check, which is about as close to the wrong moment as a test gets. In real use the race is rarer. But a secondary that dies exactly when it should take over is the failure the whole handoff exists to prevent.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fixed upstream, just not in Node 24
&lt;/h2&gt;

&lt;p&gt;undici fixed it in &lt;a href="https://github.com/nodejs/undici/pull/5547" rel="noopener noreferrer"&gt;nodejs/undici#5547&lt;/a&gt;, "fix(h1): ignore type of service errors", released in undici 8.8.0. It skips the call when nobody asked for a type of service, and it ignores the error when the call fails. Node 24 didn't get that change.&lt;/p&gt;

&lt;p&gt;I measured the difference by counting calls to &lt;code&gt;setTypeOfService&lt;/code&gt; during one plain &lt;code&gt;fetch()&lt;/code&gt; to a local server:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Node&lt;/th&gt;
&lt;th&gt;calls per fetch&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;24.21.0 (newest 24.x today)&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;26.10.0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;I'm not the only one who hit this. &lt;a href="https://github.com/get-bb/bb/pull/4627" rel="noopener noreferrer"&gt;get-bb/bb&lt;/a&gt;, &lt;a href="https://github.com/lzwwaylon/telomi/pull/30" rel="noopener noreferrer"&gt;telomi&lt;/a&gt; and &lt;a href="https://github.com/pingdotgg/t3code/pull/8389" rel="noopener noreferrer"&gt;t3code&lt;/a&gt; each have a pull request for the same uncaught &lt;code&gt;EINVAL&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  You can't schedule a race, so fail at the boundary instead
&lt;/h2&gt;

&lt;p&gt;My first instinct was to reproduce the real thing: a keep-alive server in a child process, one request to pool the socket, then a reset or a kill, then a second request onto the dead socket. Two harnesses and six timing variants on Node 24.21.0 produced zero &lt;code&gt;EINVAL&lt;/code&gt;s. The kernel race simply didn't happen on my machine, which is also why the test passed nearly every time in CI.&lt;/p&gt;

&lt;p&gt;What worked was failing at the exact boundary instead of waiting for the race. Replace the native method with one that throws what macOS throws, then let undici's own write path call it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;Socket&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:net&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;http&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;node:http&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="nx"&gt;Socket&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;prototype&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;setTypeOfService&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;function &lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nb"&gt;Object&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;assign&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;setTypeOfService EINVAL&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;code&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;EINVAL&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;server&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createServer&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;ok&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;listen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;127.0.0.1&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Promise&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;server&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;listening&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`http://127.0.0.1:&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;server&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;address&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;resolved:&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;text&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;rejected:&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;message&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On Node 24.21.0 that script prints neither line. It dies on the uncaught &lt;code&gt;EINVAL&lt;/code&gt; with exit code 1, &lt;code&gt;try/catch&lt;/code&gt; and all. On Node 26.10.0 it prints &lt;code&gt;resolved: ok&lt;/code&gt;, because undici there never makes the call.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;Upgrading Node is the real fix, but I don't pick my users' Node version, and the package supports 20 and up. So the server wraps the method once at startup and ignores &lt;code&gt;EINVAL&lt;/code&gt; only:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;ignoreTypeOfServiceEinval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;original&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;setTypeOfService&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;typeof&lt;/span&gt; &lt;span class="nx"&gt;original&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;function&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;original&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ignoresEinval&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;wrapped&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;function &lt;/span&gt;&lt;span class="p"&gt;(...&lt;/span&gt;&lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;original&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;apply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;args&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;catch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;?.&lt;/span&gt;&lt;span class="nx"&gt;code&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;EINVAL&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="k"&gt;this&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
      &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="nx"&gt;err&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="p"&gt;};&lt;/span&gt;
  &lt;span class="nx"&gt;wrapped&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;ignoresEinval&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;setTypeOfService&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;wrapped&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;With the error swallowed, undici carries on writing to a socket that's gone, the write fails the normal way, and &lt;code&gt;fetch()&lt;/code&gt; rejects into the catch that was there all along. Any other error from the method still throws. On Node 20 and 22 the method doesn't exist, so the wrapper does nothing.&lt;/p&gt;

&lt;p&gt;The test suite now runs the script above in a child process with the wrapper applied and asserts that the fetch resolves. I checked that the test can fail: with the wrapper replaced by a no-op, the child exits 1 on Node 24.21.0 and the test goes red. With the wrapper it passes, on my machine and on Node 24.20.0 in CI. The fix shipped in &lt;a href="https://github.com/achiya-automation/safari-mcp/releases/tag/v2.22.10" rel="noopener noreferrer"&gt;v2.22.10&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I got wrong
&lt;/h2&gt;

&lt;p&gt;The label. A test that fails because the process under test died of an uncaught exception isn't flaky in any sense that matters. The code crashed, and the test just couldn't make it crash often. "Flaky" told me to wait and see, and waiting meant the second report came from main instead of from me.&lt;/p&gt;

&lt;p&gt;If your project runs on Node 24 on macOS and calls &lt;code&gt;fetch()&lt;/code&gt; against something that can go away under you (a local daemon, a sidecar, a dev server you restart), search your crash logs for &lt;code&gt;setTypeOfService EINVAL&lt;/code&gt;. It won't show up as a rejected request. It shows up as a process that isn't running anymore.&lt;/p&gt;

</description>
      <category>node</category>
      <category>javascript</category>
      <category>testing</category>
      <category>debugging</category>
    </item>
    <item>
      <title>An LLM Heard Women in 71% of Our 'Male' AI Voice Replies</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Wed, 30 Sep 2026 09:02:02 +0000</pubDate>
      <link>https://dev.to/achiya-automation/an-llm-heard-women-in-71-of-our-male-ai-voice-replies-23e5</link>
      <guid>https://dev.to/achiya-automation/an-llm-heard-women-in-71-of-our-male-ai-voice-replies-23e5</guid>
      <description>&lt;p&gt;On 27 September a real caller on my business line spoke to our "male" AI voice using &lt;em&gt;at&lt;/em&gt;, the feminine Hebrew "you". The voice sits at roughly 135 Hz. We had chosen it with a pitch check, and the pitch check was the mistake.&lt;/p&gt;

&lt;p&gt;In English that would be a matter of taste. In Hebrew it's a correctness bug, because present-tense verbs carry gender. Our assistant ends its greeting with "recording and passing it on", which is מקליט ומעביר for a man and מקליטה ומעבירה for a woman. The system prompt fixes the grammatical gender, so the audio has to match it for the whole call. If it doesn't, the sentence contradicts itself.&lt;/p&gt;

&lt;p&gt;So we measured how often each voice actually sounds like the gender it's supposed to be.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The speaker was &lt;code&gt;gemini-3.8-live&lt;/code&gt; in &lt;code&gt;he-IL&lt;/code&gt;, one prebuilt voice per run, thinking budget 0.&lt;/li&gt;
&lt;li&gt;Google's Chirp 3 HD catalogue labels 16 of its voice names male. We ran 13. The other three had been recorded with feminine grammar in an earlier run, which would have confounded the test.&lt;/li&gt;
&lt;li&gt;The script was four caller turns in Hebrew, sent as text through the realtime input (that's how our speech-to-text feeds the model in production), with the production prompt built for masculine grammar. Three separate calls per voice.&lt;/li&gt;
&lt;li&gt;Every reply of 1.5 seconds or more was converted to what a phone line delivers, then sent to &lt;code&gt;gemini-3.5-flash&lt;/code&gt; at temperature 0 with a single one-word question.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The conversion and the judge call, trimmed from the harness:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;audioop&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;io&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;wave&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;soxr&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;google.genai&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;types&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;phone_wav&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pcm24&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;24 kHz model output -&amp;gt; 8 kHz mu-law round trip, i.e. what the caller hears.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="n"&gt;x&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;frombuffer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pcm24&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;dtype&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;int16&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;x8&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;soxr&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;resample&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;24000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;quality&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;HQ&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;x8&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;clip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x8&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;32768&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;32767&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;int16&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;x8&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;audioop&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;ulaw2lin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;audioop&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;lin2ulaw&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x8&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;tobytes&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;buf&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;io&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;BytesIO&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;wave&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;wb&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setnchannels&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setsampwidth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setframerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeframes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getvalue&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;Q&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Listen to the speaker in this phone recording. Does the voice sound like &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
     &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;a man or a woman? Answer with exactly one word: man, woman, or unclear.&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;judge&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pcm24&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;bytes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;aio&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;models&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate_content&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;models/gemini-3.5-flash&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;contents&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Part&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from_bytes&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nf"&gt;phone_wav&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pcm24&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;mime_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;audio/wav&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;Q&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;types&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;GenerateContentConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;text&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;lower&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;woman&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;woman&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="nf"&gt;else &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;man&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;man&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;a&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;unclear&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check for "woman" before "man". The second string is inside the first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The results
&lt;/h2&gt;

&lt;p&gt;One judging pass, run on 30 September over the recordings made on 27 September:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Voice&lt;/th&gt;
&lt;th&gt;man&lt;/th&gt;
&lt;th&gt;woman&lt;/th&gt;
&lt;th&gt;unclear&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Zubenelgenubi&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Algieba&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sadaltager&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Algenib&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Alnilam&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Charon&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Schedar&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Umbriel&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enceladus&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;8&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sadachbia&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Iapetus&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Puck&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;9&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fenrir&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;12&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That's 146 replies: 103 judged female (71%), 38 male (26%), 5 unclear. Only one voice was heard as a man more often than as a woman. None of the 39 calls got through without at least one reply judged female.&lt;/p&gt;

&lt;p&gt;These voices were speaking masculine Hebrew the whole time. If the judge were leaning on grammar, that would have pushed it toward "man", so the grammar can't explain the result.&lt;/p&gt;

&lt;h2&gt;
  
  
  The controls
&lt;/h2&gt;

&lt;p&gt;A judge that says "woman" to everything would produce the same table, so we checked it against clips where the answer is known:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Female voices from the same model, speaking feminine grammar: 34 of 34 replies judged female.&lt;/li&gt;
&lt;li&gt;Synthetic male callers from our test set, already at 8 kHz: judged male in 10 of 12 passes. Synthetic female callers: female in 6 of 6.&lt;/li&gt;
&lt;li&gt;The 135 Hz voice from the opening is called Pulcherrima, and Google's catalogue labels it female. We had it speak masculine grammar anyway. It was judged male in 0 of 11 replies.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The judge can recognise a man on a phone line. It just doesn't hear one in most of these voices.&lt;/p&gt;

&lt;h2&gt;
  
  
  Temperature 0 didn't make the judge deterministic
&lt;/h2&gt;

&lt;p&gt;This was the part I didn't expect. The same 11 Algieba clips were judged male 73% of the time on 27 September, then 45% and 55% on two passes on 30 September. One of the male control clips, an angry caller, came back man, woman, woman across three passes.&lt;/p&gt;

&lt;p&gt;What that means in practice:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;One judging pass is an anecdote. Run at least three and report the spread.&lt;/li&gt;
&lt;li&gt;Keep known-answer controls in every run, not only in the first one.&lt;/li&gt;
&lt;li&gt;Look at direction before magnitude. In both full passes, at least 12 of the 13 voices came out at or below 50% male. The exact percentages kept moving.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Telling the model to sound male made it worse
&lt;/h2&gt;

&lt;p&gt;The obvious fix was a line in the system prompt. Translated from the Hebrew, it said: "Your voice: a man's voice, low and steady. The whole call at exactly the same pitch, even in a short sentence, a thank-you or a goodbye. Never rise to a woman's pitch."&lt;/p&gt;

&lt;p&gt;Across the same 13 voices, the male share went from 26% without the line to 22% with it. Algieba, the only voice that had been winning, dropped to 11%. My guess, which I haven't tested, is that naming "a woman" in the instruction nudges the output toward one. We had shipped the line that afternoon and removed it the same evening.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pitch isn't the variable
&lt;/h2&gt;

&lt;p&gt;This is old news in speech science. Hillenbrand and Clark (&lt;em&gt;Attention, Perception &amp;amp; Psychophysics&lt;/em&gt;, 2009) resynthesised sentences to flip the speaker's apparent sex. Shifting pitch alone, or formants alone, usually failed. Shifting both worked about 82% of the time. An F0 threshold checks one of two cues, and it's the cue our 135 Hz voice already passed while still sounding like a woman.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we shipped
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Customers now choose from 11 voices, all female. The male voices are quarantined and get re-tested whenever the model changes.&lt;/li&gt;
&lt;li&gt;My own line keeps the one male voice, because I can live with the risk there.&lt;/li&gt;
&lt;li&gt;Since 28 September, the greeting on customer lines is spoken by the live model itself. Before that, a separate text-to-speech model rendered it under the same voice name, and you could hear the switch between the two engines.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you're building voice agents in a language with grammatical gender, measure the voice the way the caller will hear it, at 8 kHz rather than from the vendor's sample. Pick voices with a judge and controls, not with an F0 cutoff, and don't expect adjectives in the prompt to fix it.&lt;/p&gt;

&lt;p&gt;I build these systems at &lt;a href="https://achiya-automation.com/services/voice-bot/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=llm-judge-voice-gender" rel="noopener noreferrer"&gt;Achiya Automation&lt;/a&gt;, and the service in question is &lt;a href="https://onimli.com/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=llm-judge-voice-gender" rel="noopener noreferrer"&gt;Onimli&lt;/a&gt;, a Hebrew AI answering service for missed calls.&lt;/p&gt;

&lt;p&gt;If you've used an LLM as a judge for audio: how many passes do you run before you believe a number, and have you seen temperature 0 flip on identical clips like this?&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>testing</category>
      <category>discuss</category>
    </item>
    <item>
      <title>My 24-Month Audit Archive Was Keeping a Third of Each Day</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Mon, 28 Sep 2026 08:07:46 +0000</pubDate>
      <link>https://dev.to/achiya-automation/my-24-month-audit-archive-was-keeping-a-third-of-each-day-3h27</link>
      <guid>https://dev.to/achiya-automation/my-24-month-audit-archive-was-keeping-a-third-of-each-day-3h27</guid>
      <description>&lt;p&gt;On September 6 a disk alert fired on the server that runs my voice bots: 75% full. The culprit was &lt;code&gt;/var/log/auth.log&lt;/code&gt;, 763 MB after five days, on a box where it normally grows about 0.7 MB a week. Nearly every line started with &lt;code&gt;audispd:&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;I knew why. Six days earlier I had switched on auditd's syslog plugin across the fleet so a central collector could receive the audit trail. I compressed the file, wrote the cause down, and moved on.&lt;/p&gt;

&lt;p&gt;This morning I measured it on all eight servers. The growth turned out to be the smaller finding. The archive I built to keep audit logs for 24 months had been saving about a third of each day on my busiest server, from the week I set it up.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;Eight Ubuntu 24.04 servers running auditd 3.1.2, with 22 to 25 rules each. Most rules are file watches, &lt;code&gt;-w &amp;lt;path&amp;gt; -p wa&lt;/code&gt;, on config files and on each app's directory. Israel's data security regulations require keeping access logs for 24 months. auditd knows nothing about months. Out of the box it keeps a ring:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/audit/auditd.conf
&lt;/span&gt;&lt;span class="py"&gt;max_log_file&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;8          # MB&lt;/span&gt;
&lt;span class="py"&gt;num_logs&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;5&lt;/span&gt;
&lt;span class="py"&gt;max_log_file_action&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;ROTATE&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Five files of 8 MB each. When the ring is full, the oldest file is deleted. So on August 25 I added a cron job at 02:17 that forces a rotation, gzips every rotated file into &lt;code&gt;/var/log/audit-archive/&lt;/code&gt;, and deletes archives older than 730 days. Each archive's filename carries a hash of its content, so a file that moves from &lt;code&gt;audit.log.1&lt;/code&gt; to &lt;code&gt;audit.log.2&lt;/code&gt; isn't archived twice.&lt;/p&gt;

&lt;p&gt;On August 31 I enabled the syslog plugin:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# /etc/audit/plugins.d/syslog.conf
&lt;/span&gt;&lt;span class="py"&gt;active&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="py"&gt;args&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt; &lt;span class="s"&gt;LOG_INFO LOG_AUTHPRIV&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;LOG_AUTHPRIV&lt;/code&gt; is why the copies land in &lt;code&gt;auth.log&lt;/code&gt;. Ubuntu's rsyslog routes &lt;code&gt;auth,authpriv.*&lt;/code&gt; there, and logrotate keeps that file for 104 weeks.&lt;/p&gt;

&lt;h2&gt;
  
  
  What auth.log looks like now
&lt;/h2&gt;

&lt;p&gt;The same seven days of the week, before and after August 31, on six of the eight servers. The other two are dedicated to a client, so their numbers stay out of this post.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Server&lt;/th&gt;
&lt;th&gt;Aug 24–30&lt;/th&gt;
&lt;th&gt;Sep 21–27&lt;/th&gt;
&lt;th&gt;Growth&lt;/th&gt;
&lt;th&gt;audispd share&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;database&lt;/td&gt;
&lt;td&gt;2.3 MB&lt;/td&gt;
&lt;td&gt;734 MB&lt;/td&gt;
&lt;td&gt;×324&lt;/td&gt;
&lt;td&gt;99.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;website&lt;/td&gt;
&lt;td&gt;1.4 MB&lt;/td&gt;
&lt;td&gt;729 MB&lt;/td&gt;
&lt;td&gt;×524&lt;/td&gt;
&lt;td&gt;99.6%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;voice&lt;/td&gt;
&lt;td&gt;0.6 MB&lt;/td&gt;
&lt;td&gt;275 MB&lt;/td&gt;
&lt;td&gt;×441&lt;/td&gt;
&lt;td&gt;99.1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;WhatsApp gateway&lt;/td&gt;
&lt;td&gt;1.0 MB&lt;/td&gt;
&lt;td&gt;96 MB&lt;/td&gt;
&lt;td&gt;×101&lt;/td&gt;
&lt;td&gt;98.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;n8n&lt;/td&gt;
&lt;td&gt;1.3 MB&lt;/td&gt;
&lt;td&gt;88 MB&lt;/td&gt;
&lt;td&gt;×70&lt;/td&gt;
&lt;td&gt;95.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;chatwoot&lt;/td&gt;
&lt;td&gt;1.4 MB&lt;/td&gt;
&lt;td&gt;1.5 MB&lt;/td&gt;
&lt;td&gt;×1&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The last row is a control group: the plugin has been off on that server since September 8.&lt;/p&gt;

&lt;h2&gt;
  
  
  Who is writing all of this?
&lt;/h2&gt;

&lt;p&gt;Not people. &lt;code&gt;aureport&lt;/code&gt; shows which rule and which executable produced the events in the local ring:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;aureport &lt;span class="nt"&gt;--input-logs&lt;/span&gt; &lt;span class="nt"&gt;-k&lt;/span&gt; &lt;span class="nt"&gt;--summary&lt;/span&gt;   &lt;span class="c"&gt;# events per rule key&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;aureport &lt;span class="nt"&gt;--input-logs&lt;/span&gt; &lt;span class="nt"&gt;-x&lt;/span&gt; &lt;span class="nt"&gt;--summary&lt;/span&gt;   &lt;span class="c"&gt;# events per executable&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Keep &lt;code&gt;--input-logs&lt;/code&gt;. When stdin isn't a terminal, aureport reads stdin instead of the log files and prints &lt;code&gt;&amp;lt;no events of interest were found&amp;gt;&lt;/code&gt;. That cost me ten confused minutes this morning.&lt;/p&gt;

&lt;p&gt;On the &lt;strong&gt;database server&lt;/strong&gt;, 35,247 of the 35,659 events that matched a rule came from a single watch, &lt;code&gt;-w /opt/supabase -p wa&lt;/code&gt;, and the executable behind 35,082 of them was &lt;code&gt;postgres&lt;/code&gt;. The app directory contains the database's data volume, so every file Postgres opens for writing (paths like &lt;code&gt;base/5/2601&lt;/code&gt;) becomes an audit event.&lt;/p&gt;

&lt;p&gt;On the &lt;strong&gt;website&lt;/strong&gt;, 19,120 events hit the site's watch, and &lt;code&gt;rsync&lt;/code&gt; (14,988) and &lt;code&gt;rm&lt;/code&gt; (4,064) account for nearly all of them. That's the deploy. One deploy last night, between 23:07 and 23:12, produced 13,151 events made of 70,516 records.&lt;/p&gt;

&lt;p&gt;On the &lt;strong&gt;voice server&lt;/strong&gt; it's the same picture: 24,687 events on the app's watch, almost all from &lt;code&gt;rsync&lt;/code&gt;. The deploy there rewrites about 30,000 files on every run, changed or not.&lt;/p&gt;

&lt;p&gt;One write to a watched path isn't one line. It's a &lt;code&gt;SYSCALL&lt;/code&gt; record plus &lt;code&gt;CWD&lt;/code&gt;, one or two &lt;code&gt;PATH&lt;/code&gt; records and &lt;code&gt;PROCTITLE&lt;/code&gt;, 5.4 records per event on average in that deploy. The syslog plugin also forwards the &lt;code&gt;EOE&lt;/code&gt; (end of event) record, which auditd doesn't keep in its own log, so auth.log gets one more line per event than audit.log does.&lt;/p&gt;

&lt;h2&gt;
  
  
  The ring runs out before the archive runs
&lt;/h2&gt;

&lt;p&gt;A 40 MB ring is fine as long as 40 MB lasts longer than the gap between archive runs. Here's how long it lasts, from &lt;code&gt;aureport --input-logs -t&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;database:&lt;/strong&gt; the five files cover 13.5 hours&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;website:&lt;/strong&gt; around last night's deploy, three 8 MB files filled in 3 min 48 s, then 58 s, then 17 s&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;chatwoot:&lt;/strong&gt; the five files cover almost three days&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The archive ran once a night. Anything that left the ring before 02:17 was deleted before anyone saved it.&lt;/p&gt;

&lt;p&gt;So I measured the archive itself: for every archived file, the first record's timestamp and the file's mtime, merged into intervals and compared with each calendar day. Twenty-nine days, August 30 to September 27:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Server&lt;/th&gt;
&lt;th&gt;Mean coverage&lt;/th&gt;
&lt;th&gt;Worst day&lt;/th&gt;
&lt;th&gt;Complete days&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;database&lt;/td&gt;
&lt;td&gt;33.6%&lt;/td&gt;
&lt;td&gt;15.0%&lt;/td&gt;
&lt;td&gt;0 of 29&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;website&lt;/td&gt;
&lt;td&gt;52.6%&lt;/td&gt;
&lt;td&gt;9.4%&lt;/td&gt;
&lt;td&gt;9 of 29&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;voice&lt;/td&gt;
&lt;td&gt;75.0%&lt;/td&gt;
&lt;td&gt;22.0%&lt;/td&gt;
&lt;td&gt;16 of 29&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;chatwoot&lt;/td&gt;
&lt;td&gt;94.0%&lt;/td&gt;
&lt;td&gt;16.4%&lt;/td&gt;
&lt;td&gt;26 of 29&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the database server, the "24-month archive" holds 234 of the last 696 hours. On the website, deploy days drop to about 10%, because a deploy late in the evening pushes everything before it out of the ring. chatwoot's three incomplete days were maintenance days: a hardening pass, a Docker update and a fleet-wide reboot.&lt;/p&gt;

&lt;p&gt;The plugin didn't cause this. On August 30, the day before I enabled it, the database server's archive already held 28.4% of the day. Postgres had been filling the ring since the archive's first week.&lt;/p&gt;

&lt;h2&gt;
  
  
  The noise was the backup
&lt;/h2&gt;

&lt;p&gt;The &lt;code&gt;audispd&lt;/code&gt; lines in auth.log, the ones I spent September 6 compressing, are kept for 104 weeks. Since August 31 they have been the most complete local copy of the audit trail on these servers.&lt;/p&gt;

&lt;p&gt;They're not a perfect copy. I counted &lt;code&gt;SYSCALL&lt;/code&gt; records, one per event, by the timestamp inside each record, in both logs, for 23:04 to 02:00 last night:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;audit.log&lt;/code&gt;: 19,177 events&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;auth.log&lt;/code&gt;: 18,912 events (98.6%)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The missing 1.4% left a trail too. The plugin writes to &lt;code&gt;/dev/log&lt;/code&gt;, which on Ubuntu 24.04 is journald's socket, and journald forwards to rsyslog. During that window journald logged four &lt;code&gt;Forwarding to syslog missed N messages&lt;/code&gt; lines, 2,789 messages in total. The journal's own copy fared worse: between 23:00 and 23:20 its rate limiter logged &lt;code&gt;Suppressed … messages from auditd.service&lt;/code&gt; six times, 101,004 messages in all.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed today
&lt;/h2&gt;

&lt;p&gt;The smallest change that stops the loss is to archive more often without forcing a rotation every time. The script got a flag and a lock:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;exec &lt;/span&gt;9&amp;gt;/run/audit-archive.lock
flock &lt;span class="nt"&gt;-w&lt;/span&gt; 120 9

&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;&lt;span class="k"&gt;:-}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="s2"&gt;"--no-rotate"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  /sbin/auditctl &lt;span class="nt"&gt;--signal&lt;/span&gt; rotate &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null 2&amp;gt;&amp;amp;1 &lt;span class="o"&gt;||&lt;/span&gt; pkill &lt;span class="nt"&gt;-USR1&lt;/span&gt; auditd 2&amp;gt;/dev/null &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;true
  sleep &lt;/span&gt;2
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The cron file got a second line:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;17 2 * * * root /usr/local/bin/audit-archive.sh &amp;gt;/dev/null 2&amp;gt;&amp;amp;1
*/15 * * * * root /usr/local/bin/audit-archive.sh --no-rotate &amp;gt;/dev/null 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The nightly run still forces a rotation, so quiet servers get a daily file. The quarter-hourly run only picks up files auditd already rotated by size, which is exactly the part that used to fall off the end. The lock keeps the 02:15 and 02:17 runs from writing the same temporary file. I rolled it out to all eight servers with the old files backed up. The first run on the database server archived three files that would have been gone before tonight's 02:17.&lt;/p&gt;

&lt;p&gt;It doesn't close every hole. A burst bigger than about 32 MB inside fifteen minutes, like two website deploys back to back, can still push a file out before the archive sees it. The durable fixes all mean writing less, and I haven't made them yet:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Stop watching the database's data directory.&lt;/strong&gt; On August 25 I narrowed a &lt;code&gt;/var/lib/docker&lt;/code&gt; watch to &lt;code&gt;/var/lib/docker/containers&lt;/code&gt; for the same reason: Postgres block writes say nothing about who accessed what. The &lt;code&gt;/opt/supabase&lt;/code&gt; watch needs the same change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make the deploys skip unchanged files.&lt;/strong&gt; Thirty thousand rewritten files per deploy is an rsync problem, and auditd is only reporting it faithfully.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Give the syslog plugin its own facility,&lt;/strong&gt; &lt;code&gt;LOG_LOCAL6&lt;/code&gt; with its own file, so auth.log goes back to being about logins.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Check your own servers
&lt;/h2&gt;

&lt;p&gt;Three commands:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# how big is the ring?&lt;/span&gt;
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-E&lt;/span&gt; &lt;span class="s1"&gt;'^(max_log_file|num_logs|max_log_file_action) '&lt;/span&gt; /etc/audit/auditd.conf

&lt;span class="c"&gt;# how many hours does it hold right now?&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;aureport &lt;span class="nt"&gt;--input-logs&lt;/span&gt; &lt;span class="nt"&gt;-t&lt;/span&gt;

&lt;span class="c"&gt;# who is filling it?&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;aureport &lt;span class="nt"&gt;--input-logs&lt;/span&gt; &lt;span class="nt"&gt;-x&lt;/span&gt; &lt;span class="nt"&gt;--summary&lt;/span&gt; | &lt;span class="nb"&gt;head&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the second command shows fewer hours than the gap between your archive runs, your retention policy exists only in a document. I build and run automation infrastructure for small businesses at &lt;a href="https://achiya-automation.com/services/voice-bot/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=auditd-syslog-duplication" rel="noopener noreferrer"&gt;Achiya Automation&lt;/a&gt;, including the voice server in this post. This is the second time in five weeks that a retention number I had written down turned out to be one I had never measured.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;A question for anyone who archives auditd logs for compliance:&lt;/strong&gt; does your archive job run on rotation or on the clock, and how many hours does the ring actually hold on your busiest server? I'd especially like to hear from anyone running &lt;code&gt;max_log_file_action = keep_logs&lt;/code&gt; with a separate mover script, and whether that held up the first time the disk got tight.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>security</category>
      <category>devops</category>
      <category>discuss</category>
    </item>
    <item>
      <title>My sudo Logs Hit Nearly 10 GB. Session Count Told Me Nothing.</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Thu, 24 Sep 2026 07:41:26 +0000</pubDate>
      <link>https://dev.to/achiya-automation/my-sudo-logs-hit-10-gb-session-count-told-me-nothing-1bn5</link>
      <guid>https://dev.to/achiya-automation/my-sudo-logs-hit-10-gb-session-count-told-me-nothing-1bn5</guid>
      <description>&lt;p&gt;A disk-usage alert sent me looking at &lt;code&gt;/var/log/sudo-io&lt;/code&gt; on a Chatwoot production box. It held 5.9 GB and was adding roughly 654 MB every week. Nothing on that server had changed. No new services, no new admins, no new cron jobs.&lt;/p&gt;

&lt;p&gt;My first instinct was the wrong one: I assumed the server was simply busier than the others. It isn't. It runs the &lt;em&gt;fewest&lt;/em&gt; sudo sessions of any box in the fleet that logs them.&lt;/p&gt;

&lt;p&gt;Here is what the numbers look like, measured live across the whole fleet this morning.&lt;/p&gt;

&lt;h2&gt;
  
  
  The measurement
&lt;/h2&gt;

&lt;p&gt;Two commands. The first counts sessions, the second measures bytes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# how many sudo sessions were recorded&lt;/span&gt;
&lt;span class="nb"&gt;sudo &lt;/span&gt;find /var/log/sudo-io &lt;span class="nt"&gt;-name&lt;/span&gt; log &lt;span class="nt"&gt;-type&lt;/span&gt; f | &lt;span class="nb"&gt;wc&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt;

&lt;span class="c"&gt;# how much disk those sessions occupy&lt;/span&gt;
&lt;span class="nb"&gt;sudo du&lt;/span&gt; &lt;span class="nt"&gt;-sk&lt;/span&gt; /var/log/sudo-io
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ten hosts in my SSH config. Only four of them have &lt;code&gt;/var/log/sudo-io&lt;/code&gt; at all, and one of those four is dedicated to a client, so its numbers stay out of this post. The other five never had &lt;code&gt;log_output&lt;/code&gt; enabled, and one was unreachable during the run. That inconsistency is its own finding, and I'll come back to it.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Host&lt;/th&gt;
&lt;th&gt;Log volume&lt;/th&gt;
&lt;th&gt;Sessions&lt;/th&gt;
&lt;th&gt;KB per session&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;chatwoot&lt;/td&gt;
&lt;td&gt;5.48 GB&lt;/td&gt;
&lt;td&gt;16,168&lt;/td&gt;
&lt;td&gt;356&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;supabase&lt;/td&gt;
&lt;td&gt;2.33 GB&lt;/td&gt;
&lt;td&gt;56,794&lt;/td&gt;
&lt;td&gt;43&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;website&lt;/td&gt;
&lt;td&gt;1.71 GB&lt;/td&gt;
&lt;td&gt;14,289&lt;/td&gt;
&lt;td&gt;126&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;9.52 GB&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;87,251&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;114&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Read the last two columns together. &lt;code&gt;supabase&lt;/code&gt; recorded &lt;strong&gt;3.5x more sudo sessions&lt;/strong&gt; than &lt;code&gt;chatwoot&lt;/code&gt; and produced &lt;strong&gt;less than half&lt;/strong&gt; the volume. Per session, &lt;code&gt;chatwoot&lt;/code&gt; is 8.3x heavier.&lt;/p&gt;

&lt;p&gt;Session count and log size are not measuring the same thing, and I had been reading the wrong one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why bytes and sessions diverge
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;log_output&lt;/code&gt; in sudoers does not record &lt;em&gt;which command ran&lt;/em&gt;. It records everything the command writes to its pty. A short admin command that prints two lines costs a few hundred bytes. A command that streams a file costs the size of that file.&lt;/p&gt;

&lt;p&gt;So this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;systemctl restart nginx
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;costs almost nothing, while this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo cat&lt;/span&gt; /opt/chatwoot-backups/dump-2026-09-14.sql
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;costs the full size of the dump, every night. The dump gets written to disk a second time, into the audit log, carrying no information the original didn't already have.&lt;/p&gt;

&lt;p&gt;That was the leak. The offsite backup job pulls a database dump every night by reading it through sudo. &lt;code&gt;supabase&lt;/code&gt; and &lt;code&gt;website&lt;/code&gt; already had those reads exempted in &lt;code&gt;/etc/sudoers.d/iolog-exemptions&lt;/code&gt;. &lt;code&gt;chatwoot&lt;/code&gt; did not. The fourth host had no exemptions file at all. Same fleet, same backup design, three different configurations — because the exemption file was written by hand on each host, one host at a time, and nothing ever checked that they matched.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix is an exemption, not a shutdown
&lt;/h2&gt;

&lt;p&gt;The tempting fix is to turn &lt;code&gt;log_output&lt;/code&gt; off. Don't. The recording is the point; you want to know what a human typed at 2am on a box holding customer conversations. What you don't want is a byte-for-byte copy of files you already have.&lt;/p&gt;

&lt;p&gt;sudoers lets you carve out specific commands:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cmnd_Alias IOLOG_EXEMPT = /usr/bin/cat /opt/chatwoot-backups/*, \
                          /usr/bin/stat /opt/chatwoot-backups/*, \
                          /usr/bin/zcat /var/log/caddy/*, \
                          /usr/bin/tail /var/log/caddy/*

Defaults!IOLOG_EXEMPT !log_output
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The session still gets logged: who ran what, when. Only the byte stream is dropped.&lt;/p&gt;

&lt;p&gt;Two things worth saying out loud about this file:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Validate before you apply it.&lt;/strong&gt; &lt;code&gt;visudo -cf /etc/sudoers.d/iolog-exemptions&lt;/code&gt; parses the file without installing it. A syntax error in &lt;code&gt;sudoers.d&lt;/code&gt; can lock you out of sudo on a machine you may only reach through sudo. Keep a timestamped backup next to it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The paths matter more than the commands.&lt;/strong&gt; Exempting &lt;code&gt;/usr/bin/cat&lt;/code&gt; outright would let anyone pipe any file past the audit log. Exempting &lt;code&gt;cat&lt;/code&gt; &lt;em&gt;restricted to the backup directory&lt;/em&gt; keeps the hole the size of the actual problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  What ten days later looks like
&lt;/h2&gt;

&lt;p&gt;I applied the exemptions across all four hosts on September 14 and measured again today, September 24.&lt;/p&gt;

&lt;p&gt;At 654 MB/week, &lt;code&gt;chatwoot&lt;/code&gt; should have been near 6.8 GB by now. It reads 5.48 GB. The curve turned over. Two things are doing that work together. New writes stopped, and the 60-day retention cron keeps deleting old sessions, so I'd expect the number to keep falling into mid-October as the pre-fix backlog ages out. I'm not claiming the exemption alone recovered 1.3 GB. I'm claiming the growth stopped, and that's the part I was chasing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part I'd actually change
&lt;/h2&gt;

&lt;p&gt;The exemption fixed one symptom. The real defect is that five of ten hosts don't record sudo I/O, two of the four that do had drifted apart, and I only noticed because a disk filled up. A config that exists on some machines and not others isn't a policy, it's a habit that happened to repeat.&lt;/p&gt;

&lt;p&gt;So the check I added isn't "is the log too big." It's this, run across the fleet:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# does this host record sudo I/O at all, and is the exemption file present?&lt;/span&gt;
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rl&lt;/span&gt; log_output /etc/sudoers /etc/sudoers.d/ 2&amp;gt;/dev/null | &lt;span class="nb"&gt;head&lt;/span&gt; &lt;span class="nt"&gt;-1&lt;/span&gt;
&lt;span class="nb"&gt;test&lt;/span&gt; &lt;span class="nt"&gt;-f&lt;/span&gt; /etc/sudoers.d/iolog-exemptions &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"exemptions: yes"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"exemptions: MISSING"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two lines, and it answers the question the disk alert was too late to ask. This is the same reasoning behind most of the fleet automation I build at &lt;a href="https://achiya-automation.com/services/integrations/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=sudo-io-log-growth" rel="noopener noreferrer"&gt;achiya-automation.com&lt;/a&gt; — the alert that fires when a threshold breaks is always a worse version of the check that confirms a configuration is still what you think it is.&lt;/p&gt;

&lt;h2&gt;
  
  
  The metric to steal
&lt;/h2&gt;

&lt;p&gt;If you have &lt;code&gt;log_output&lt;/code&gt; enabled anywhere, don't look at total size and don't look at session count. Look at bytes per session:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo &lt;/span&gt;bash &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'kb=$(du -sk /var/log/sudo-io | cut -f1); \
  n=$(find /var/log/sudo-io -name log -type f | wc -l); \
  echo "$((kb / (n&amp;gt;0?n:1))) KB per session across $n sessions"'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Across my fleet the number is 114 KB. Anything in that range is ordinary admin work. A host sitting at 350+ has a command streaming a file through sudo, and you can find it in one pass. A host at 43 is doing plenty of work and costing you nothing.&lt;/p&gt;




&lt;p&gt;&lt;strong&gt;A question for anyone running &lt;code&gt;log_output&lt;/code&gt; in production:&lt;/strong&gt; what's your bytes-per-session number, and did the outlier turn out to be a backup job like mine, or something you'd never have guessed? I'm especially curious whether anyone exempts by &lt;em&gt;path&lt;/em&gt; the way I did here, or whether you found a cleaner pattern than hand-maintained &lt;code&gt;Cmnd_Alias&lt;/code&gt; blocks that drift between hosts.&lt;/p&gt;

</description>
      <category>linux</category>
      <category>devops</category>
      <category>sysadmin</category>
      <category>discuss</category>
    </item>
    <item>
      <title>My Monitoring Cron Never Ran Once. crontab -l Showed It Fine.</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Sun, 20 Sep 2026 06:49:59 +0000</pubDate>
      <link>https://dev.to/achiya-automation/my-monitoring-cron-never-ran-once-crontab-l-showed-it-fine-27hd</link>
      <guid>https://dev.to/achiya-automation/my-monitoring-cron-never-ran-once-crontab-l-showed-it-fine-27hd</guid>
      <description>&lt;p&gt;I write monitoring so I don't have to watch things by hand. So there is a special kind of dread in discovering that the monitor you installed to watch production has itself been dead since the day you installed it — and that everything you would normally check said it was fine.&lt;/p&gt;

&lt;p&gt;This one was dead for five days. Here's the autopsy, because the failure mode is boring, extremely common, and almost invisible.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;I run WhatsApp automation for businesses, and a session can silently drop: the container stays "up", the health check stays green, but the line stops receiving messages. So I built a small watchdog, &lt;code&gt;waha-session-watch&lt;/code&gt;, that polls the sessions API every ten minutes and alerts if a line goes quiet. It runs as a dedicated unprivileged user, &lt;code&gt;waha&lt;/code&gt;, and needs &lt;code&gt;sudo&lt;/code&gt; for one docker call.&lt;/p&gt;

&lt;p&gt;The crontab line looked completely ordinary:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;*/10 * * * * sudo /usr/local/bin/waha-session-watch &amp;gt;&amp;gt; /opt/waha-watch/cron.log 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Install it, watch &lt;code&gt;cron.log&lt;/code&gt; fill up, move on. Except &lt;code&gt;cron.log&lt;/code&gt; never filled up. And I didn't notice, because I wasn't looking at &lt;code&gt;cron.log&lt;/code&gt;. I was looking at the tool's own log, &lt;code&gt;watch.log&lt;/code&gt;, which had plenty of recent entries. Green across the board.&lt;/p&gt;

&lt;p&gt;Five days later a production line went dark and nothing alerted. That's when I actually went digging.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first lie: &lt;code&gt;crontab -l&lt;/code&gt;
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;crontab &lt;span class="nt"&gt;-l&lt;/span&gt; &lt;span class="nt"&gt;-u&lt;/span&gt; waha
&lt;span class="k"&gt;*&lt;/span&gt;/10 &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="k"&gt;*&lt;/span&gt; &lt;span class="nb"&gt;sudo&lt;/span&gt; /usr/local/bin/waha-session-watch &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; /opt/waha-watch/cron.log 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There it is. It's installed. &lt;code&gt;crontab -l&lt;/code&gt; printing your line feels like proof the job runs. It is not. It only proves the line is &lt;em&gt;stored&lt;/em&gt;. Whether cron can actually &lt;em&gt;execute&lt;/em&gt; it is a completely separate question, and the answer here was no. Not once.&lt;/p&gt;

&lt;h2&gt;
  
  
  The second lie: a log full of runs
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;watch.log&lt;/code&gt; had entries. Recent ones. So the tool clearly &lt;em&gt;was&lt;/em&gt; running... right?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-09-08 21:40:12  OK  all sessions healthy
2026-09-08 21:52:03  OK  all sessions healthy
2026-09-08 21:59:47  OK  all sessions healthy
2026-09-08 22:22:15  OK  all sessions healthy
2026-09-08 22:32:51  OK  all sessions healthy
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Look at the minutes: &lt;code&gt;:40&lt;/code&gt;, &lt;code&gt;:52&lt;/code&gt;, &lt;code&gt;:59&lt;/code&gt;, &lt;code&gt;:22&lt;/code&gt;, &lt;code&gt;:32&lt;/code&gt;. A &lt;code&gt;*/10&lt;/code&gt; cron job fires at &lt;code&gt;:00&lt;/code&gt;, &lt;code&gt;:10&lt;/code&gt;, &lt;code&gt;:20&lt;/code&gt;, &lt;code&gt;:30&lt;/code&gt;, &lt;code&gt;:40&lt;/code&gt;, on the round ten. These timestamps are ragged. Every single one of them was &lt;strong&gt;me&lt;/strong&gt;, running the tool by hand from the terminal on install day to confirm it worked. Not one line in that log came from cron.&lt;/p&gt;

&lt;p&gt;That's the trap in one sentence: &lt;strong&gt;a log full of successful runs is not proof of scheduled runs.&lt;/strong&gt; Manual runs and cron runs land in the same file and look identical unless you read the clock.&lt;/p&gt;

&lt;h2&gt;
  
  
  The tell: one line in syslog
&lt;/h2&gt;

&lt;p&gt;Here's where it actually gives itself up. I grepped the system log for the job:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;grep &lt;/span&gt;CRON /var/log/syslog | &lt;span class="nb"&gt;grep &lt;/span&gt;waha
... &lt;span class="o"&gt;(&lt;/span&gt;CRON&lt;span class="o"&gt;)&lt;/span&gt; info &lt;span class="o"&gt;(&lt;/span&gt;No MTA installed, discarding output&lt;span class="o"&gt;)&lt;/span&gt;
... CMD &lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;sudo&lt;/span&gt; /usr/local/bin/waha-session-watch &lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt; /opt/waha-watch/cron.log 2&amp;gt;&amp;amp;1&lt;span class="o"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;No MTA installed, discarding output&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That message is the whole case. cron ran the job, the job produced output on &lt;strong&gt;stderr&lt;/strong&gt;, and cron tried to email that output to the user, because that's what cron does with anything a job writes when it can't otherwise capture it. There's no mail server, so it threw the text away. Which means: something printed an error, every ten minutes, into a void, for five days.&lt;/p&gt;

&lt;p&gt;If your command's output were really being captured by &lt;code&gt;&amp;gt;&amp;gt; cron.log&lt;/code&gt;, cron would have nothing to mail and this line would never appear. &lt;strong&gt;&lt;code&gt;(No MTA installed, discarding output)&lt;/code&gt; sitting next to a CMD is always a sign the command printed something nobody saw.&lt;/strong&gt; It is one of the highest-signal lines in all of syslog and almost nobody greps for it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The root cause: who opens the &lt;code&gt;&amp;gt;&amp;gt;&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;The bug is the redirect. Specifically, &lt;em&gt;who&lt;/em&gt; opens it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sudo /usr/local/bin/waha-session-watch &amp;gt;&amp;gt; /opt/waha-watch/cron.log 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's intuitive to read this as "run the whole thing as root and append to the log." That's not what happens. Redirections are set up by the &lt;strong&gt;shell that launches the command, before the command is exec'd&lt;/strong&gt;, and cron runs this line as the &lt;code&gt;waha&lt;/code&gt; user. So the &lt;code&gt;&amp;gt;&amp;gt; /opt/waha-watch/cron.log&lt;/code&gt; is opened with &lt;code&gt;waha&lt;/code&gt;'s permissions, &lt;em&gt;before&lt;/em&gt; &lt;code&gt;sudo&lt;/code&gt; ever runs.&lt;/p&gt;

&lt;p&gt;And the directory:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-ld&lt;/span&gt; /opt/waha-watch
drwxr-xr-x 2 root root 4096 Sep  8 /opt/waha-watch
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;root:root&lt;/code&gt;, mode &lt;code&gt;755&lt;/code&gt;. The &lt;code&gt;cron.log&lt;/code&gt; file didn't exist yet, and &lt;code&gt;waha&lt;/code&gt; has no write permission in a &lt;code&gt;755&lt;/code&gt; directory it doesn't own. So the shell can't create the file, the redirection fails, the shell bails out &lt;strong&gt;before executing &lt;code&gt;sudo&lt;/code&gt;&lt;/strong&gt;, and the command never runs at all. cron dutifully mails the "permission denied" to nobody.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;sudo&lt;/code&gt; was never the problem. The unprivileged redirect target in front of it was. The privilege you granted the command is irrelevant when the shell dies setting up its plumbing first.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;Anticlimactic, as these always are:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sudo touch&lt;/span&gt; /opt/waha-watch/cron.log
&lt;span class="nb"&gt;sudo chown &lt;/span&gt;waha:waha /opt/waha-watch/cron.log
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Create the log file up front, owned by the user cron runs as. First round-numbered timestamp showed up ten minutes later and the watchdog has been real ever since. No code changed. The script was always fine.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I actually changed
&lt;/h2&gt;

&lt;p&gt;The fix took ten seconds. The lesson took five days, so it's worth more than the fix.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;After installing any cron job or systemd timer, verify the first &lt;em&gt;scheduled&lt;/em&gt; run in two independent places:&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The scheduler's own record — &lt;code&gt;grep CRON /var/log/syslog&lt;/code&gt; and confirm a &lt;code&gt;CMD&lt;/code&gt; line &lt;strong&gt;with no &lt;code&gt;discarding output&lt;/code&gt; next to it&lt;/strong&gt;, at a round-numbered minute.&lt;/li&gt;
&lt;li&gt;The tool's own log — an entry whose timestamp lands on the schedule, not a ragged manual one.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If those two don't agree, you have a job that is stored but not running. &lt;code&gt;crontab -l&lt;/code&gt; is not on this list on purpose: it can only ever confirm storage.&lt;/p&gt;

&lt;p&gt;And the redirect rule specifically: &lt;strong&gt;if a scheduled command redirects into a log, create that file ahead of time, owned by the user the scheduler runs as.&lt;/strong&gt; Don't assume &lt;code&gt;sudo&lt;/code&gt; on the command buys you anything for the &lt;code&gt;&amp;gt;&amp;gt;&lt;/code&gt;. That file is opened by the caller, not the callee.&lt;/p&gt;

&lt;p&gt;I build this kind of infrastructure for a living — production WhatsApp automation for Israeli businesses at &lt;a href="https://achiya-automation.com/services/whatsapp-bot/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=cron-silent-death-redirect" rel="noopener noreferrer"&gt;Achiya Automation&lt;/a&gt; — and this watchdog exists precisely so a silent line doesn't go unnoticed. The irony of the watchdog itself going silent for five days is not lost on me. That's exactly why the two-place verification is now non-negotiable in my install checklist.&lt;/p&gt;

&lt;h2&gt;
  
  
  Your turn
&lt;/h2&gt;

&lt;p&gt;What's the longest one of your "installed" monitors has been quietly dead before you noticed, and what finally gave it away? I'm collecting the tells that actually work in the wild, beyond the syslog one above. Drop yours below. #discuss&lt;/p&gt;

</description>
      <category>devops</category>
      <category>linux</category>
      <category>cron</category>
      <category>monitoring</category>
    </item>
    <item>
      <title>I Audited 38 Production Containers. Zero Use depends_on restart.</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Thu, 17 Sep 2026 07:52:44 +0000</pubDate>
      <link>https://dev.to/achiya-automation/i-audited-38-production-containers-zero-use-dependson-restart-3a2o</link>
      <guid>https://dev.to/achiya-automation/i-audited-38-production-containers-zero-use-dependson-restart-3a2o</guid>
      <description>&lt;p&gt;My n8n upgrade runbook has a line in it that I wrote by hand, sometime in 2026:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;After the main container reports healthy, run &lt;code&gt;docker restart n8n-docker-caddy-n8n-worker-1&lt;/code&gt;.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;I have run that line at every version bump for months. It works. I never questioned it.&lt;/p&gt;

&lt;p&gt;Last week I finally read the Compose specification closely enough to notice that I had hand-implemented a field that has existed since Compose &lt;strong&gt;2.17.0&lt;/strong&gt;, released in 2023. The field is &lt;code&gt;depends_on.&amp;lt;service&amp;gt;.restart&lt;/code&gt;. My runbook is a human being standing in for one boolean.&lt;/p&gt;

&lt;p&gt;So I went and measured how widespread the gap actually is — not in a demo repo, in the fleet that runs my clients' work.&lt;/p&gt;

&lt;h2&gt;
  
  
  The measurement
&lt;/h2&gt;

&lt;p&gt;Eight servers, read-only, on 2026-09-17. I pulled two things from every running container: its healthcheck configuration, and the &lt;code&gt;com.docker.compose.depends_on&lt;/code&gt; label that Compose v2 writes at create time. That label is the ground truth — it records the resolved dependency as &lt;code&gt;service:condition:restart&lt;/code&gt;, so I didn't have to trust the YAML on disk matching what is actually running.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;c &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;docker ps &lt;span class="nt"&gt;--format&lt;/span&gt; &lt;span class="s1"&gt;'{{.Names}}'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;docker inspect &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$c&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--format&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="s1"&gt;'{{index .Config.Labels "com.docker.compose.project"}}|{{.Name}}|{{index .Config.Labels "com.docker.compose.depends_on"}}'&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Results across &lt;strong&gt;38 running containers&lt;/strong&gt;, in &lt;strong&gt;14 Compose projects&lt;/strong&gt;, on &lt;strong&gt;8 servers&lt;/strong&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;count&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Containers with a healthcheck&lt;/td&gt;
&lt;td&gt;33 / 38&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Containers with any &lt;code&gt;depends_on&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;7 / 38&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Of those 7, using &lt;code&gt;condition: service_healthy&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;7 / 7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Of those 7, using &lt;code&gt;restart: true&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0 / 7&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Every dependency I had bothered to declare was declared &lt;em&gt;well&lt;/em&gt; — all seven wait for health, none of them settle for &lt;code&gt;service_started&lt;/code&gt;. And every single one of them ends in &lt;code&gt;:false&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That trailing &lt;code&gt;false&lt;/code&gt; is the restart flag. It means: when Compose updates the dependency, do not restart me.&lt;/p&gt;

&lt;p&gt;The 31 containers with no &lt;code&gt;depends_on&lt;/code&gt; at all are a separate confession, and I'll come back to them.&lt;/p&gt;

&lt;h2&gt;
  
  
  What depends_on actually promises
&lt;/h2&gt;

&lt;p&gt;The long syntax takes three sub-options, and it is worth being precise about the scope of each:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;condition: service_started&lt;/code&gt; — the dependency's container has been created and started. Says nothing about the process inside it.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;condition: service_healthy&lt;/code&gt; — the dependency's healthcheck has passed. This is the one everybody reaches for.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;condition: service_completed_successfully&lt;/code&gt; — the dependency ran to exit 0. For migration jobs and seeders.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;restart: true&lt;/code&gt; — &lt;em&gt;"Compose restarts this service after it updates the dependency service."&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;required: false&lt;/code&gt; — downgrade a missing dependency from an error to a warning.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Read the &lt;code&gt;restart&lt;/code&gt; line again. It is scoped to &lt;strong&gt;Compose-controlled updates&lt;/strong&gt; — the &lt;code&gt;docker compose up -d&lt;/code&gt; that replaces a container with a new image. It is explicitly not about runtime crashes; your restart policy covers those.&lt;/p&gt;

&lt;p&gt;So &lt;code&gt;condition&lt;/code&gt; and &lt;code&gt;restart&lt;/code&gt; answer two genuinely different questions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;condition&lt;/strong&gt; → in what order do these containers come up &lt;em&gt;the first time&lt;/em&gt;?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;restart&lt;/strong&gt; → what happens to me when the thing underneath me gets &lt;em&gt;replaced&lt;/em&gt;?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A stack with &lt;code&gt;condition: service_healthy&lt;/code&gt; everywhere and &lt;code&gt;restart&lt;/code&gt; nowhere is fully ordered on a cold boot and fully unordered on every deploy after that. Which, on a server that boots once a year and deploys twice a month, means the feature is protecting the rare case and sitting out the common one.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where condition alone was not enough
&lt;/h2&gt;

&lt;p&gt;Two dated incidents from this fleet, both of which I originally filed as something other than what they were.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2026-07-31 — the healthy database that wasn't ready.&lt;/strong&gt; Upgrading n8n from 2.30.6 to 2.32.6 on the queue-mode server. The worker's declared dependency reads, to this day, &lt;code&gt;postgres:service_healthy,redis:service_healthy&lt;/code&gt;. Postgres was genuinely healthy. The worker still died — stack trace in &lt;code&gt;MigrationExecutor.executePendingMigrations&lt;/code&gt;, then &lt;code&gt;unhealthy&lt;/code&gt; and stuck there.&lt;/p&gt;

&lt;p&gt;The reason is that &lt;code&gt;service_healthy&lt;/code&gt; asked Postgres whether it was accepting connections, and Postgres honestly said yes. Nobody asked the question that mattered: &lt;em&gt;has the schema been migrated to the new version yet?&lt;/em&gt; The main container and the worker both came up against the same database and both started running migrations. Main won. The worker crashed on a half-applied schema.&lt;/p&gt;

&lt;p&gt;There is no healthcheck you can write on Postgres that answers "is the application's schema current", because the answer doesn't live in Postgres — it lives in whichever sibling container is doing the migrating. The fix I actually deployed was that runbook line: wait for main, restart the worker. Which is &lt;code&gt;depends_on: { n8n: { condition: service_healthy, restart: true } }&lt;/code&gt; performed manually.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2026-09-04 — the update that kept everything running and broke one thing anyway.&lt;/strong&gt; Docker engine 29.7.2 → 29.8.0 across the fleet. &lt;code&gt;live-restore&lt;/code&gt; did its job: every container survived, 8→8, 5→5, 3→3. Except &lt;code&gt;supabase-docker-proxy&lt;/code&gt;, which went &lt;code&gt;unhealthy&lt;/code&gt; with:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ERROR Docker API proxy error error="dial unix /var/run/docker.sock: connect: connection refused"
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It mounts &lt;code&gt;/var/run/docker.sock&lt;/code&gt;. Replacing the daemon replaces the socket; the process inside kept holding the old one and never reconnected. &lt;code&gt;live-restore&lt;/code&gt; keeps the &lt;em&gt;container&lt;/em&gt; running — it cannot heal a client that lost its connection to the API.&lt;/p&gt;

&lt;p&gt;The tell was ugly: &lt;code&gt;docker ps&lt;/code&gt; showed &lt;code&gt;Up 3 days (unhealthy)&lt;/code&gt;. The uptime was true. It really had not fallen over. &lt;code&gt;FailingStreak&lt;/code&gt; was 75, and that was the number that said when it actually broke. A &lt;code&gt;docker restart&lt;/code&gt; fixed it in fifteen seconds.&lt;/p&gt;

&lt;p&gt;Note what both incidents have in common. Neither container crashed. Both needed to be restarted &lt;em&gt;because something underneath them changed&lt;/em&gt;, and in both cases the thing that noticed was a person.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 31 with no depends_on
&lt;/h2&gt;

&lt;p&gt;The honest part of the audit. Four of them are application containers in production: the Rails and Sidekiq containers of two self-hosted Chatwoot instances. They have no declared dependency on their own Postgres or Redis. On a cold boot they race, lose, crash, and get resurrected by &lt;code&gt;restart: always&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That works. It has worked for a year. It is also the reason that stack's real recovery time is invisible — the recovery is a crash loop that happens to terminate, and nothing records how long it took.&lt;/p&gt;

&lt;p&gt;While I was in there I measured the other number that deploys care about. Those Rails containers run &lt;code&gt;start_period: 60s&lt;/code&gt;, &lt;code&gt;interval: 60s&lt;/code&gt;, &lt;code&gt;retries: 3&lt;/code&gt;. Add it up: a Rails container that comes back up broken can present as &lt;code&gt;starting&lt;/code&gt;, then &lt;code&gt;healthy&lt;/code&gt;-adjacent, for up to &lt;strong&gt;four minutes&lt;/strong&gt; before Docker is willing to call it &lt;code&gt;unhealthy&lt;/code&gt;. If your deploy script polls health to decide whether to roll back, that is the size of your blind spot.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed
&lt;/h2&gt;

&lt;p&gt;For the dependencies that already exist, adding one line each:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;n8n-worker&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;depends_on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;postgres&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;condition&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;service_healthy&lt;/span&gt;
      &lt;span class="na"&gt;n8n&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;condition&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;service_healthy&lt;/span&gt;
        &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;     &lt;span class="c1"&gt;# ← replace main, and I get restarted after it&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two changes in that block, not one. The &lt;code&gt;restart: true&lt;/code&gt; is the field I had been simulating by hand. The new &lt;code&gt;n8n&lt;/code&gt; entry is the fix for the 2026-07-31 incident: the worker's real dependency was never Postgres, it was &lt;em&gt;whoever migrates Postgres&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;Run this on your own fleet before you assume you're clean — it reads the resolved labels, not the YAML you think is deployed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;for &lt;/span&gt;c &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;docker ps &lt;span class="nt"&gt;--format&lt;/span&gt; &lt;span class="s1"&gt;'{{.Names}}'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;&lt;span class="nv"&gt;d&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;docker inspect &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$c&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;--format&lt;/span&gt; &lt;span class="s1"&gt;'{{index .Config.Labels "com.docker.compose.depends_on"}}'&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;
  &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nt"&gt;-n&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$c&lt;/span&gt;&lt;span class="s2"&gt; -&amp;gt; &lt;/span&gt;&lt;span class="nv"&gt;$d&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Anything ending in &lt;code&gt;:false&lt;/code&gt; is a container that will keep running against a dependency you just replaced.&lt;/p&gt;

&lt;p&gt;Two caveats I'd rather state than have corrected in the comments. First, &lt;code&gt;restart: true&lt;/code&gt; is not free — it widens each deploy, because more containers cycle. On a stack where the dependency changes often, that is the trade you are making. Second, it is still not zero-downtime; it is &lt;em&gt;ordered&lt;/em&gt; downtime. Real zero-downtime means two versions serving at once behind a proxy, and Compose is not the tool for that. What this fixes is the cheaper, more common failure: the deploy that finishes green while something behind it is quietly talking to a socket that no longer exists.&lt;/p&gt;

&lt;p&gt;Most of the infrastructure work at &lt;a href="https://achiya-automation.com/services/custom-solutions/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=docker-compose-zero-downtime" rel="noopener noreferrer"&gt;Achiya Automation&lt;/a&gt; looks like this — not clever, just the boring field nobody read the docs for.&lt;/p&gt;

&lt;h2&gt;
  
  
  The question
&lt;/h2&gt;

&lt;p&gt;I'm curious whether the &lt;code&gt;:false&lt;/code&gt; ratio is a me problem or an everyone problem. Run the loop above and post your two numbers: &lt;strong&gt;how many of your running containers declare &lt;code&gt;depends_on&lt;/code&gt; at all, and how many of those end in &lt;code&gt;:true&lt;/code&gt;?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;And if you're above zero on the second number — what made you go looking? I only found it because I noticed I'd written the feature into a runbook by hand.&lt;/p&gt;

</description>
      <category>docker</category>
      <category>devops</category>
      <category>compose</category>
      <category>reliability</category>
    </item>
    <item>
      <title>I Measured 27,257 MCP Connections. The p90 Session Waits 35s.</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Wed, 16 Sep 2026 05:45:26 +0000</pubDate>
      <link>https://dev.to/achiya-automation/i-measured-27257-mcp-connections-the-p90-session-waits-35s-55dp</link>
      <guid>https://dev.to/achiya-automation/i-measured-27257-mcp-connections-the-p90-session-waits-35s-55dp</guid>
      <description>&lt;p&gt;In July I wrote a 56-line shell script called &lt;code&gt;mcp-optional&lt;/code&gt; whose entire job was to &lt;strong&gt;remove&lt;/strong&gt; two MCP servers from my config.&lt;/p&gt;

&lt;p&gt;Not fix them. Remove them.&lt;/p&gt;

&lt;p&gt;Both were stdio servers, so each one spawns its own Node process &lt;em&gt;per session&lt;/em&gt;. With around 13 Claude Code sessions open on a 16GB M4, that was roughly 1.4GB of duplicated RAM, which meant swap thrash, which meant a hot laptop. Disabling them by default and re-enabling on demand fixed it.&lt;/p&gt;

&lt;p&gt;I filed that under "memory problem, solved" and stopped thinking about it.&lt;/p&gt;

&lt;p&gt;This morning, chasing something unrelated, I started reading the MCP connection logs Claude Code leaves on disk. It turns out one of those two servers wasn't just the RAM tax. It's also the slowest thing I connect to, by a wide margin — and the reason isn't its code at all.&lt;/p&gt;

&lt;p&gt;Here's what 35 days of logs actually say.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I measured
&lt;/h2&gt;

&lt;p&gt;Claude Code writes a JSONL log per MCP server, per session, under &lt;code&gt;~/Library/Caches/claude-cli-nodejs/&amp;lt;project&amp;gt;/mcp-logs-&amp;lt;server&amp;gt;/&lt;/code&gt;. Two lines matter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Starting connection with timeout of 30000ms
Connection established with capabilities: {"hasTools":true,...}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Pair them inside one file and you get a real connection latency. I did that across every project directory on this machine:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;33,599 log files&lt;/strong&gt;, 2026-08-13 → 2026-09-16 (35 days)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;38,876&lt;/strong&gt; connection attempts started&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;27,257&lt;/strong&gt; clean start→established pairs, across &lt;strong&gt;22 servers&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;2,888&lt;/strong&gt; sessions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is one developer's machine, not a lab. That's the point — it's the distribution you actually live in.&lt;/p&gt;

&lt;h2&gt;
  
  
  The headline numbers
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;percentile&lt;/th&gt;
&lt;th&gt;connect time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;median&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;582 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p75&lt;/td&gt;
&lt;td&gt;1,650 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p90&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3,678 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p95&lt;/td&gt;
&lt;td&gt;6,428 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;14,049 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The median looks great. Under six tenths of a second — nobody would ever file a bug.&lt;/p&gt;

&lt;p&gt;But &lt;strong&gt;39% of connections take longer than a second&lt;/strong&gt;, and the p99 is fourteen seconds. Against a client timeout of 30,000ms, the tail is not a rounding error. It's most of the way to the wall.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause #1: it's OAuth, not your server
&lt;/h2&gt;

&lt;p&gt;This is the finding that reframed the whole thing for me. I split every connection by whether a token refresh happened inside that same connection window:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;n&lt;/th&gt;
&lt;th&gt;median&lt;/th&gt;
&lt;th&gt;p90&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;token refresh in window&lt;/td&gt;
&lt;td&gt;748&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3,170 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;9,619 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;no refresh&lt;/td&gt;
&lt;td&gt;26,509&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;550 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;3,400 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;5.8× slower.&lt;/strong&gt; Same servers, same network, same machine. The only difference is whether the client had to go get a new access token first.&lt;/p&gt;

&lt;p&gt;Your MCP server didn't take three seconds to start. Your MCP server took 550ms to start, and an OAuth round-trip took the other 2.6 seconds while the server sat there doing nothing.&lt;/p&gt;

&lt;p&gt;That's why "why is this server slow" is usually the wrong question. Only 2.7% of my connections hit a refresh — but that 2.7% is where a disproportionate share of the visible pain lives, because it's the path that turns a fast connection into a slow one non-deterministically. You can't reproduce it on demand, so you blame the server.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause #2: transport, with a twist
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;transport&lt;/th&gt;
&lt;th&gt;n&lt;/th&gt;
&lt;th&gt;median&lt;/th&gt;
&lt;th&gt;p90&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;HTTP&lt;/td&gt;
&lt;td&gt;19,894&lt;/td&gt;
&lt;td&gt;676 ms&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3,395 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;stdio&lt;/td&gt;
&lt;td&gt;7,363&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;222 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4,693 ms&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;stdio is 3× faster at the median — no surprise, there's no network. But look at p90: &lt;strong&gt;stdio is worse in the tail.&lt;/strong&gt; Spawning a process has a floor of roughly nothing and a ceiling of "npm decided to do something." An HTTP server that's already running is slower on average and far more predictable.&lt;/p&gt;

&lt;p&gt;If you're choosing a transport, that trade is the actual decision. Not "stdio is faster."&lt;/p&gt;

&lt;h2&gt;
  
  
  Cause #3: nobody connects to one server
&lt;/h2&gt;

&lt;p&gt;Per-connection numbers hide the thing that actually costs you time, which is fan-out. Per session on this machine:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;median 8&lt;/strong&gt; MCP servers, max 16&lt;/li&gt;
&lt;li&gt;median total connect time: &lt;strong&gt;5.9 s&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;p90: &lt;strong&gt;35.5 s&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;p99: &lt;strong&gt;81.8 s&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;33.7%&lt;/strong&gt; of sessions spend more than 10 seconds just connecting&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That p90 is the number that made me stop and re-read my own script. A third of the time I open a session, I'm waiting a meaningful fraction of a minute before the first token — not because any single server is broken, but because eight of them each drew from that fat tail and nothing amortizes.&lt;/p&gt;

&lt;p&gt;Worst offender by median, unsurprisingly: one of the two servers I'd already disabled for RAM reasons, at &lt;strong&gt;2,261 ms median and 10,413 ms at p90&lt;/strong&gt;. I removed it in July for the wrong reason and got the right outcome.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 29.7% nobody talks about
&lt;/h2&gt;

&lt;p&gt;Of 38,876 connection attempts, &lt;strong&gt;11,556 never logged an established line at all&lt;/strong&gt; — 29.7%.&lt;/p&gt;

&lt;p&gt;Some of that is log rotation cutting a file mid-handshake, so treat it as an upper bound rather than a failure rate. But it isn't evenly spread: a single remote connector accounts for 7,422 of them on its own. That's not noise, that's one integration failing over and over while everything upstream stays quiet about it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The trap I almost published
&lt;/h2&gt;

&lt;p&gt;My first pass had a max connect time of &lt;strong&gt;119 seconds&lt;/strong&gt; and a p99 of 14.5s, and I nearly wrote a paragraph about it.&lt;/p&gt;

&lt;p&gt;Then I re-read the client's own log line: &lt;code&gt;timeout of 30000ms&lt;/code&gt;. A connection cannot establish at 119 seconds if the client gives up at 30. Those samples weren't slow connections — they were a closed laptop. The wall clock kept running through sleep; the connection didn't.&lt;/p&gt;

&lt;p&gt;So I capped every measurement at the client's declared 30s timeout and threw the rest out. That removed &lt;strong&gt;24 of 27,281 samples — 0.09%&lt;/strong&gt;. Every number above is post-cap.&lt;/p&gt;

&lt;p&gt;I mention it because the uncapped version would have been a better story and a false one. If you run this on your own machine, cap it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually helps
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Count your servers before you tune any of them.&lt;/strong&gt; Median 8, p90 35 seconds. Removing one mediocre server beats optimizing a good one.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make heavy servers opt-in.&lt;/strong&gt; That's all &lt;code&gt;mcp-optional&lt;/code&gt; does — &lt;code&gt;claude mcp remove&lt;/code&gt; by default, &lt;code&gt;claude mcp add&lt;/code&gt; when a task needs it. It's the highest-leverage 56 lines I have written this year, and I wrote it for the wrong reason.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Look at your refresh path, not your server code.&lt;/strong&gt; If connections are erratic rather than uniformly slow, that's the OAuth signature: 550ms most of the time, 3.2s when the token expired.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prefer an already-running HTTP server for anything heavy.&lt;/strong&gt; Worse median, much better p90.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Read the logs you already have.&lt;/strong&gt; Nothing here required instrumentation. It was sitting in a cache directory the whole time.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The reason I care about any of this: at &lt;a href="https://achiya-automation.com/services/ai-agents/?utm_source=devto&amp;amp;utm_medium=article&amp;amp;utm_campaign=mcp-server-startup-time" rel="noopener noreferrer"&gt;Achiya Automation&lt;/a&gt; I run agent tooling against real client infrastructure all day, and a 35-second session start that lands on one session in ten — with a third of them over 10 seconds — is a tax I pay dozens of times a day without ever seeing it on a bill.&lt;/p&gt;

&lt;h2&gt;
  
  
  One question
&lt;/h2&gt;

&lt;p&gt;If you've got MCP servers configured right now — &lt;strong&gt;how many, and when did you last check whether you still use all of them?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I'd genuinely like to know whether median-8 is normal or whether I'm the outlier. And if you've measured a refresh-vs-no-refresh split on a different client, I want to see that number, because I only have one machine to look at.&lt;/p&gt;

</description>
      <category>mcp</category>
      <category>ai</category>
      <category>devops</category>
      <category>performance</category>
    </item>
    <item>
      <title>My WhatsApp Lines Died Silently. Docker Said 'Healthy'.</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Mon, 14 Sep 2026 05:47:45 +0000</pubDate>
      <link>https://dev.to/achiya-automation/3-of-7-whatsapp-lines-died-in-9-days-docker-said-healthy-6le</link>
      <guid>https://dev.to/achiya-automation/3-of-7-whatsapp-lines-died-in-9-days-docker-said-healthy-6le</guid>
      <description>&lt;p&gt;At 08:33 one morning I ran &lt;code&gt;GET /api/sessions&lt;/code&gt; against my own WhatsApp gateway and read the line I least wanted to read:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"name"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"session_&amp;lt;redacted&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"status"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"FAILED"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"me"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"pushName"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;business name&amp;gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"id"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;redacted&amp;gt;@c.us"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;&lt;span class="w"&gt;
  &lt;/span&gt;&lt;span class="nl"&gt;"timestamps"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"activity"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"&amp;lt;days ago&amp;gt;"&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The last activity on that session was days old. The line had been dead for days, and every dashboard I own was green.&lt;/p&gt;

&lt;p&gt;I run WhatsApp automation for Israeli small businesses. Each client's business number lives as a session inside a self-hosted &lt;a href="https://waha.devlike.pro/" rel="noopener noreferrer"&gt;WAHA&lt;/a&gt; container, which bridges their line into a CRM and a pile of n8n workflows. Several businesses, one container.&lt;/p&gt;

&lt;p&gt;Over a few days in September, several of those lines went down. Here is what they taught me, and what I check now.&lt;/p&gt;

&lt;h2&gt;
  
  
  The health check was answering the wrong question
&lt;/h2&gt;

&lt;p&gt;My monitor checked that the &lt;code&gt;waha&lt;/code&gt; container was running and responsive. It was. It stayed running, responsive and &lt;code&gt;healthy&lt;/code&gt; through every one of those outages, because a container that is perfectly happy to serve you a list of dead sessions is, by every definition Docker cares about, working.&lt;/p&gt;

&lt;p&gt;This is the ordinary liveness-versus-readiness mistake, except the readiness that mattered was not the service's. It was the readiness of the independent sessions &lt;em&gt;inside&lt;/em&gt; the service, each of which can fail on its own, for unrelated reasons, without the process noticing or caring.&lt;/p&gt;

&lt;p&gt;One client's line failed one evening and I found it days later; in between, the container reported itself healthy hundreds of times.&lt;/p&gt;

&lt;h2&gt;
  
  
  The failure that looks like nothing at all
&lt;/h2&gt;

&lt;p&gt;This failure mode is the reason I am writing this.&lt;/p&gt;

&lt;p&gt;The gateway logs a &lt;code&gt;device removed stream error&lt;/code&gt; when the linked device is removed from the phone, either on purpose or while someone clears out old sessions. Which of the two it was hardly matters.&lt;/p&gt;

&lt;p&gt;What matters is what the business sees afterwards: nothing. Messages keep arriving on the owner's phone exactly as before. Customers still get replies, because a human is reading the phone. From the business's side the line looks completely, visibly fine.&lt;/p&gt;

&lt;p&gt;Only the CRM goes blind. Only the automations stop firing. The scheduled reminders, the routing, the logging, the follow-ups — all silently not happening, behind a line that looks perfectly alive to the only person in a position to notice.&lt;/p&gt;

&lt;p&gt;If your failure mode is invisible to the user &lt;em&gt;and&lt;/em&gt; invisible to your health check, you do not have a monitoring gap. You have no monitoring.&lt;/p&gt;

&lt;h2&gt;
  
  
  You cannot always tell whose line just died
&lt;/h2&gt;

&lt;p&gt;When I sat down to write the alerting, I hit a problem I had not anticipated: a dead session may not be able to tell you who it belongs to.&lt;/p&gt;

&lt;p&gt;While a session is &lt;code&gt;WORKING&lt;/code&gt;, &lt;code&gt;/api/sessions&lt;/code&gt; gives you &lt;code&gt;me.id&lt;/code&gt; (the number) and &lt;code&gt;me.pushName&lt;/code&gt; (the business name). When a session is torn down by an unlink, those fields are wiped. The identity disappears at precisely the moment you need it to address an alert.&lt;/p&gt;

&lt;p&gt;I want to be careful here, because I wrote this down as an absolute rule in my own notes last week and it is not one. Checking the live API this morning while drafting this post, the currently-failed line &lt;em&gt;does&lt;/em&gt; still carry its &lt;code&gt;me&lt;/code&gt; block, as you can see in the JSON above. So it depends on how the session died: a removed device wipes the identity, a connection-level failure does not.&lt;/p&gt;

&lt;p&gt;Which is worse than either rule on its own, because it means you cannot depend on the field being there. The fix is the same either way: keep your own roster. My watcher rewrites a &lt;code&gt;roster.json&lt;/code&gt; on every pass — but only from sessions currently in &lt;code&gt;WORKING&lt;/code&gt;, because a single pass during an outage would otherwise overwrite good identity with nulls.&lt;/p&gt;

&lt;p&gt;And you cannot recover the mapping from billing, either. I searched our accounting system for these business numbers and found none: what sits in an invoice is the owner's personal mobile, which is a different number from the business line the bot runs on. The mapping is manual, or it does not exist.&lt;/p&gt;

&lt;h2&gt;
  
  
  The watchdog that never ran
&lt;/h2&gt;

&lt;p&gt;On 8 September I built the thing that should have caught all of this: a watcher on a ten-minute cron that diffs session states, notifies the business owner when their own line drops, and sends them a pairing code to reconnect it themselves.&lt;/p&gt;

&lt;p&gt;I tested it. Ten test scenarios, three real bugs caught before install. I watched it run. I marked the problem solved.&lt;/p&gt;

&lt;p&gt;It did not run once from cron until 13 September.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;*/10 * * * * sudo /usr/local/bin/waha-session-watch &amp;gt;&amp;gt; /opt/waha-watch/cron.log 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;/opt/waha-watch&lt;/code&gt; is owned by &lt;code&gt;root&lt;/code&gt;, mode 755. The cron user is &lt;code&gt;waha&lt;/code&gt;. The redirect is evaluated by the shell &lt;em&gt;before&lt;/em&gt; &lt;code&gt;sudo&lt;/code&gt; runs, as the unprivileged user, so creating &lt;code&gt;cron.log&lt;/code&gt; failed, so the shell exited, so the command never executed at all. Cron dutifully tried to mail me the error and found no MTA installed, so it dropped it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;(CRON) info (No MTA installed, discarding output)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Five days of no monitoring, hidden behind a crontab entry that reads perfectly.&lt;/p&gt;

&lt;p&gt;What fooled me during the install was &lt;code&gt;watch.log&lt;/code&gt;. It had runs in it — 21:40, 21:52, 21:59, 22:22. Non-round minutes, every one of them, because they were my own manual test runs. Scheduled runs land on round minutes. I looked at a log full of evidence that the tool worked and read it as evidence that the &lt;em&gt;schedule&lt;/em&gt; worked.&lt;/p&gt;

&lt;p&gt;Another line fell into that exact blind window. The first genuinely scheduled run found it immediately, days late.&lt;/p&gt;

&lt;p&gt;The fix was &lt;code&gt;touch cron.log &amp;amp;&amp;amp; chown waha:waha&lt;/code&gt;. That is the entire fix. The lesson costs more than the fix: &lt;strong&gt;that &lt;code&gt;No MTA installed, discarding output&lt;/code&gt; line in syslog is never noise.&lt;/strong&gt; It means your scheduled command printed something nobody has ever read.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I check now
&lt;/h2&gt;

&lt;p&gt;Four things changed:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Health-check the units, not the process.&lt;/strong&gt; The container being up says nothing about the sessions inside it. Whatever your service multiplexes — tenants, sessions, connections, device links — that is the thing with a state worth alerting on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Verify the first scheduled run, in two places.&lt;/strong&gt; Not "is it in &lt;code&gt;crontab -l&lt;/code&gt;", which was true and meaningless for five days. Check syslog for the &lt;code&gt;CMD&lt;/code&gt; line, and check the tool's own log for an entry on a round minute. If a cron entry redirects to a file, create the file first, owned by the cron user.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Keep a persistent roster, written only from healthy state.&lt;/strong&gt; Never let a failure pass overwrite identity you will need to send the alert.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Alert the person who can actually fix it.&lt;/strong&gt; A dropped line is repaired by someone holding the phone, not by me. The watcher now messages the business owner directly with a pairing code, inside working hours, once per incident. The one-message-per-incident rule matters as much as the alert: the fastest way to get a monitoring channel ignored is to send the same thing every ten minutes.&lt;/p&gt;

&lt;p&gt;It matters that this is WhatsApp specifically. Bezeq's 2025 Internet Report — the annual survey of Israeli internet use, published in December 2025 — &lt;a href="https://www.bezeq.co.il/bloghome/Digital/internetreport_2025/" rel="noopener noreferrer"&gt;notes in passing&lt;/a&gt; that reaching 90% WhatsApp usage in Israel took about a decade. For a small business here, that line is not a channel. It is the front door. Which is exactly why &lt;a href="https://achiya-automation.com/services/whatsapp-bot/customer-service/" rel="noopener noreferrer"&gt;the customer-service bots running on those lines&lt;/a&gt; need a liveness signal independent of the phone in the owner's pocket, because the owner's pocket will never report the outage. It does not look like one from there.&lt;/p&gt;

&lt;p&gt;Several lines, a few days, one container that never once said anything was wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  The question I am actually asking
&lt;/h2&gt;

&lt;p&gt;I want to hear about the second category specifically — not the outage your monitoring missed, but the one where &lt;strong&gt;the user experience stayed completely normal while the system underneath was dead&lt;/strong&gt;, so nobody had any reason to report it.&lt;/p&gt;

&lt;p&gt;What was the failure, and what signal finally caught it? I am collecting these, because I suspect the honest answer for most of them is "a human noticed weeks later," and I would like to be proven wrong.&lt;/p&gt;

</description>
      <category>devops</category>
      <category>monitoring</category>
      <category>postmortem</category>
      <category>whatsapp</category>
    </item>
    <item>
      <title>My safety guard was checked in 6 places. Nothing ever set it.</title>
      <dc:creator>אחיה כהן</dc:creator>
      <pubDate>Fri, 11 Sep 2026 06:43:45 +0000</pubDate>
      <link>https://dev.to/achiya-automation/my-safety-guard-was-checked-in-6-places-nothing-ever-set-it-55cd</link>
      <guid>https://dev.to/achiya-automation/my-safety-guard-was-checked-in-6-places-nothing-ever-set-it-55cd</guid>
      <description>&lt;p&gt;A user of my Safari MCP server — the user was me, from a scheduled run — hit a failure I could not talk my way out of.&lt;/p&gt;

&lt;p&gt;Right after the background daemon restarted, the server opened a browser tab and handed back a capability token for it. The token is how every later command proves it is allowed to touch that tab, because the whole point of the ownership system is that the server must never act on a tab &lt;em&gt;you&lt;/em&gt; opened. Thirty-three seconds later, the very next command was refused:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Tab safety: receipt is forged, stale, ambiguous, or not valid for this origin
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Forged. By me. For a tab I had just opened, four lines earlier.&lt;/p&gt;

&lt;p&gt;Worse, it was unrecoverable from the client side. The token could not be rotated. Switching to the tab by index was refused — "not opened by this MCP session." Closing it was refused too, which is &lt;em&gt;correct&lt;/em&gt;, since closing is the one action you cannot undo and ownership could no longer be proven. So the tab sat there, stranded and unusable, and the agent had to reopen its work somewhere else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Ruling things out
&lt;/h2&gt;

&lt;p&gt;I had two obvious hypotheses and both died in the same session.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"A newer tab invalidates the older token."&lt;/strong&gt; Minted a token, opened a second tab, used the first token afterwards. Worked fine.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;"Switching by index corrupts the next command."&lt;/strong&gt; Switched by index, ran a command with no token at all. Worked fine.&lt;/p&gt;

&lt;p&gt;The one thing every failure had in common was timing. Everything opened once the daemon had been up for a couple of minutes behaved perfectly. Only the tabs created inside the restart window went bad. The first tab creation in that window took &lt;strong&gt;22 seconds&lt;/strong&gt; — the extension's background worker was still cold-starting — and that number turned out to be the entire story.&lt;/p&gt;

&lt;h2&gt;
  
  
  The actual mechanism
&lt;/h2&gt;

&lt;p&gt;Tokens are stamped with a browser-session epoch: a random identifier that ties a token to one continuous run of the browser. If the browser restarts, the epoch rotates, and every old token is correctly rejected instead of being allowed to point at a recycled tab id. Good design. The epoch lives in session storage, which is the authority.&lt;/p&gt;

&lt;p&gt;Here is the function that reads it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;_ensureBrowserSessionEpoch&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="nx"&gt;_browserSessionStorageAvailable&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;throw&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;Error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="s2"&gt;Browser session storage is unavailable; refusing durable tab authority&lt;/span&gt;&lt;span class="dl"&gt;"&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sr"&gt;/^&lt;/span&gt;&lt;span class="se"&gt;[&lt;/span&gt;&lt;span class="sr"&gt;a-f0-9&lt;/span&gt;&lt;span class="se"&gt;]{36}&lt;/span&gt;&lt;span class="sr"&gt;$/&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;test&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;_browserSessionEpoch&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;_browserSessionEpoch&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="c1"&gt;// ... otherwise read storage, or mint and persist a new epoch&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That last &lt;code&gt;if&lt;/code&gt; is the problem. Once the worker has resolved an epoch into memory, it never consults storage again for the rest of its life. The cached copy is treated as the truth.&lt;/p&gt;

&lt;p&gt;So when the epoch rotates in storage while the worker is still alive — which is exactly what a daemon restart plus an extension re-handshake does — the worker keeps stamping brand-new tokens with the &lt;strong&gt;old&lt;/strong&gt; epoch, and writes them to durable storage that way. Nothing complains, because nothing is looking.&lt;/p&gt;

&lt;p&gt;The complaint arrives later, from the one path that &lt;em&gt;does&lt;/em&gt; read storage. When the worker eventually cold-starts, it rehydrates its tables and filters the stored tokens:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;receiptsFresh&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;receiptEnvelope&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
  &lt;span class="nx"&gt;receiptEnvelope&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;version&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nx"&gt;_TAB_RECEIPTS_VERSION&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;
  &lt;span class="nx"&gt;receiptEnvelope&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;browserEpoch&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="nx"&gt;browserEpoch&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="p"&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every stored record carries the stale epoch. &lt;code&gt;receiptsFresh&lt;/code&gt; is false. The entire record array is discarded, the in-memory tables are cleared and repopulated from that empty set, and every token minted during the stale window dies at the same instant. Which is precisely what the report showed: two tabs, created five seconds apart, rejected together.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that actually bothers me
&lt;/h2&gt;

&lt;p&gt;There was a guard for this. It just could not fire.&lt;/p&gt;

&lt;p&gt;The code tracks a counter whose whole job is to notice that the browser-session identity changed underneath an in-flight operation. It is compared at six call sites, across the three functions that own the epoch:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2885, 2896, 2920   _ensureBrowserSessionEpoch
3043               _hydrateOwnedTabs
3075, 3095         _persistOwnedTabs
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each comparison guards a real invariant, and each one has a hand-written error message explaining what went wrong: &lt;em&gt;"Browser-session identity changed during initialization."&lt;/em&gt; &lt;em&gt;"Browser session changed during tab ownership recovery."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The variable is assigned exactly once, on line 2819:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;_browserEpochGeneration&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the declaration. Nothing increments it. Ever.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;$ &lt;/span&gt;&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-nE&lt;/span&gt; &lt;span class="s1"&gt;'_browserEpochGeneration\s*(\+\+|--|=[^=])'&lt;/span&gt; extension/background.js
2819:let _browserEpochGeneration &lt;span class="o"&gt;=&lt;/span&gt; 0&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;One hit, and it is the &lt;code&gt;let&lt;/code&gt;. Six comparisons against a constant, plus three sites that dutifully capture it into a local first.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why it survived review, including mine
&lt;/h2&gt;

&lt;p&gt;I have been staring at this file for months, and I want to be precise about why I did not see it, because "you should have looked harder" is not a lesson anyone can apply.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The guards read as protection.&lt;/strong&gt; Every site looks like careful concurrency code. There is a captured local, an await, and a re-check afterwards — the exact shape you are trained to pattern-match as &lt;em&gt;correct&lt;/em&gt;. Reading for shape rather than for reachability, it passes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Three of the six had a live half.&lt;/strong&gt; They are compound:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;generation&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="nx"&gt;_browserEpochGeneration&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nx"&gt;browserEpoch&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="nx"&gt;_browserSessionEpoch&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The second clause works. So those three guards do fire, sometimes, on the epoch comparison alone — which is exactly why this bug shows up at rehydration time rather than never at all. Half of the protection was real, and the working half masked the dead half. If all six had been dead, the failure would have been louder and I would have found it sooner.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The other three, the fully dead ones, were the ones that mattered most.&lt;/strong&gt; They sit inside &lt;code&gt;_ensureBrowserSessionEpoch&lt;/code&gt; and have no second clause, because that function is the thing that &lt;em&gt;sets&lt;/em&gt; the value they would compare against. Those three were supposed to catch the rotation &lt;strong&gt;at mint time&lt;/strong&gt;, before a doomed token is ever handed to a caller. Instead the rotation is caught at rehydration, long after the bad tokens are in the client's hands and in durable storage.&lt;/p&gt;

&lt;p&gt;A dead guard is worse than a missing one. A missing guard is an obvious gap. A dead guard is a gap plus a comment asserting the gap is covered, and it makes the next person to read the file — me, repeatedly — stop looking.&lt;/p&gt;

&lt;h2&gt;
  
  
  What would have caught it
&lt;/h2&gt;

&lt;p&gt;Not a unit test. Every behavioural test over these paths passes, and would still pass with all six comparisons deleted, because the code under test never rotates the epoch. The defect is an &lt;strong&gt;omission&lt;/strong&gt;, and you cannot observe an omission by exercising what is there.&lt;/p&gt;

&lt;p&gt;This project already had one test for exactly that category, written after a different omission bug, and its own comment says so:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;This is a source-level check on purpose: the failure is &lt;em&gt;omission&lt;/em&gt; at a call site, which no behavioural test over the existing sites can see.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That test reads the source as text and asserts a property about the call sites. The same trick works here, and the rule is embarrassingly simple:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A variable that is compared must be assigned somewhere other than its declaration.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;That is a grep. It is cheap enough to run over every mutable module-level variable in a file, and it would have failed loudly the day this landed. Not a linter rule I had to invent either — "assigned once, read many, never reassigned" is the shape, and the moment you write it down you realise how many state machines and generation counters and dirty-flags could rot this way without a single test turning red.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I have not done
&lt;/h2&gt;

&lt;p&gt;I have not shipped the fix.&lt;/p&gt;

&lt;p&gt;The change itself is small: stop trusting the cached epoch, re-verify it against storage, and increment the counter when the two disagree — which lights up all six existing guards at once. I know what to type.&lt;/p&gt;

&lt;p&gt;But this is the tab-ownership guard, and the reason it exists in the first place is that early versions of this server genuinely clicked into and closed people's tabs. The failure mode of getting the fix wrong is inverted from the current bug: today a good token is rejected rarely, in a narrow restart window. An over-eager rotation check would reject good tokens &lt;em&gt;routinely&lt;/em&gt;. That is a strictly worse trade, and I am not making it against a theory. I want a staged reproduction that captures the epoch from both storage and the persisted record at mint time, and shows them differing, before I touch the file.&lt;/p&gt;

&lt;p&gt;So for now the bug is open, documented, and worse than the fix — on purpose. The guard is dead either way; at least now it is dead and labelled.&lt;/p&gt;




&lt;p&gt;If you want to look: the server is &lt;a href="https://github.com/achiya-automation/safari-mcp" rel="noopener noreferrer"&gt;safari-mcp&lt;/a&gt;, and the full analysis lives on &lt;a href="https://github.com/achiya-automation/safari-mcp/issues/105" rel="noopener noreferrer"&gt;issue #105&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The question I am genuinely stuck on:&lt;/strong&gt; how do you catch a dead guard in your own code? Grepping for compared-but-never-assigned finds this exact shape, but it is one instance of a bigger family — the &lt;code&gt;if&lt;/code&gt; that can never be true, the &lt;code&gt;catch&lt;/code&gt; that can never be reached, the flag nobody flips. Has anyone found a check for that family that is cheap enough to run on every commit and quiet enough that people do not start ignoring it?&lt;/p&gt;

</description>
      <category>javascript</category>
      <category>debugging</category>
      <category>opensource</category>
      <category>testing</category>
    </item>
  </channel>
</rss>
