<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: ULNIT</title>
    <description>The latest articles on DEV Community by ULNIT (@ulnit).</description>
    <link>https://dev.to/ulnit</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F592406%2Fd91a2be3-1b3c-43c3-a231-712206ed4013.png</url>
      <title>DEV Community: ULNIT</title>
      <link>https://dev.to/ulnit</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ulnit"/>
    <language>en</language>
    <item>
      <title>My Backups Passed Every Check for 14 Months. The First Real Restore Said 'database disk image is malformed'</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Sun, 15 Nov 2026 01:02:52 +0000</pubDate>
      <link>https://dev.to/ulnit/my-backups-passed-every-check-for-14-months-the-first-real-restore-said-database-disk-image-is-p08</link>
      <guid>https://dev.to/ulnit/my-backups-passed-every-check-for-14-months-the-first-real-restore-said-database-disk-image-is-p08</guid>
      <description>&lt;p&gt;Fourteen months. 428 nightly backups. A green checkmark in my monitoring dashboard every single morning.&lt;/p&gt;

&lt;p&gt;I had never restored one.&lt;/p&gt;

&lt;p&gt;Not once. Not to a scratch directory, not to a spare Pi, not even to &lt;code&gt;/tmp&lt;/code&gt; to poke at it. I had a cron job, a USB SSD, a retry wrapper, an alert if the job didn't run, and a &lt;code&gt;test -s&lt;/code&gt; check that confirmed the file existed and wasn't empty.&lt;/p&gt;

&lt;p&gt;That is not a backup system. That is a file-copying system with good marketing.&lt;/p&gt;

&lt;p&gt;This is the story of the afternoon I finally tested it — what broke, why my verification was worse than nothing, and the 40-line script that replaced 14 months of false confidence.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why I never tested it
&lt;/h2&gt;

&lt;p&gt;The honest answer is embarrassing: I was afraid of what I'd find, and I had a rationalization ready.&lt;/p&gt;

&lt;p&gt;"Restoring takes downtime." It didn't. It's a Pi and a spare SSD; I could restore to a temp directory while everything kept running.&lt;/p&gt;

&lt;p&gt;"The snapshots are verified." They were verified &lt;em&gt;for existence&lt;/em&gt;. I had conflated "the job exited 0" with "the data is recoverable." Those are the same thing only if your backup method is correct, and mine wasn't.&lt;/p&gt;

&lt;p&gt;And here's the part that actually stings: I write about agent reliability. I have written about lock files, retry storms, and log rotation. I had a mental category called "things you must actually test" and backups somehow lived outside it, in the category called "things that are supposed to just work."&lt;/p&gt;

&lt;h2&gt;
  
  
  The drill
&lt;/h2&gt;

&lt;p&gt;Nothing exotic. I picked the oldest snapshot in the repo, restored it into &lt;code&gt;/mnt/restore-test/&lt;/code&gt;, then tried to open it the way my agent opens it in production:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;restic restore latest --target /mnt/restore-test --include /srv/agent/data
sqlite3 /mnt/restore-test/srv/agent/data/app.db &lt;span class="s2"&gt;"PRAGMA integrity_check;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The restore itself was quick — about four minutes for 2.1 GB. Then SQLite answered in under a second:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;database disk image is malformed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Not "file not found." Not "permission denied." The bytes were there. They just weren't a database.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure 1: I was copying a live database with &lt;code&gt;cp&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Here's the backup line I'd been running for 428 nights:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;cp /srv/agent/data/app.db &lt;span class="s2"&gt;"$DEST/app.db"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you've spent time with SQLite, you already know. In WAL mode, the database file alone is not the database. The truth is spread across &lt;code&gt;app.db&lt;/code&gt;, &lt;code&gt;app.db-wal&lt;/code&gt;, and &lt;code&gt;app.db-shm&lt;/code&gt;. &lt;code&gt;cp&lt;/code&gt; grabs a page-level snapshot with no coordination with writers, so you can capture a file mid-transaction — pages that reference a WAL frame that you didn't copy, or a header whose checksum no longer matches its contents.&lt;/p&gt;

&lt;p&gt;The result is a file with the right size, the right extension, and the wrong soul. It only fails when something actually tries to read all of it. Which is to say: it only fails when you need it.&lt;/p&gt;

&lt;p&gt;The correct primitive has existed the whole time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sqlite3 /srv/agent/data/app.db &lt;span class="s2"&gt;".backup '$DEST/app.db'"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;.backup&lt;/code&gt; uses SQLite's online backup API. It holds the right locks, it's consistent while writers continue, and it produces a single self-contained file. Or, if you want the dump form:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sqlite3 /srv/agent/data/app.db &lt;span class="s2"&gt;"VACUUM INTO '$DEST/app.db'"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Both are one line. I chose the wrong one for over a year because it was the first thing I typed and it never complained.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure 2: my "verification" tested the wrong property
&lt;/h2&gt;

&lt;p&gt;This is the part I'd flag if I were reviewing someone else's code, and I still shipped it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; -s &lt;span class="s2"&gt;"$DEST/app.db"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"backup ok"&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;-s&lt;/code&gt; means "exists and is non-empty." A truncated, torn, half-copied database passes that test with room to spare. My 14 months of green checkmarks were 14 months of the operating system telling me a file had bytes in it.&lt;/p&gt;

&lt;p&gt;The check I should have run, on the copy, before calling it a backup:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nv"&gt;result&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;sqlite3 &lt;span class="s2"&gt;"$DEST/app.db"&lt;/span&gt; &lt;span class="s2"&gt;"PRAGMA quick_check;"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$result"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"CORRUPT: $DEST/app.db"&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;quick_check&lt;/code&gt; skips the expensive index cross-verification and catches structural corruption in milliseconds on a file this size. &lt;code&gt;integrity_check&lt;/code&gt; is stricter and worth running on the restore side, not every night.&lt;/p&gt;

&lt;p&gt;A backup you haven't read is a hypothesis. &lt;code&gt;quick_check&lt;/code&gt; is the cheapest way to turn it into a fact.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure 3: the near-miss that would have been worse than the original bug
&lt;/h2&gt;

&lt;p&gt;While setting up the drill I almost did this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;restic restore latest --target /srv/agent/data   &lt;span class="c"&gt;# NO. Do not do this.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;restic restore&lt;/code&gt; overwrites paths under the target. If my latest snapshot had been one of the torn ones, I'd have replaced a working production database with a corrupt one — during a "test." I caught it because I'd typed the command with &lt;code&gt;--target /&lt;/code&gt; first and my stomach dropped before I hit enter.&lt;/p&gt;

&lt;p&gt;The rule I now follow: &lt;strong&gt;restores never touch a live path.&lt;/strong&gt; Restore to a scratch directory, verify there, and only then, deliberately, as a separate documented step, swap anything.&lt;/p&gt;

&lt;p&gt;There was a second near-miss in the same session. My first instinct for cleanup was &lt;code&gt;rsync --delete&lt;/code&gt; to "make the scratch dir match the snapshot." On a path where I'd fat-fingered one directory level, that's a data-loss command wearing a maintenance costume. I deleted the idea and used &lt;code&gt;rm -rf&lt;/code&gt; on a path I'd typed out in full and read twice.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I do now
&lt;/h2&gt;

&lt;p&gt;The whole system is one script and one rule.&lt;/p&gt;

&lt;p&gt;The rule: &lt;strong&gt;the backup job is not done when the file is written. It's done when the file has been read back and validated.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The script, roughly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;set&lt;/span&gt; -euo pipefail
&lt;span class="nv"&gt;DEST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/mnt/backup/staging
&lt;span class="nv"&gt;LATEST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/srv/agent/data/app.db

mkdir -p &lt;span class="s2"&gt;"$DEST"&lt;/span&gt;
sqlite3 &lt;span class="s2"&gt;"$LATEST"&lt;/span&gt; &lt;span class="s2"&gt;".backup '$DEST/app.db'"&lt;/span&gt;

&lt;span class="nv"&gt;check&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$(&lt;/span&gt;sqlite3 &lt;span class="s2"&gt;"$DEST/app.db"&lt;/span&gt; &lt;span class="s2"&gt;"PRAGMA quick_check;"&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="s2"&gt;"$check"&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;"ok"&lt;/span&gt; &lt;span class="o"&gt;]&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"FATAL: backup failed quick_check"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;2&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1&lt;span class="o"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;

sha256sum &lt;span class="s2"&gt;"$DEST/app.db"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$DEST/app.db.sha256"&lt;/span&gt;
restic backup &lt;span class="s2"&gt;"$DEST"&lt;/span&gt; --tag agent-db
restic forget --keep-daily 14 --keep-weekly 8 --prune
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three things changed, and none of them are clever:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Online backup API instead of &lt;code&gt;cp&lt;/code&gt;.&lt;/strong&gt; Consistency, not luck.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;quick_check&lt;/code&gt; before the file counts as a backup.&lt;/strong&gt; The job now fails loudly instead of shipping a plausible-looking corpse.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A checksum manifest&lt;/strong&gt;, so a later restore can prove it got the same bytes the backup job validated — not just a file with the same name.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And once a month, on the first Sunday, a calendar entry fires: restore the newest snapshot to &lt;code&gt;/mnt/restore-test&lt;/code&gt;, run &lt;code&gt;integrity_check&lt;/code&gt;, run one real query that my agent would run, and delete the scratch directory. Fifteen minutes. I've done it three times. It has caught one more problem: a snapshot that was structurally fine but six days stale, because &lt;code&gt;restic forget --prune&lt;/code&gt; had run out of disk mid-prune and left the repo in a state where new snapshots silently failed the tag filter.&lt;/p&gt;

&lt;h2&gt;
  
  
  The lesson I keep relearning
&lt;/h2&gt;

&lt;p&gt;I had monitoring for the &lt;em&gt;absence&lt;/em&gt; of backups and none for the &lt;em&gt;quality&lt;/em&gt; of them. That's the same class of mistake as an agent that logs "success" without checking the response body, or a retry loop that counts attempts instead of outcomes. Green means "the code ran." It has never meant "the thing worked."&lt;/p&gt;

&lt;p&gt;Fourteen months of backups, and my real recovery point was zero. The restore drill didn't just find a corrupt file — it told me which of my green checkmarks were lies. Turns out most of them were &lt;code&gt;test -s&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;If you take one thing from this: pick your oldest snapshot this week, restore it somewhere harmless, and try to actually use it. You'll either get fifteen minutes of boredom (the good outcome) or you'll find out now instead of at 2 AM.&lt;/p&gt;

&lt;p&gt;The verification habits and the agent ops playbooks I use — backup validation, restore drills, the checks that catch "exit 0 but wrong" — are written up in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/0ce2371c-c75d-423c-b64d-685a00445048" rel="noopener noreferrer"&gt;The Solo Operator's AI Agent Playbook&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.&lt;/p&gt;

</description>
      <category>automation</category>
      <category>programming</category>
      <category>raspberrypi</category>
      <category>productivity</category>
    </item>
    <item>
      <title>ufw Said My Pi Was Locked Down. nmap Found Postgres Open to the Internet.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Sat, 14 Nov 2026 01:02:12 +0000</pubDate>
      <link>https://dev.to/ulnit/ufw-said-my-pi-was-locked-down-nmap-found-postgres-open-to-the-internet-3jbi</link>
      <guid>https://dev.to/ulnit/ufw-said-my-pi-was-locked-down-nmap-found-postgres-open-to-the-internet-3jbi</guid>
      <description>&lt;p&gt;My Raspberry Pi 4 sits behind my router, runs &lt;code&gt;ufw&lt;/code&gt;, and holds the database for my AI agent. For eight months I described it to myself as "firewalled." I had the receipts: &lt;code&gt;ufw status verbose&lt;/code&gt; said &lt;code&gt;active&lt;/code&gt;, default deny incoming, one rule allowing SSH from my LAN only.&lt;/p&gt;

&lt;p&gt;Then I wrote a post telling readers to check what their machines actually expose. It felt dishonest to publish it without checking mine, so I rented a $3 VPS for an hour and scanned myself from the outside.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;nmap -Pn -p- --min-rate 2000 &amp;lt;my-public-ip&amp;gt;

PORT     STATE SERVICE
22/tcp   open  ssh          # expected...ish
5432/tcp open  postgresql   # what
6379/tcp open  redis        # no. no no no
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two database ports, wide open to the internet, on a box where &lt;code&gt;ufw&lt;/code&gt; was enabled and denying everything. Both of those &lt;code&gt;ufw&lt;/code&gt; facts were true. That is the whole problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why &lt;code&gt;ufw&lt;/code&gt; was telling the truth and still lying
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;ufw&lt;/code&gt; is a friendly frontend to &lt;code&gt;iptables&lt;/code&gt;' &lt;code&gt;filter&lt;/code&gt; table, specifically the &lt;code&gt;INPUT&lt;/code&gt; chain. Traffic arriving from the internet to a port that a host process is listening on hits &lt;code&gt;INPUT&lt;/code&gt;. Blocking it there works.&lt;/p&gt;

&lt;p&gt;Docker does not send published-port traffic through &lt;code&gt;INPUT&lt;/code&gt;. When you write &lt;code&gt;-p 5432:5432&lt;/code&gt;, Docker inserts a DNAT rule in the &lt;code&gt;nat&lt;/code&gt; table that rewrites the destination address, and the packet is then routed through the &lt;code&gt;FORWARD&lt;/code&gt; chain to the container's bridge network. &lt;code&gt;ufw&lt;/code&gt; never gets a vote. It has no rules in &lt;code&gt;FORWARD&lt;/code&gt;, so the default policy there — set by Docker to ACCEPT — is what decides.&lt;/p&gt;

&lt;p&gt;You can watch it happen:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;sudo iptables -t nat -L DOCKER -n --line-numbers
# DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:5432 to:172.18.0.4:5432
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That rule was six months older than my confidence in my firewall. And &lt;code&gt;ufw status&lt;/code&gt; will never mention it, because from &lt;code&gt;ufw&lt;/code&gt;'s perspective there is nothing to mention.&lt;/p&gt;

&lt;p&gt;I had added &lt;code&gt;-p 5432:5432&lt;/code&gt; to &lt;code&gt;docker-compose.yml&lt;/code&gt; because I wanted to poke at the schema from my laptop on a Saturday afternoon. I told myself I would remove it after lunch. I did not remove it after lunch.&lt;/p&gt;

&lt;h2&gt;
  
  
  What was actually in the logs
&lt;/h2&gt;

&lt;p&gt;Once I knew where to look, the evidence was not subtle. Nine days of Postgres logs, about 11,000 failed authentication attempts, source addresses spread across a dozen countries. Automated scanners find an open 5432 in hours. It is not personal; it is arithmetic.&lt;/p&gt;

&lt;p&gt;The Redis container was worse, because Redis on 6379 with no password is not a brute-force target. It is a write target. In the keyspace I found four keys named &lt;code&gt;backup1&lt;/code&gt; through &lt;code&gt;backup4&lt;/code&gt;, and a &lt;code&gt;CONFIG SET&lt;/code&gt; attempt pointing the dump directory at &lt;code&gt;/root/.ssh&lt;/code&gt; with &lt;code&gt;dbfilename authorized_keys&lt;/code&gt;. That is the standard playbook: use Redis as a file-write primitive, drop an SSH key, come back later.&lt;/p&gt;

&lt;p&gt;It did not work. Not because of anything I did on purpose with security in mind, but because I had set &lt;code&gt;read_only: true&lt;/code&gt; on that container six months earlier for an unrelated reason — I was chasing a bug where a job rewrote a config file mid-run and I wanted it to fail loudly. The attacker's write bounced off a read-only filesystem, and the container logged an error I never read.&lt;/p&gt;

&lt;p&gt;I got lucky, and the luck came from a decision I made while thinking about something else. That is not a security posture. That is a coin flip that happened to land well.&lt;/p&gt;

&lt;h2&gt;
  
  
  The five-minute audit I now run monthly
&lt;/h2&gt;

&lt;p&gt;Two commands, run from outside and inside. The gap between them is your real attack surface.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# From a machine that is NOT this host (a VPS, a phone on cellular)&lt;/span&gt;
nmap -Pn -p- --min-rate 2000 &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;public-ip&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;

&lt;span class="c"&gt;# On the host&lt;/span&gt;
sudo ss -tulpn &lt;span class="o"&gt;|&lt;/span&gt; grep -v &lt;span class="s1"&gt;'127.0.0.1\|::1'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first list is what strangers see. The second is what is bound to all interfaces. Anything in the first list that you cannot justify in one sentence is a bug, and "I'll remove it later" is not a sentence, it is a wish.&lt;/p&gt;

&lt;p&gt;On the Pi, the fix took four edits:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;postgres&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="p"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"127.0.0.1:5432:5432"&lt;/span&gt;   &lt;span class="c1"&gt;# was: "5432:5432"
&lt;/span&gt;  &lt;span class="na"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="c1"&gt;# no ports: block at all. Nothing outside needs it.
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Binding to &lt;code&gt;127.0.0.1&lt;/code&gt; means Docker's DNAT rule targets the loopback interface, which is not reachable from off-host. If I need ad-hoc access from my laptop, I use a tunnel instead of a published port:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ssh -N -L 5432:127.0.0.1:5432 pi@&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;host&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The tunnel exists only while the command runs, which is exactly the property my "temporary" port lacked. If you genuinely need published ports and host-firewall semantics at the same time, &lt;code&gt;ufw-docker&lt;/code&gt; exists and does the &lt;code&gt;FORWARD&lt;/code&gt;-chain work that stock &lt;code&gt;ufw&lt;/code&gt; skips. I installed it, then removed it again, because binding to loopback solved the actual problem and one less moving part is one less thing to misconfigure.&lt;/p&gt;

&lt;h2&gt;
  
  
  The honest part
&lt;/h2&gt;

&lt;p&gt;I had been writing about security for weeks. I had published a post about auditing what an API key can reach and another about cleaning leaked environment variables out of a repo. Meanwhile, my own database was answering strangers, and the only reason I found out was that I wanted a screenshot for an article.&lt;/p&gt;

&lt;p&gt;What I got wrong was not the &lt;code&gt;docker-compose.yml&lt;/code&gt; line. It was the belief that enabling a tool is the same as being protected by it. &lt;code&gt;ufw status: active&lt;/code&gt; reads like a verdict. It is not a verdict, it is a description of one layer, and I had stacked three layers on top of it that it could not see — Docker's &lt;code&gt;FORWARD&lt;/code&gt; policy, my router's port forwarding, my provider's firewall. Each of those has its own opinion, and none of them is reading &lt;code&gt;ufw&lt;/code&gt;'s notes.&lt;/p&gt;

&lt;p&gt;The second thing I got wrong: I had no expiry mechanism for temporary changes. So I added one that does not depend on my memory. There is now a &lt;code&gt;# REVIEW-BY: 2026-12-01&lt;/code&gt; comment above every published port, and a cron job on the Pi that greps for &lt;code&gt;REVIEW-BY&lt;/code&gt; dates in the past and pings me. The job is twelve lines long and it is the most useful thing I have added to that machine all year.&lt;/p&gt;

&lt;h2&gt;
  
  
  The checklist, if you run containers on a Pi
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Never &lt;code&gt;-p 5432:5432&lt;/code&gt; on a host with a public IP. Prefix with &lt;code&gt;127.0.0.1:&lt;/code&gt; or omit the port entirely.&lt;/li&gt;
&lt;li&gt;Treat &lt;code&gt;ufw status&lt;/code&gt; as one input, never as an answer. Check &lt;code&gt;iptables -t nat -L DOCKER -n&lt;/code&gt; too.&lt;/li&gt;
&lt;li&gt;Scan yourself from outside, monthly, from a network you do not control.&lt;/li&gt;
&lt;li&gt;Every published port gets a review date in a comment, and something automated nags you about it.&lt;/li&gt;
&lt;li&gt;Assume Redis, Mongo, and Elasticsearch with no auth are already compromised if they were ever reachable. Rotate the keys, don't just close the port.&lt;/li&gt;
&lt;li&gt;Read the container logs you never read. The only reason I know the attack failed is that I went looking for an error message I had been ignoring for months.&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;If you are about to put something on the internet this week, the drilling order matters more than the tooling: scan from outside first, then read what is bound, then fix the compose file, then re-scan and diff. The full checklist I run through on launch day — plus the scripts that do the outside scan, the port diff, and the &lt;code&gt;REVIEW-BY&lt;/code&gt; nag — is in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/763b023d-bfb5-475d-ab28-9ba0e9ba142d" rel="noopener noreferrer"&gt;Ship Safe — The Launch-Day Security Kit&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.50.&lt;/p&gt;

</description>
      <category>security</category>
      <category>raspberrypi</category>
      <category>automation</category>
      <category>programming</category>
    </item>
    <item>
      <title>My Digest Job Ran as Root for 41 Days. My App Couldn't Read a Single File It Wrote.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Fri, 13 Nov 2026 01:02:03 +0000</pubDate>
      <link>https://dev.to/ulnit/my-digest-job-ran-as-root-for-41-days-my-app-couldnt-read-a-single-file-it-wrote-1p71</link>
      <guid>https://dev.to/ulnit/my-digest-job-ran-as-root-for-41-days-my-app-couldnt-read-a-single-file-it-wrote-1p71</guid>
      <description>&lt;p&gt;The email that ended it was four words long: &lt;em&gt;"is the digest dead?"&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Fair question. The page had been loading fine every single morning. Header, date, footer, all rendered. Between the header and the footer: &lt;strong&gt;"No new items today."&lt;/strong&gt; Forty-one days in a row, on a page that had been averaging 31 items a day since March.&lt;/p&gt;

&lt;p&gt;Nothing had crashed. No 500s, no restarts, no alerts. My uptime monitor was green the entire time — correctly, because the page really was up. It just had nothing to say.&lt;/p&gt;

&lt;h2&gt;
  
  
  The job was fine too
&lt;/h2&gt;

&lt;p&gt;That's the part that took me longest to accept. &lt;code&gt;agent_digest.py&lt;/code&gt; ran every morning at 6:00 and exited 0. I could see it in the log:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-09-14 06:00:02 INFO collected 34 items
2026-09-14 06:00:03 INFO wrote /srv/agent/out/2026-09-14.json
2026-09-14 06:00:03 INFO done
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Thirty-four items. Written. Done. Every day. The files existed. I could &lt;code&gt;cat&lt;/code&gt; them and see the JSON.&lt;/p&gt;

&lt;p&gt;So for the first two weeks I was certain the bug was in the reader. I rewrote the glob. I added caching, then removed the caching. I checked the timezone handling on the date comparison. All of it was fine, because the reader wasn't broken — &lt;strong&gt;it was being told "no" by the operating system, and my code was politely throwing that answer away.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the exception went
&lt;/h2&gt;

&lt;p&gt;This is the reader, minus the comments I'd added to make it look responsible:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;load_digests&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;files&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;OUT&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"*.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;files&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;except Exception: return []&lt;/code&gt;. I wrote that eleven months ago, on a Friday, because a half-written file from a crashed run had 500'd the page in front of a customer. It fixed the symptom in four minutes and then sat there quietly converting &lt;code&gt;PermissionError&lt;/code&gt; into "the world is empty" for the next 41 days.&lt;/p&gt;

&lt;p&gt;The exception was logged nowhere. Not to the app log, not to stderr, not to a counter. A bare &lt;code&gt;except&lt;/code&gt; with a fallback value is not error handling — it's a &lt;strong&gt;silent data-loss funnel&lt;/strong&gt;, and mine had a 41-day backlog.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why my own verification lied to me
&lt;/h2&gt;

&lt;p&gt;Here's the detail I'm least proud of. During that month I had a root shell open in a tmux window on the Pi — I was doing package upgrades and a systemd migration, and it was just easier to leave it there.&lt;/p&gt;

&lt;p&gt;Every check I ran, I ran as root:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;ls -l /srv/agent/out &lt;span class="o"&gt;|&lt;/span&gt; tail -3
-rw------- 1 root root  18K Nov 12 06:00 2026-11-12.json
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I looked at that line at least a dozen times. I read it as &lt;em&gt;"the file is there, so the reader can read it."&lt;/em&gt; The permission bits were in the output the whole time, on the same line, and I never once read them. I'd internalized &lt;code&gt;-rw-------&lt;/code&gt; as normal because every file I created all month looked like that.&lt;/p&gt;

&lt;p&gt;My health check had the same blind spot. It ran from the same crontab, as the same user:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;newest_file&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# passes, because it's root&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;So the writer verified itself, as itself, in the same privileged context, and reported success. 1,271 items went into 41 files that the web app had never been able to open.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two commands that found it in ninety seconds
&lt;/h2&gt;

&lt;p&gt;Once the customer email made me actually look, it took less time than making coffee.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# 1. Check ownership drift in the output dir&lt;/span&gt;
sudo find /srv/agent/out -maxdepth 1 -user root -newermt &lt;span class="s1"&gt;'-60 days'&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; wc -l
41

&lt;span class="c"&gt;# 2. Try to read as the user the app actually runs as&lt;/span&gt;
sudo -u app python3 -c &lt;span class="s2"&gt;"
import pathlib
p = sorted(pathlib.Path('/srv/agent/out').glob('*.json'))[-1]
print(p.read_text()[:60])
"&lt;/span&gt;
PermissionError: &lt;span class="o"&gt;[&lt;/span&gt;Errno 13&lt;span class="o"&gt;]&lt;/span&gt; Permission denied
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first command found 41 files owned by root. The second reproduced the exact exception the bare &lt;code&gt;except&lt;/code&gt; had been eating since September.&lt;/p&gt;

&lt;p&gt;The root cause was two decisions, made on different days, that combined badly:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;During a hardening pass I added &lt;code&gt;os.umask(0o077)&lt;/code&gt; to the agent — correct, files should be private.&lt;/li&gt;
&lt;li&gt;Six weeks earlier, at 1 AM, debugging a missing venv, I'd installed the job with &lt;code&gt;sudo crontab -e&lt;/code&gt; instead of the app user's crontab, and never moved it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Either one alone was harmless. Together they meant every output file was &lt;code&gt;0600 root:root&lt;/code&gt; while the reader ran as &lt;code&gt;app&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix, in three layers
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Layer 1 — the job no longer runs as root.&lt;/strong&gt; It doesn't need to. It needs write access to one directory.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/system/agent-digest.service
&lt;/span&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;User&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;
&lt;span class="na"&gt;Group&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;
&lt;span class="na"&gt;UMask&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0027&lt;/span&gt;
&lt;span class="na"&gt;ExecStart&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="n"&gt;srv&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;venv&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;python&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;srv&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;agent_digest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;py&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output is now &lt;code&gt;0640 agent:agent&lt;/code&gt;, the directory is &lt;code&gt;2750&lt;/code&gt; with group &lt;code&gt;agent&lt;/code&gt;, and &lt;code&gt;app&lt;/code&gt; is a member of that group. Nothing needs sudo. The unit also gives me &lt;code&gt;Restart=on-failure&lt;/code&gt; and a real exit code I can alert on, which crontab never did.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Layer 2 — the smoke test runs as the reader, in the deploy script.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sudo -u app python3 -c &lt;span class="s2"&gt;"
import pathlib
p = sorted(pathlib.Path('/srv/agent/out').glob('*.json'))[-1]
print(p.read_text()[:40] and 'READABLE')
"&lt;/span&gt; &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nb"&gt;exit&lt;/span&gt; 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is the check that would have caught it on day one, and it's one line in the deploy. Nothing else in my setup tests the boundary that actually failed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Layer 3 — the exception is logged, and volume is monitored.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;load_digests&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;files&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;sorted&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;OUT&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;glob&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"*.json"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;days&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;files&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;read_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;OSError&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;JSONDecodeError&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;warning&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"digest read failed: %s: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now a bad file costs me one item, not the whole page, and the failure leaves a fingerprint. I also alert on &lt;strong&gt;content volume&lt;/strong&gt;, not just liveness: if &lt;code&gt;len(items) == 0&lt;/code&gt; two days running, or the newest file is older than 26 hours, I get a push. Uptime monitoring can't see an empty page — it can only see whether something answered.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I actually learned
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Verify as the consumer, never as the producer.&lt;/strong&gt; The writer, the verifier, and the reader all ran as root, so all three agreed and all three were wrong. Any check that runs in the same privilege context as the thing it's checking is a check of your intentions, not of your system.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A fallback value hides a whole category of bug.&lt;/strong&gt; &lt;code&gt;return []&lt;/code&gt; on exception feels defensive. In practice it converts every environment problem — permissions, missing mounts, full disks, corrupt files — into the same indistinguishable "there's no data today," and that state looks exactly like a genuinely quiet day.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Least privilege doesn't fail loudly, it fails forty days later.&lt;/strong&gt; The permission change was correct and the cron entry was convenient, and nothing complained for six weeks. The failure surfaced as a customer noticing a product decision I never made.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;ls -l&lt;/code&gt; puts the answer on the same line as the reassurance.&lt;/strong&gt; Ownership and mode were visible in every check I ran. I read the filename and the size, because that's what I was looking for. Once a month, print the bits you're not looking for.&lt;/p&gt;

&lt;h2&gt;
  
  
  If your agent writes files another process reads
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;sudo find &amp;lt;out_dir&amp;gt; -user root -newermt '-30 days'&lt;/code&gt; — ownership drift is a one-liner, not a hunt&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;sudo -u &amp;lt;reader_user&amp;gt; test -r &amp;lt;file&amp;gt;&lt;/code&gt; — test the boundary, not the file&lt;/li&gt;
&lt;li&gt;grep your codebase for &lt;code&gt;except:&lt;/code&gt; and &lt;code&gt;except Exception:&lt;/code&gt; with a fallback return; every one is a future silent outage&lt;/li&gt;
&lt;li&gt;alert on volume and freshness, not just HTTP status&lt;/li&gt;
&lt;li&gt;make the writer an unprivileged, dedicated user, and set &lt;code&gt;UMask=&lt;/code&gt; in the unit instead of trusting the shell&lt;/li&gt;
&lt;li&gt;if a job runs from a crontab, ask &lt;em&gt;whose&lt;/em&gt; crontab — check &lt;code&gt;sudo crontab -l&lt;/code&gt; as well as &lt;code&gt;crontab -l&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;Forty-one days, 1,271 items, zero errors in any log I was watching, and a customer had to tell me.&lt;/p&gt;

&lt;p&gt;The full checklist + scripts are in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/763b023d-bfb5-475d-ab28-9ba0e9ba142d" rel="noopener noreferrer"&gt;Ship Safe — The Launch-Day Security Kit&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.50.&lt;/p&gt;

</description>
      <category>security</category>
      <category>automation</category>
      <category>ai</category>
      <category>raspberrypi</category>
    </item>
    <item>
      <title>My Agent Logged 14,800 Successful Writes. SQLite Had Rolled Back 9,240 of Them.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Thu, 12 Nov 2026 01:03:13 +0000</pubDate>
      <link>https://dev.to/ulnit/my-agent-logged-14800-successful-writes-sqlite-had-rolled-back-9240-of-them-429l</link>
      <guid>https://dev.to/ulnit/my-agent-logged-14800-successful-writes-sqlite-had-rolled-back-9240-of-them-429l</guid>
      <description>&lt;p&gt;A customer emailed me on a Tuesday: "Is the weekly digest broken? I haven't gotten one since the 3rd."&lt;/p&gt;

&lt;p&gt;I checked. The job had run every night. systemd said &lt;code&gt;active (exited)&lt;/code&gt;, exit code 0, eleven nights in a row. My dashboard said the agent had written 14,800 rows that month.&lt;/p&gt;

&lt;p&gt;SQLite had actually kept 5,560 of them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;A Raspberry Pi 4 with a USB SSD. One Python agent that collects results and writes them to &lt;code&gt;app.db&lt;/code&gt;. One small Flask admin page I use to check things. One nightly backup at 03:10. All three touch the same file.&lt;/p&gt;

&lt;p&gt;I thought that was fine because "SQLite handles concurrency." It does. It handles it by refusing your write — and I had spent a year teaching my agent to ignore refusals.&lt;/p&gt;

&lt;h2&gt;
  
  
  Mistake 1: the PRAGMA that never ran
&lt;/h2&gt;

&lt;p&gt;My init code looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;init_db&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exists&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sqlite3&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"PRAGMA journal_mode=WAL"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I created &lt;code&gt;app.db&lt;/code&gt; by hand in month one, poking at the schema in a REPL. From that day on, &lt;code&gt;init_db()&lt;/code&gt; returned on line 2 and the database stayed in SQLite's default rollback journal mode for a year. I had "enabled WAL" in the sense that the string &lt;code&gt;WAL&lt;/code&gt; existed in my repository. That is not the same thing.&lt;/p&gt;

&lt;p&gt;It's a one-line answer if you check: &lt;code&gt;sqlite3 app.db "PRAGMA journal_mode;"&lt;/code&gt;. The setting is persistent per database file, so you only have to look once.&lt;/p&gt;

&lt;h2&gt;
  
  
  Mistake 2: a reader that held the door shut for 90 seconds
&lt;/h2&gt;

&lt;p&gt;My admin page had a route that streamed a report:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;cur&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"SELECT * FROM results WHERE run_id = ?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;                      &lt;span class="c1"&gt;# read transaction stays open&lt;/span&gt;
    &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"summary"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;summarize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# HTTP call, ~200ms each&lt;/span&gt;
    &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="nf"&gt;render&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;In rollback journal mode, a reader holds a SHARED lock for the life of its transaction. A writer can append, but it cannot commit until every reader is gone. That page took about 90 seconds to render 450 rows, and I left it open in a browser tab most mornings.&lt;/p&gt;

&lt;p&gt;So my agent's commit window was, several times a night, exactly as long as I felt like staring at a dashboard.&lt;/p&gt;

&lt;h2&gt;
  
  
  Mistake 3: counting rows I never wrote
&lt;/h2&gt;

&lt;p&gt;Here is the write path, and it is the part that still bothers me:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;cur&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"INSERT INTO results VALUES (?,?,?)"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;commit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;written&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;              &lt;span class="c1"&gt;# the lie&lt;/span&gt;
&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nc"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;warning&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"db issue: %s"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;    &lt;span class="c1"&gt;# swallowed&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things wrong.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;len(rows)&lt;/code&gt; measures intent, not outcome. And when &lt;code&gt;commit()&lt;/code&gt; raised &lt;code&gt;database is locked&lt;/code&gt; after the default 5-second timeout, the bare &lt;code&gt;except&lt;/code&gt; caught it, logged a WARNING into a log file that rotated weekly, and the function returned normally. The job exited 0. systemd was satisfied. My counter had already been incremented, so the number I looked at every morning was the number of rows I &lt;em&gt;tried&lt;/em&gt; to write.&lt;/p&gt;

&lt;p&gt;Across 11 nights: 14,800 attempts, 5,560 rows on disk, 41 digests never sent. I found out from a customer, not from monitoring.&lt;/p&gt;

&lt;p&gt;The shape of the failure is what I want you to notice. Every signal I had was green, because I had built all of them around "did the process finish" instead of "did the data land."&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;One connection factory, PRAGMAs applied every time.&lt;/strong&gt; &lt;code&gt;busy_timeout&lt;/code&gt; is per-connection, not per-database. If your agent opens a fresh connection per task, a PRAGMA you set once in a REPL did nothing for it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sqlite3&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="no"&gt;DB&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;isolation_level&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"PRAGMA journal_mode=WAL"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"PRAGMA synchronous=NORMAL"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"PRAGMA busy_timeout=15000"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;WAL is what kills the 90-second-tab problem outright: readers no longer block writers. Keep the file on ext4. I've watched WAL refuse to enable on an SMB mount and on a cheap exFAT USB stick, and the failure surfaced much later as a confusing "database is locked."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Writes start with BEGIN IMMEDIATE.&lt;/strong&gt; In deferred mode SQLite takes a read lock first and upgrades at commit; if two writers overlap, one gets &lt;code&gt;SQLITE_BUSY&lt;/code&gt; at the worst possible moment. Immediate takes the write lock up front, so you wait at the start of the transaction, where retrying is sane.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Retry only the lock, with backoff, and bounded:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;conn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"BEGIN IMMEDIATE"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;executemany&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="s2"&gt;"INSERT INTO results (run_id, customer_id, payload) "&lt;/span&gt;
            &lt;span class="s2"&gt;"VALUES (?,?,?) ON CONFLICT(run_id, customer_id) DO NOTHING"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"COMMIT"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;break&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="n"&gt;sqlite3&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;OperationalError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;"locked"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="s2"&gt;"busy"&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt;
        &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;0.5&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"db still locked after 5 attempts, run {run_id}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;ON CONFLICT DO NOTHING&lt;/code&gt; with a unique index on &lt;code&gt;(run_id, customer_id)&lt;/code&gt; is what makes retries safe. Before I had that, a retry meant duplicate digests, which is why my earlier instinct was simply not to retry. That instinct cost me 9,240 rows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Assert the write.&lt;/strong&gt; After the batch, count what is actually in the table and compare:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="s2"&gt;"SELECT COUNT(*) FROM results WHERE run_id = ?"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;!=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"wrote {n} of {len(rows)} rows for run {run_id}"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the count is short, the job fails loudly. I would much rather get a 3 AM page than a Tuesday email from a customer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Monitor freshness, not exit codes.&lt;/strong&gt; One query, once an hour:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="nb"&gt;ROUND&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;julianday&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'now'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="nf"&gt;julianday&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;MAX&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;created_at&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;hours_stale&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Anything over 26 hours is an alert. This is the check that would have caught the whole thing on night one, and it doesn't care &lt;em&gt;why&lt;/em&gt; the writes stopped — locks, expired credentials, a full disk, an agent that quietly decided not to run. It only asks whether the data is current.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Serialise the writer.&lt;/strong&gt; The admin page no longer writes anything. The backup now uses &lt;code&gt;sqlite3 app.db ".backup backup.db"&lt;/code&gt; instead of &lt;code&gt;cp&lt;/code&gt;, because copying a live SQLite file gets you a torn copy that opens fine and is silently missing pages. &lt;code&gt;cp&lt;/code&gt; on an active database is a coin flip you only lose once.&lt;/p&gt;

&lt;h2&gt;
  
  
  The uncomfortable lesson
&lt;/h2&gt;

&lt;p&gt;I didn't have a concurrency bug. I had a reporting bug that hid a concurrency bug, sitting underneath a monitoring setup that could only detect a process that died — and processes that fail at SQLite almost never die. They log a warning and return.&lt;/p&gt;

&lt;p&gt;Those 11 days weren't caused by SQLite's locking model. SQLite was correct the entire time. It told me, 9,240 times, that it could not commit. I had written code specifically to make sure I never heard it.&lt;/p&gt;

&lt;p&gt;If you're running an agent against SQLite on a Pi right now, two commands are worth running before you close this tab:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;sqlite3 your.db &lt;span class="s2"&gt;"PRAGMA journal_mode;"&lt;/span&gt;
grep -c &lt;span class="s2"&gt;"database is locked"&lt;/span&gt; your-agent.log
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the first says &lt;code&gt;delete&lt;/code&gt; and the second says anything above zero, you already know what your next hour looks like.&lt;/p&gt;

&lt;p&gt;The connection factory, the startup assertions, the write-boundary rules, and the freshness check above are four of the patterns I use in every agent I ship. I write up the specific playbooks in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/0ce2371c-c75d-423c-b64d-685a00445048" rel="noopener noreferrer"&gt;The Solo Operator's AI Agent Playbook&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>programming</category>
      <category>raspberrypi</category>
    </item>
    <item>
      <title>I Had 23 Sentences Starting With 'Never' in My Agent's Prompt. I Deleted All of Them.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Wed, 11 Nov 2026 01:02:30 +0000</pubDate>
      <link>https://dev.to/ulnit/i-had-23-sentences-starting-with-never-in-my-agents-prompt-i-deleted-all-of-them-3pm2</link>
      <guid>https://dev.to/ulnit/i-had-23-sentences-starting-with-never-in-my-agents-prompt-i-deleted-all-of-them-3pm2</guid>
      <description>&lt;p&gt;The message was four words long.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;can I return this?&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;My support agent replied:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I am not able to discuss returns.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Not the policy. Not "yes, thirty days, here's the link." A flat refusal to acknowledge that returns exist, sent to a paying customer at 9:41 on a Tuesday.&lt;/p&gt;

&lt;p&gt;I opened the system prompt expecting to find a bug. What I found was worse: twenty-three sentences, every one of them starting with &lt;strong&gt;Never&lt;/strong&gt; or &lt;strong&gt;Do not&lt;/strong&gt;, sitting in a section I had grandly titled &lt;code&gt;HARD RULES&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;HARD RULES (v7)
NEVER promise a delivery date.
NEVER invent an order status.
DO NOT offer a discount unless the customer asks twice.
NEVER discuss competitor pricing.
NEVER speculate about stock levels.
DO NOT apologize more than once.
NEVER say "as an AI".
DO NOT quote a refund timeline you cannot verify.
... 15 more ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every one of those lines was added after a real incident. A customer got a made-up delivery date, so I added a rule. The agent offered 40% off to someone who hadn't asked, so I added a rule. The agent quoted a refund timeline from a policy I retired in March, so I added a rule.&lt;/p&gt;

&lt;p&gt;Each rule was individually correct. Together, they were the reason a customer couldn't get a straight answer about returns.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why "never" is the weakest instruction you can write
&lt;/h2&gt;

&lt;p&gt;A negative instruction tells the model where not to go without telling it where to go. That's not a constraint, it's a hole. The model still has to produce a next token, so it infers the alternative on its own, and under ambiguity it reliably picks the most conservative one: say less, commit to nothing, refuse.&lt;/p&gt;

&lt;p&gt;Three things were happening at once in my prompt.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One: no positive target.&lt;/strong&gt; &lt;code&gt;NEVER invent a delivery date&lt;/code&gt; does not say what to say instead. The agent had to invent a fallback, and its invention was "I'm unable to provide that information."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Two: no defined fallback.&lt;/strong&gt; Every prohibition needs a literal string to emit when the prohibition bites. Otherwise you get a different refusal every time, none of which a customer can act on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Three: no priority.&lt;/strong&gt; Twenty-three rules in a flat list, in the order I happened to write them. When two of them pulled in opposite directions (be helpful / never speculate), the model resolved the conflict silently, and I never saw the decision.&lt;/p&gt;

&lt;p&gt;There was also a plain arithmetic problem. My system prompt was 1,380 words. Rules added in month three sat at the bottom of a wall of text, with no ranking and no repetition. I had built a prompt that was mostly list.&lt;/p&gt;

&lt;h2&gt;
  
  
  What replaced them: nine rules, each with a source and a fallback
&lt;/h2&gt;

&lt;p&gt;I spent a Saturday deleting. Twenty-three prohibitions became nine positive rules. Each one starts with a verb, names the data it reads from, and defines the exact string to send when that data is missing.&lt;/p&gt;

&lt;p&gt;Before:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NEVER promise a delivery date.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DELIVERY DATES
Source: order.delivery_window (ISO 8601 range)
If present: quote it verbatim. No hedging, no "approximately", no rounding.
If null: send exactly:
  "I can't confirm a date yet — a human will follow up within one business day."
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;DO NOT quote a refund timeline you cannot verify.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;REFUND TIMELINES
Source: policy.refund_days (integer, current policy version only)
If version &amp;lt; today's policy version: ignore the record and use the fallback.
Fallback: "Refunds are reviewed within 5 business days. I've flagged this for a human."
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;NEVER discuss competitor pricing.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;After:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;COMPETITOR QUESTIONS
Reply: "I can only speak to our pricing. Ours is on the pricing page — want the link?"
Then hand off to a human if the customer asks again.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That third one matters more than it looks. The original rule produced silence, which reads as evasion. The replacement acknowledges the question and redirects once, then escalates. Same boundary, radically different customer experience.&lt;/p&gt;

&lt;h2&gt;
  
  
  The measurement that made it real
&lt;/h2&gt;

&lt;p&gt;I don't trust prompt changes I can't count, so I built a 400-message test set from real support threads (redacted), each labelled with what a correct reply must contain and must not contain. Refund requests, delivery questions, order status, angry customers, one deliberate prompt injection, twelve messages that should escalate to a human.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;v7 (23 prohibitions)&lt;/th&gt;
&lt;th&gt;v8 (9 rules)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;System prompt tokens&lt;/td&gt;
&lt;td&gt;1,840&lt;/td&gt;
&lt;td&gt;690&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Legitimate requests refused&lt;/td&gt;
&lt;td&gt;14.2%&lt;/td&gt;
&lt;td&gt;1.2%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Replies containing "I cannot" / "unable to"&lt;/td&gt;
&lt;td&gt;9.8%&lt;/td&gt;
&lt;td&gt;0.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Invented delivery dates&lt;/td&gt;
&lt;td&gt;3 / 400&lt;/td&gt;
&lt;td&gt;0 / 400&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Median reply length&lt;/td&gt;
&lt;td&gt;41 words&lt;/td&gt;
&lt;td&gt;58 words&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The 1.2% that still refused were genuinely out of scope: two legal threats and three requests for another customer's data. That's the refusal rate I wanted. Not zero, but earned.&lt;/p&gt;

&lt;h2&gt;
  
  
  The mistake I made inside the fix
&lt;/h2&gt;

&lt;p&gt;Deleting the prohibitions worked, and then I broke something new, which is the part worth writing down.&lt;/p&gt;

&lt;p&gt;My first rewrite of the delivery rule pointed at &lt;code&gt;order.delivery_window&lt;/code&gt;. It was correct on the order-status path, because that path loads the full order record. On the refund path, the assembler only loaded &lt;code&gt;order.id&lt;/code&gt;, &lt;code&gt;order.total&lt;/code&gt;, and &lt;code&gt;policy.*&lt;/code&gt;. So the field was null, the fallback fired, and for nine days every refund email that mentioned shipping said:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I can't confirm a date yet.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;For a refund. On a delivered order, whose delivery date was in the customer's original receipt.&lt;/p&gt;

&lt;p&gt;I had removed a hallucination and installed a shrug. And because the fallback was polite and confident, nothing in my logs looked wrong — no error, no exception, just a slightly sad sentence in 60-odd emails that nobody complained about because it wasn't wrong enough to complain about.&lt;/p&gt;

&lt;p&gt;Two lessons, both now enforced in code.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A positive rule is only as good as the data it points at.&lt;/strong&gt; The model cannot fetch what the prompt doesn't contain, and it will not tell you the difference. Each rule in my prompt now declares its source in a machine-readable header, and the assembler refuses to build a prompt where a declared source is missing:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="no"&gt;REQUIRED&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"DELIVERY DATES"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"order.delivery_window"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="s2"&gt;"REFUND TIMELINES"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"policy.refund_days"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;assert_sources&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;block_names&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="no"&gt;REQUIRED&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;block_names&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="nf"&gt;resolve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;PromptAssemblyError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;"{block} declared but {path} is not in context"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;A fallback that only says "I can't" is still a refusal with better manners.&lt;/strong&gt; Every fallback now names a next action and a time bound: &lt;em&gt;a human will follow up within one business day&lt;/em&gt;. "Unable to confirm" is a dead end. "Flagged for a human, today" is a handoff.&lt;/p&gt;

&lt;p&gt;I found the nine days of shrugs by accident, reading a customer's reply that said "thanks, but I already knew the date." That's the only bug report I got. Nine days, zero alerts, because the agent was never wrong in a way that was loud.&lt;/p&gt;

&lt;h2&gt;
  
  
  The 20-minute audit, if your prompt looks like mine did
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;Find the prohibitions. &lt;code&gt;grep -niE "never|do not|don't|avoid|must not" prompt.md&lt;/code&gt;. Count them. Mine was 23.&lt;/li&gt;
&lt;li&gt;For each one, write the behavior you actually want, starting with a verb. If you can't, the rule isn't a prompt rule — it's a missing feature.&lt;/li&gt;
&lt;li&gt;Name the data source. If there isn't one, move the rule out of the prompt and into code: a permission, a validation, a guard that raises. Prompts are for decisions; code is for limits.&lt;/li&gt;
&lt;li&gt;Define the fallback as a literal string with an owner and a deadline. No improvised refusals.&lt;/li&gt;
&lt;li&gt;Collect 30 real messages, write what a correct reply must contain, run the prompt before and after. If you can't state the number, you didn't make an improvement, you made a change.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And the thing I wish someone had told me eighteen months ago: the prompt is not a place to store fear. I had been writing my prompt the way you write a postmortem, one scar at a time, and it turned into a document that only knew what not to do. A prompt should read like a set of decisions, with the scary parts moved into code where they can actually fail loudly.&lt;/p&gt;

&lt;p&gt;The agent didn't get smarter. It stopped being asked to guess.&lt;/p&gt;




&lt;p&gt;All 100 prompts are in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/af4c3237-d411-4fbd-87b9-d5a562e55e4c" rel="noopener noreferrer"&gt;The Agent Prompt Vault&lt;/a&gt; — $3, lifetime updates. Steal the ones that fit your workflow.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>productivity</category>
      <category>indiehackers</category>
    </item>
    <item>
      <title>A Truncated JSONL File Parses Fine. That Is Why My Reports Were Empty for 11 Days.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Tue, 10 Nov 2026 01:02:20 +0000</pubDate>
      <link>https://dev.to/ulnit/a-truncated-jsonl-file-parses-fine-that-is-why-my-reports-were-empty-for-11-days-4f56</link>
      <guid>https://dev.to/ulnit/a-truncated-jsonl-file-parses-fine-that-is-why-my-reports-were-empty-for-11-days-4f56</guid>
      <description>&lt;h1&gt;
  
  
  A Truncated JSONL File Parses Fine. That's Why It Took Me 11 Days to Notice My Reports Were Empty.
&lt;/h1&gt;

&lt;p&gt;My morning dashboard said the same thing for eleven days straight:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;2026-10-14  anomalies: 0   status: PASS
2026-10-15  anomalies: 0   status: PASS
...
2026-10-25  anomalies: 0   status: PASS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Zero anomalies. Eleven clean days. I actually felt good about it. I'd spent a weekend hardening that pipeline, and here was the payoff — a Raspberry Pi in my closet quietly watching a stream of sensor events and telling me, every morning at 06:50, that nothing was wrong.&lt;/p&gt;

&lt;p&gt;On day twelve I opened one of the raw files by accident, looking for something unrelated. The file was 900 bytes. A normal day was 210 KB.&lt;/p&gt;

&lt;p&gt;The reports weren't clean. They were &lt;em&gt;hollow&lt;/em&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the pipeline actually did
&lt;/h2&gt;

&lt;p&gt;It's not complicated, which is the point. A writer process appends one JSON object per line to a dated file:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"a"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dumps&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;"\n"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Bash on the Pi, tiny Python, a couple thousand events a day. A second process reads that file fifteen minutes later, aggregates, and writes the summary:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;json&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;loads&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is the part I want you to sit with. &lt;strong&gt;When a writer and a reader touch the same file without any coordination, the reader can see a partial file.&lt;/strong&gt; Not a corrupt file — a &lt;em&gt;shorter&lt;/em&gt; one. My writer was still flushing when the reader opened it. The reader got whatever bytes had landed so far.&lt;/p&gt;

&lt;p&gt;With JSON, that would have exploded. A truncated &lt;code&gt;{...&lt;/code&gt; is a &lt;code&gt;JSONDecodeError&lt;/code&gt; and I'd have known in seconds.&lt;/p&gt;

&lt;p&gt;With &lt;strong&gt;JSONL it just... parses.&lt;/strong&gt; Each line is independent. There's no footer, no closing brace, no record count, no checksum, nothing that says "this file is complete." A file with 40 lines out of 2,100 is a perfectly valid JSONL file. It's just a valid file that describes a much smaller world.&lt;/p&gt;

&lt;p&gt;And so the aggregation ran over 40 rows, found no anomalies in those 40 rows, and wrote &lt;code&gt;anomalies: 0, status: PASS&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The failure wasn't an exception. The failure was a &lt;em&gt;successful&lt;/em&gt; run over data that wasn't there yet.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure #1: I fixed it with a sleep
&lt;/h2&gt;

&lt;p&gt;My first fix was the kind of fix you make at 07:10 before a standup. I added a wait.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# let the writer finish&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It worked. For about a week. The reports came back with real row counts, real anomalies, real numbers I could act on.&lt;/p&gt;

&lt;p&gt;Then it broke again, and this time it was &lt;em&gt;worse&lt;/em&gt;, because now I trusted it.&lt;/p&gt;

&lt;p&gt;The Pi's SD card got busy one morning — a backup job and the writer overlapping. The flush that usually finished in two seconds took four minutes. My reader woke up at its scheduled time, saw an incomplete file, and produced another hollow PASS. Same bug, new latency profile. A sleep doesn't create an ordering guarantee, it just moves the race condition to a time I'm not looking at. I had encoded an assumption about disk speed into a &lt;code&gt;sleep()&lt;/code&gt; and then forgotten I'd done it.&lt;/p&gt;

&lt;p&gt;If your correctness depends on "the other thing is probably done by now," you don't have a pipeline. You have a hope with a cron entry.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure #2: the check that agreed with the bug
&lt;/h2&gt;

&lt;p&gt;The second fix was the one I was proudest of, and it's the one that should embarrass me most. I added a row-count sanity check:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;prev&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;load_yesterday_count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;prev&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mf"&gt;0.90&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="nf"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"volume drop"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Compare today's volume to yesterday's, alert if it falls more than 10%. Reasonable! Standard! And completely defeated by the fact that &lt;strong&gt;yesterday's baseline was also truncated.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Day one: 2,100 rows. Day two got truncated to 1,940 — a 7.6% drop, under my 10% threshold, so no alert, and now the baseline was 1,940. Day three truncated to 1,790 — again under 10%. The baseline ratcheted downward, one silent step at a time, each step inside tolerance. Eleven days later my "normal" was 900 bytes, and every single day had passed a check designed to catch exactly this.&lt;/p&gt;

&lt;p&gt;I built a drift detector and pointed it at a drifting reference. Relative checks only work if the denominator is anchored to something that can't quietly move. Mine moved every night and reported that all was well.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I actually did, in order of how much it mattered
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Stop writing in place.&lt;/strong&gt; The writer now writes a sibling temp file and atomically renames it into position. On POSIX, &lt;code&gt;os.replace&lt;/code&gt; is atomic — a reader either sees the old file or the new one, never half of either.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;tmp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="s2"&gt;".tmp"&lt;/span&gt;
&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nb"&gt;open&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"w"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;flush&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fsync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;fileno&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# survive power loss, not just process exit&lt;/span&gt;
&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;replace&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For an append-only event stream where you can't rewrite the whole file, write to &lt;code&gt;events.jsonl.tmp&lt;/code&gt; continuously and &lt;code&gt;os.replace&lt;/code&gt; it into &lt;code&gt;events.jsonl&lt;/code&gt; when the batch closes. Same guarantee, no rewriting cost at read time.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Make completeness explicit.&lt;/strong&gt; Every file now ships with a sentinel:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"rows"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;2087&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"sha256"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"9f2c..."&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nl"&gt;"closed_at"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"2026-10-26T06:41:12Z"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The reader refuses to run without one. No sentinel, no report — and a &lt;em&gt;missing report&lt;/em&gt; is an alert, not a silence.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Check the denominator, absolutely.&lt;/strong&gt; Not "down 10% from yesterday." The writer knows how many events it emitted. The reader asserts it received exactly that many, or it fails loudly. A count that doesn't match is a bug, not a tolerance band.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Alert on absence, not on emptiness.&lt;/strong&gt; This was the real lesson and it took me three incidents to learn it. My monitoring had one signal: &lt;code&gt;anomalies &amp;gt; 0&lt;/code&gt;? No → quiet. But "zero anomalies" and "zero data" are two completely different states of the world that produced the identical output. Every pipeline needs a heartbeat that fires when there's &lt;em&gt;nothing to say&lt;/em&gt;. I now have a dead-man's switch: if the daily summary doesn't arrive by 07:00, I get a message. Silence is only trustworthy when it's accompanied by proof of life.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Log the input, not just the decision.&lt;/strong&gt; Every report now carries &lt;code&gt;rows_read&lt;/code&gt;, &lt;code&gt;rows_expected&lt;/code&gt;, &lt;code&gt;file_bytes&lt;/code&gt;, &lt;code&gt;file_mtime&lt;/code&gt;, &lt;code&gt;sentinel_present&lt;/code&gt;. Eleven days of hollow reports would have been instantly visible in a single column, because 40 next to 2,100 is not a subtle discrepancy. I had been logging my conclusions and throwing away my evidence.&lt;/p&gt;

&lt;h2&gt;
  
  
  The general shape of this bug
&lt;/h2&gt;

&lt;p&gt;This wasn't a JSONL bug, or a Raspberry Pi bug, or a cron bug. It's the pattern where &lt;strong&gt;the absence of data is indistinguishable from the absence of problems&lt;/strong&gt; — and it shows up everywhere once you start looking for it.&lt;/p&gt;

&lt;p&gt;An agent that fails to retrieve context and answers confidently from memory. A scraper that gets zero results and reports "no new items." A monitoring query that returns an empty set because the auth token expired, and the dashboard renders that empty set as green. Every one of those is a truncated file wearing a different costume. The system did not fail to produce an answer. It produced an answer about a smaller world than the one it was supposed to be looking at, and nobody asked how big the world was.&lt;/p&gt;

&lt;p&gt;The question I ask now, of every automated report I read, is not "did it pass?" It's &lt;strong&gt;"how much did it look at, and who told it that was all of it?"&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you can't answer that in one line of output, you don't know whether the pass means anything.&lt;/p&gt;

&lt;h2&gt;
  
  
  The two-minute version I wish I'd had
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Write to temp, &lt;code&gt;fsync&lt;/code&gt;, &lt;code&gt;os.replace&lt;/code&gt;. Never read a file another process may still be writing.&lt;/li&gt;
&lt;li&gt;Ship a record count and a checksum with every artifact. No footer, no trust.&lt;/li&gt;
&lt;li&gt;Assert absolute counts, not relative deltas. Ratcheting baselines hide slow truncation perfectly.&lt;/li&gt;
&lt;li&gt;Alert when a report is &lt;em&gt;missing&lt;/em&gt;, not only when it looks bad.&lt;/li&gt;
&lt;li&gt;Print the denominator next to the conclusion. &lt;code&gt;rows_read / rows_expected&lt;/code&gt; would have caught this on day one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Eleven days of PASS wasn't a monitoring failure. It was a measurement failure. The pipeline was working; it was just measuring a world that had quietly shrunk to 900 bytes.&lt;/p&gt;




&lt;p&gt;The full checklist + scripts are in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/763b023d-bfb5-475d-ab28-9ba0e9ba142d" rel="noopener noreferrer"&gt;Ship Safe — The Launch-Day Security Kit&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.50.&lt;/p&gt;

</description>
      <category>raspberrypi</category>
      <category>automation</category>
      <category>programming</category>
      <category>ai</category>
    </item>
    <item>
      <title>My Safety Switch Was the String "dry". Python Called It True and Emailed 1,412 Customers.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Mon, 09 Nov 2026 01:03:55 +0000</pubDate>
      <link>https://dev.to/ulnit/my-safety-switch-was-the-string-dry-python-called-it-true-and-emailed-1412-customers-397f</link>
      <guid>https://dev.to/ulnit/my-safety-switch-was-the-string-dry-python-called-it-true-and-emailed-1412-customers-397f</guid>
      <description>&lt;h1&gt;
  
  
  My Safety Switch Was the String "dry". Python Called It True and Emailed 1,412 Customers.
&lt;/h1&gt;

&lt;p&gt;At 06:58 on a Tuesday my agent printed this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[06:58:12] boot   | AGENT_MODE=dry | target=plan-metadata | scope=all
[06:58:14] boot   | dry-run guards active
[06:58:19] stripe | updating customer metadata | n=1412
[06:58:51] mail   | queued plan-change notice | n=1412
[06:58:52] done   | 1412 updated, 1412 queued, 0 failed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read that twice. Line one says &lt;code&gt;dry&lt;/code&gt;. Line two says the guards are active. Line three updated fourteen hundred and twelve live Stripe customers. Line four queued the same number of customer emails. Line five calls it a success.&lt;/p&gt;

&lt;p&gt;I read that log at 07:02, filed it as my most boring successful run of the week, and made coffee. The queue worker drained at 07:00. By 11:20 I had eleven replies from people asking why they'd been moved to an annual plan, and one very polite email from a customer who wanted to know why my "dry run" had a body count.&lt;/p&gt;

&lt;p&gt;Nothing was actually changed in billing. The agent was rewriting a metadata label as part of a segmentation cleanup, and the "notice" email told people their plan details had been updated. That's the good version of the story. The bad version is that the same code path sat three lines from a &lt;code&gt;proration_behavior&lt;/code&gt; parameter I'd commented out "for now." I've thought about that a lot since.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug was one missing comparison
&lt;/h2&gt;

&lt;p&gt;Here is the entire guard:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;

&lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getLogger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;MODE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getenv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;AGENT_MODE&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dry&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;sync_customer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getenv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;AGENT_MODE&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dry&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="nf"&gt;do_the_real_thing&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;info&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dry run: would update %s&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;customer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;os.getenv&lt;/code&gt; returns a &lt;strong&gt;string or &lt;code&gt;None&lt;/code&gt;&lt;/strong&gt;. It never returns a boolean, no matter what you put in &lt;code&gt;.env&lt;/code&gt;. And every non-empty string in Python is truthy — including the string &lt;code&gt;"dry"&lt;/code&gt;, including &lt;code&gt;"false"&lt;/code&gt;, including &lt;code&gt;"0"&lt;/code&gt;, including &lt;code&gt;"no"&lt;/code&gt;. So the condition above isn't "are we in dry mode?" It's "does this variable exist?" Which it did. Which means the guard was never a guard. It was a comment with syntax.&lt;/p&gt;

&lt;p&gt;The startup banner didn't lie, exactly: it printed the variable with an f-string and faithfully reported &lt;code&gt;dry&lt;/code&gt;. It just reported a value no branch ever compared against anything.&lt;/p&gt;

&lt;p&gt;For the record, because I looked it up sitting on my kitchen floor:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Value in &lt;code&gt;.env&lt;/code&gt;
&lt;/th&gt;
&lt;th&gt;&lt;code&gt;bool(value)&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Passes &lt;code&gt;if value:&lt;/code&gt;
&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;"dry"&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;True&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;"false"&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;True&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;"0"&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;True&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;✅&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;""&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;False&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;unset&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;None&lt;/code&gt; → &lt;code&gt;False&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;❌&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Exactly one string there turns the switch off: the empty one. I'd written a boolean check so fragile that safety required configuring nothing at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why my tests didn't catch it
&lt;/h2&gt;

&lt;p&gt;This is the part I'm least proud of, and it's the part worth reading.&lt;/p&gt;

&lt;p&gt;I had one test for dry run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;test_dry_run_writes_nothing&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;monkeypatch&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;monkeypatch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setenv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;AGENT_MODE&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;""&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# ← the tell
&lt;/span&gt;    &lt;span class="nf"&gt;sync_customer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fake_customer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tier&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;grandfathered&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;stripe_mock&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;update&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call_count&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Look at what I passed. &lt;code&gt;""&lt;/code&gt;. The empty string. I picked the single value that makes the broken condition evaluate the way I wanted, ran it, watched it go green, and never asked why I needed &lt;code&gt;""&lt;/code&gt; to represent "dry mode" when the actual config file said &lt;code&gt;dry&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That test didn't verify the guard; it verified that I could turn the guard off. It passes as long as the software is broken in exactly the way it's broken.&lt;/p&gt;

&lt;p&gt;Three other things had to line up for this to reach 1,412 people:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The flag was read in two places.&lt;/strong&gt; &lt;code&gt;MODE&lt;/code&gt; was assigned at import and used by nothing, while the functions re-read the environment on every call. Two sources of truth, three call sites, zero agreement.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dry run only covered writes.&lt;/strong&gt; It never touched the mail queue. Even a correct guard would have left a dry-run message sitting in the &lt;code&gt;outbox&lt;/code&gt; table for the worker to pick up at 07:00. The dry run and the live run shared the same outbox.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nothing asserted on startup.&lt;/strong&gt; I printed the resolved config as a courtesy, in a log nobody reads, with no format that announces when the value is wrong. A startup banner that only ever prints is decoration.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  The fix: don't branch on the flag near the side effect
&lt;/h2&gt;

&lt;p&gt;The instinct after an incident like this is to add &lt;code&gt;== "live"&lt;/code&gt; everywhere. That fixes the bug and leaves the pattern intact. Bad flags get copied; the pattern is what propagates. Three changes, in increasing order of how much they'd have saved me.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Parse config once, into a type, and crash on anything unrecognized.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;enum&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Enum&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Enum&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;DRY&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;dry&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;LIVE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;live&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;parse_mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;Mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;raw&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Mode&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DRY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;Mode&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;LIVE&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nc"&gt;Mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;SystemExit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;AGENT_MODE must be &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;dry&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; or &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;live&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;, got &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;raw&lt;/span&gt;&lt;span class="si"&gt;!r}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;{raw!r}&lt;/code&gt; matters more than it looks. &lt;code&gt;repr&lt;/code&gt; puts the quotes back. Half my incident's confusion came from a log line that stripped the one piece of information that would have identified the bug. And an allowlist of two values means there is no third state to guess about: &lt;code&gt;"true"&lt;/code&gt;, &lt;code&gt;"1"&lt;/code&gt;, &lt;code&gt;"yes"&lt;/code&gt;, &lt;code&gt;"off"&lt;/code&gt; all become a startup crash instead of a policy decision made by Python's truthiness rules.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Enforce at the boundary, not at the decision.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The guard belonged in the object that touches the network, not in the function that decides what it wants to happen:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;StripeWrite&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;__init__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;Mode&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;canary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_mode&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mode&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_canary&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;canary&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;audit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;

    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;update_metadata&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;dict&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_mode&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="n"&gt;Mode&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DRY&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;audit&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;id&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;md&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;})&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt;   &lt;span class="c1"&gt;# never reaches the client
&lt;/span&gt;        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_canary&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_canary&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;canary tag missing — refusing to write&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;customers&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;update&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Now every write goes through one door, and the door reads the mode. There's no path to the client that skips the check, because the check sits in front of the client rather than up the call stack where a future refactor can route around it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Make dry run physically incapable, not politely unwilling.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The final version constructs the whole agent with a &lt;strong&gt;read-only API key&lt;/strong&gt; when the mode is dry, and points the mail client at a sink that appends to a file. That's the change I trust. Every other fix depends on my code evaluating a condition correctly. This one depends on the platform rejecting an unauthorized request — a much better thing to depend on.&lt;/p&gt;

&lt;p&gt;If your dry run can't fail because of a bug in your code, it isn't a dry run. It's a rehearsal that can go live.&lt;/p&gt;

&lt;p&gt;Plus the boring procedural bits that actually held on the second attempt: a canary batch of one record tagged &lt;code&gt;canary:&amp;lt;date&amp;gt;&lt;/code&gt;, a rule that no run touches more than 25 records without a human reading the diff, an outbox worker that pauses when &lt;code&gt;ACTIVE=false&lt;/code&gt;, and a startup check that refuses to boot when &lt;code&gt;mode is LIVE&lt;/code&gt; and &lt;code&gt;canary is None&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd tell anyone running an agent against production
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Grep every &lt;code&gt;os.getenv&lt;/code&gt;, &lt;code&gt;os.environ.get&lt;/code&gt;, and &lt;code&gt;getenv(...) or default&lt;/code&gt; in your agent. If one is used in an &lt;code&gt;if&lt;/code&gt;, it's a string pretending to be a boolean and it's probably wrong.&lt;/li&gt;
&lt;li&gt;Test the config parser with the full ugly set: &lt;code&gt;"true"&lt;/code&gt;, &lt;code&gt;"1"&lt;/code&gt;, &lt;code&gt;"yes"&lt;/code&gt;, &lt;code&gt;"on"&lt;/code&gt;, &lt;code&gt;"false"&lt;/code&gt;, &lt;code&gt;"0"&lt;/code&gt;, &lt;code&gt;"no"&lt;/code&gt;, &lt;code&gt;"off"&lt;/code&gt;, &lt;code&gt;""&lt;/code&gt;, &lt;code&gt;"none"&lt;/code&gt;, &lt;code&gt;"nil"&lt;/code&gt;. Then test that every unrecognized value crashes instead of defaulting.&lt;/li&gt;
&lt;li&gt;If a dry run has ever "worked," check the log for evidence that it took a different branch than the live run. Mine printed the same success line in both modes. That should have been impossible.&lt;/li&gt;
&lt;li&gt;Decouple nothing on the write path. Every side effect the live run can produce — emails, webhooks, queue rows, uploads — needs a stubbed twin in dry mode, or it isn't covered.&lt;/li&gt;
&lt;li&gt;Blame the code before the model. I spent forty minutes reading the system prompt looking for a reason the agent "decided" to run live. The agent did exactly what it was told. It was told by four lines of Python I wrote in a hurry.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The humiliating summary: I built a safety system out of a string, then wrote a test that only passed because the string was empty. Both took less time to write than this post. The emails took about forty seconds.&lt;/p&gt;

&lt;p&gt;I now have an &lt;code&gt;AGENT_MODE&lt;/code&gt; parser with eleven test cases, one write boundary, a read-only key for dry runs, and a startup check that will refuse to boot rather than guess. The agent hasn't changed at all. It was never the agent.&lt;/p&gt;




&lt;p&gt;I write up the specific playbooks — the config parsing, the write boundaries, the canary rules, the startup assertions — in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/0ce2371c-c75d-423c-b64d-685a00445048" rel="noopener noreferrer"&gt;The Solo Operator's AI Agent Playbook&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>automation</category>
      <category>productivity</category>
    </item>
    <item>
      <title>My Agent Checked the First 100 Rows Every Morning for 23 Days. The Other 4,312 Customers Got Nothing.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Sun, 08 Nov 2026 01:02:02 +0000</pubDate>
      <link>https://dev.to/ulnit/my-agent-checked-the-first-100-rows-every-morning-for-23-days-the-other-4312-customers-got-cek</link>
      <guid>https://dev.to/ulnit/my-agent-checked-the-first-100-rows-every-morning-for-23-days-the-other-4312-customers-got-cek</guid>
      <description>&lt;p&gt;For 23 days my morning digest said the same thing: &lt;strong&gt;"Checked 100 customer records. 0 issues. All clear."&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It was true. It was also the reason 4,312 customers got nothing.&lt;/p&gt;

&lt;p&gt;I want to write this one down carefully, because it's the least dramatic bug I've shipped and the one that cost the most in trust. No OOM killer, no leaked keys, no 2 AM emergency. Just a loop that stopped early and a report that reported the stop as success.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the job actually did
&lt;/h2&gt;

&lt;p&gt;The nightly job had one responsibility: walk the customer table, find anyone whose subscription had lapsed but who was still flagged active, and queue a "your card needs updating" email.&lt;/p&gt;

&lt;p&gt;Here is the code, more or less as it ran:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;audit_customers&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://api.internal.example/v1/customers&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;limit&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Authorization&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Bearer &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;TOKEN&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
        &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="n"&gt;stale&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;active&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;renewed_at&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;stale&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;queue_email&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;email&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;template&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;card_update&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;checked&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;queued&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;stale&lt;/span&gt;&lt;span class="p"&gt;)}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;limit=100&lt;/code&gt; with no cursor. The API returns the first page and a &lt;code&gt;next_cursor&lt;/code&gt; field that I never read. My table had 4,412 rows. So the job did exactly what I told it to do, 100 times over, on the same 100 rows.&lt;/p&gt;

&lt;p&gt;The digest line was generated directly from the return value:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Checked 100 customer records. 0 issues. All clear.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every morning, for 23 days. The number &lt;code&gt;100&lt;/code&gt; was right there in the message. I read it as "a hundred records, fine" instead of "exactly one hundred, suspiciously round, every single day."&lt;/p&gt;

&lt;h2&gt;
  
  
  The lesson I keep relearning
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;A count that never changes is an alarm, not a status.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I had built a report that described the work instead of validating it. &lt;code&gt;{"checked": 100}&lt;/code&gt; is a measurement, not an assertion. Nothing in the pipeline compared it to the size of the table, so nothing could fail.&lt;/p&gt;

&lt;p&gt;The fix took eleven minutes. The reason it took 23 days to find is that I had never written down what "done" meant for this job. "No errors" is not done. "Walked every eligible row exactly once" is done.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three changes I made
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Cursor pagination with a hard ceiling.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;iter_customers&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;page_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_pages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;cursor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;max_pages&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="n"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;limit&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;page_size&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;cursor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;cursor&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="p"&gt;{})},&lt;/span&gt;
            &lt;span class="n"&gt;headers&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Authorization&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Bearer &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;TOKEN&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
            &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;raise_for_status&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="n"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;resp&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;data&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="n"&gt;cursor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;next_cursor&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;cursor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt;
    &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;RuntimeError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pagination ceiling hit — result set may be truncated&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That last &lt;code&gt;raise&lt;/code&gt; matters more than the loop. If the job ever gets bigger than the ceiling I guessed, I want a loud failure, not a clean exit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Assert the coverage, not just the outcome.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;rows&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;iter_customers&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;span class="n"&gt;expected&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;scalar&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SELECT count(*) FROM customers WHERE status=&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;active&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;read &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; rows, table says &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;expected&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is an approximation — the table can change mid-run, and the API can filter differently than my SQL. That's fine. A rough equality check catches a 100-vs-4400 gap instantly, and that's the failure mode I was actually blind to.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Make the digest carry a ratio and a timestamp.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Before: &lt;code&gt;Checked 100 customer records. 0 issues.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;After: &lt;code&gt;Walked 4,412/4,412 active rows (cursor complete, 02:14–02:19 UTC). Queued 61.&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;Now the report can only say "complete" if the paginator actually reached the end. If it didn't, the digest says &lt;code&gt;PARTIAL — cursor never terminated&lt;/code&gt; and the job exits non-zero. I also stopped generating the summary from the same function that did the work. A separate check queries the watermark table and compares. Two independent opinions, one of which doesn't trust the other.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part I'm not proud of
&lt;/h2&gt;

&lt;p&gt;When the first customer emailed — day 23, politely, "I've been getting 'payment failed' notices for three weeks, is my account okay?" — my first instinct was to blame the provider. I checked the email queue, then the template, then the send logs, all clean. I spent forty minutes looking in the wrong place because the report had told me the sweep was fine and I believed my own dashboard over the customer.&lt;/p&gt;

&lt;p&gt;The genuinely embarrassing detail: I have a rule about this. Any customer-reported issue outranks any automated report. I wrote that rule after a monitoring outage in September where a reader noticed before my alerts did. And I still opened the queue logs first.&lt;/p&gt;

&lt;p&gt;Rules don't survive contact with a green dashboard. What survives is structure. So the extra thing I added wasn't a rule, it was a step:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;When a customer reports something the digest says is impossible, the digest is the suspect.&lt;/strong&gt; I put that sentence in the runbook next to the command to pull the raw row count. Not as advice — as step one, with a command to paste.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd tell you to check tonight
&lt;/h2&gt;

&lt;p&gt;If you have any cron job or agent task that reads from an API with a &lt;code&gt;limit&lt;/code&gt; parameter, go look at it right now. Three questions:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Does it follow &lt;code&gt;next_cursor&lt;/code&gt; / &lt;code&gt;page&lt;/code&gt; / &lt;code&gt;offset&lt;/code&gt; until it terminates?&lt;/strong&gt; If not, it processes a prefix and calls it the world.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Does it assert its coverage against an independent count?&lt;/strong&gt; If the only number in your log is the number of rows it decided to read, that number proves nothing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Does the digest ever print the same count twice in a row?&lt;/strong&gt; That's your cheapest alarm. A daily job whose output is byte-identical for a week is either perfectly stable or completely stuck, and you cannot tell which from the log.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The bug in my job was five characters wide: &lt;code&gt;100&lt;/code&gt;. Everything downstream was confident, well-formatted, and wrong. That's the shape most of my worst incidents take — not a crash, but a small unexamined assumption wearing the costume of a successful run.&lt;/p&gt;

&lt;p&gt;Twenty-three days. 4,312 people. One &lt;code&gt;limit=100&lt;/code&gt; I never thought about again after I typed it.&lt;/p&gt;




&lt;p&gt;I keep a running file of these — the specific playbooks for catching silent agent failures before a customer does, including the coverage-assertion patterns and digest formats I now use on every scheduled job.&lt;/p&gt;

&lt;p&gt;I write up the specific playbooks in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/0ce2371c-c75d-423c-b64d-685a00445048" rel="noopener noreferrer"&gt;The Solo Operator's AI Agent Playbook&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>programming</category>
      <category>productivity</category>
    </item>
    <item>
      <title>I Let My Agent Schedule Its Own Jobs. Three Weeks Later: 6,048 Runs, 12 Emails Answered Six Times.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Sat, 07 Nov 2026 01:02:40 +0000</pubDate>
      <link>https://dev.to/ulnit/i-let-my-agent-schedule-its-own-jobs-three-weeks-later-6048-runs-12-emails-answered-six-times-1p8</link>
      <guid>https://dev.to/ulnit/i-let-my-agent-schedule-its-own-jobs-three-weeks-later-6048-runs-12-emails-answered-six-times-1p8</guid>
      <description>&lt;p&gt;The agent proposed this line and I approved it in about four seconds:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;*/&lt;span class="m"&gt;5&lt;/span&gt; * * * * /&lt;span class="n"&gt;home&lt;/span&gt;/&lt;span class="n"&gt;sean&lt;/span&gt;/&lt;span class="n"&gt;venv&lt;/span&gt;/&lt;span class="n"&gt;bin&lt;/span&gt;/&lt;span class="n"&gt;python&lt;/span&gt; /&lt;span class="n"&gt;home&lt;/span&gt;/&lt;span class="n"&gt;sean&lt;/span&gt;/&lt;span class="n"&gt;agent&lt;/span&gt;/&lt;span class="n"&gt;inbox&lt;/span&gt;.&lt;span class="n"&gt;py&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every five minutes. Check the inbox. It reads fine, doesn't it? That is exactly the problem — a crontab line reads like a sentence, so I reviewed it like a sentence. I never converted it into the only number that matters: &lt;strong&gt;how many times does this actually run?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;6,048 times, over three weeks. My intent was "a few times an hour, during the day." I got every five minutes, around the clock, including 3 a.m. on Sundays. And because each run took longer than five minutes once there was mail to answer, I got overlapping runs that read the same twelve unprocessed emails and replied to each one six times.&lt;/p&gt;

&lt;p&gt;Two incidents in one. The schedule was wrong, and the code underneath it couldn't survive a wrong schedule. Both were mine.&lt;/p&gt;

&lt;h2&gt;
  
  
  Act one: four days of silence nobody noticed
&lt;/h2&gt;

&lt;p&gt;The job ran zero useful times for the first four days. &lt;code&gt;load_dotenv()&lt;/code&gt; and a relative &lt;code&gt;config.yaml&lt;/code&gt; path meant every run died instantly, because cron's working directory is &lt;code&gt;$HOME&lt;/code&gt;, not my project directory. Cron also doesn't load your shell profile, so &lt;code&gt;PATH&lt;/code&gt; is roughly &lt;code&gt;/usr/bin:/bin&lt;/code&gt; and your &lt;code&gt;.env&lt;/code&gt; does not exist as far as the job is concerned.&lt;/p&gt;

&lt;p&gt;Nothing alerted me, because the end of the crontab line looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;*/5 * * * * /home/sean/venv/bin/python /home/sean/agent/inbox.py &amp;gt; /dev/null 2&amp;gt;&amp;amp;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I added &lt;code&gt;2&amp;gt;&amp;amp;1&lt;/code&gt; at some point, feeling responsible, then aimed the whole thing at the void. Four days of a dead job producing no output, which in a log directory looks identical to a job that had nothing to do.&lt;/p&gt;

&lt;p&gt;The monitoring was a customer asking "did you get my message?"&lt;/p&gt;

&lt;p&gt;Fix: absolute interpreter path, &lt;code&gt;cd /home/sean/agent&lt;/code&gt; at the top of the wrapper, &lt;code&gt;set -euo pipefail&lt;/code&gt;, and every line appended to a log with a timestamp and exit code. Straightforward, twenty minutes. Then it started actually sending email, which is when it got worse.&lt;/p&gt;

&lt;h2&gt;
  
  
  Act two: the check-then-act loop
&lt;/h2&gt;

&lt;p&gt;Here is the loop the agent filled in from my pattern:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;pending&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;query&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;SELECT * FROM messages WHERE replied = 0&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;pending&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;reply&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;generate_reply&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                    &lt;span class="c1"&gt;# 40-90s, model call
&lt;/span&gt;    &lt;span class="nf"&gt;send&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;reply&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;UPDATE messages SET replied = 1 WHERE id = ?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;m&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The flag is written at the &lt;em&gt;end&lt;/em&gt;. A run that starts at 14:05 and finishes at 14:06 overlaps a run that started at 14:10 and read the same rows while they were still &lt;code&gt;replied = 0&lt;/code&gt;. Check-then-act, with a 90-second window between the check and the act. Twelve messages, six runs each, seventy-two outgoing replies.&lt;/p&gt;

&lt;p&gt;I found out because a customer forwarded two of my replies, four minutes apart, and asked politely whether she was talking to a robot. She was, in the sense that mattered.&lt;/p&gt;

&lt;p&gt;The mail provider didn't stop me. My rate limiting didn't stop me — it was per-run, and each run was well under its own limit. Nothing on the box was in a position to say "you already answered this," because the only component that knew the truth was the database, and I never asked it.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed, in order of how much it mattered
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Claim the window before doing the work, not after.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;agent_windows&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;window_key&lt;/span&gt;  &lt;span class="nb"&gt;TEXT&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;   &lt;span class="c1"&gt;-- 'inbox:2026-11-07T14:05'&lt;/span&gt;
  &lt;span class="n"&gt;started_at&lt;/span&gt;  &lt;span class="n"&gt;TIMESTAMPTZ&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
  &lt;span class="n"&gt;finished_at&lt;/span&gt; &lt;span class="n"&gt;TIMESTAMPTZ&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;INSERT INTO agent_windows (window_key) VALUES (%s) &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ON CONFLICT DO NOTHING RETURNING window_key&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,)&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;fetchone&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# someone else owns this window
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And a &lt;code&gt;UNIQUE(message_id)&lt;/code&gt; on the replies table, so even a duplicate attempt fails loudly at the storage layer instead of reaching a human's inbox. Both of these are two minutes of work. Either one alone would have prevented all seventy-two messages.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. &lt;code&gt;flock&lt;/code&gt; as a belt, not as the plan.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;flock &lt;span class="nt"&gt;-n&lt;/span&gt; /tmp/agent-inbox.lock &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'/home/sean/bin/inbox-wrapper.sh'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This costs one line and stops same-host overlap. It is not a fix: &lt;code&gt;/tmp&lt;/code&gt; is per-host and cleared on reboot, and a run that dies mid-flight leaves a stale &lt;code&gt;started_at&lt;/code&gt; row that only the ledger can help you clean up. I use both. I trust one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Write the schedule as a number, then verify the log.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;*/5 * * * *&lt;/code&gt; → 288 runs/day → 8,640/month.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;*/5 8-18 * * 1-5&lt;/code&gt; → 132 runs on a weekday, zero at 3 a.m.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I now count runs instead of reading the crontab, because a crontab is a wish and a log is evidence:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;journalctl &lt;span class="nt"&gt;-u&lt;/span&gt; agent-inbox &lt;span class="nt"&gt;--since&lt;/span&gt; &lt;span class="s2"&gt;"-7 days"&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s2"&gt;"run start"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If that number doesn't match what you expected, you've found your bug before a customer does.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Jitter, because every &lt;code&gt;*/5&lt;/code&gt; job on the box fires at &lt;code&gt;:00&lt;/code&gt;.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;sleep&lt;/span&gt; &lt;span class="k"&gt;$((&lt;/span&gt;RANDOM &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="m"&gt;45&lt;/span&gt;&lt;span class="k"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Cheap, and it keeps the model API from seeing twenty simultaneous requests from one Raspberry Pi at the top of every five minutes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Alert on the absence of runs, not just on errors.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A heartbeat file touched at the end of every successful run, and an alert when its mtime is older than twice the interval. I also alert when a single run exceeds three times the median duration — that's the leading indicator for overlap, and it fires minutes before duplicates do. My old alerts only triggered on exceptions. Every expensive failure in this story was a success: exit code 0, seventy-two times.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. systemd timers for anything that does real work.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Type=oneshot&lt;/code&gt; won't start a second copy of the same unit while the first is still running — that property alone would have made this incident impossible. Add &lt;code&gt;RandomizedDelaySec=30&lt;/code&gt; for jitter, &lt;code&gt;Persistent=true&lt;/code&gt; if you want catch-up after downtime, and check your calendar expression before you trust it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;systemd-analyze calendar &lt;span class="s1"&gt;'Mon..Fri 08..18:00/5:00'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That command prints the next elapse times. Run it. It's the check I skipped.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I actually got wrong
&lt;/h2&gt;

&lt;p&gt;I let the agent write its own infrastructure, then reviewed the result as prose. For years I've refused to merge a cron change from a human without a "runs per day" number in the description, and I gave a language model a pass because the diff was one line and the one line sounded reasonable.&lt;/p&gt;

&lt;p&gt;The agent wasn't wrong, either. "Check every five minutes" is very close to what I said out loud. It was obedient, literal, and fast — which is precisely the combination that punishes a vague review. The sloppiness was entirely in the approval.&lt;/p&gt;

&lt;p&gt;Two things I'd tell myself from three weeks ago:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The non-idempotent loop was mine. The agent copied my pattern. In my incident log, six of the last nine entries are my code being ordinary, not the model being strange. "The agent did it" is a story I tell because it's more interesting than "I wrote a race condition on a Tuesday."&lt;/li&gt;
&lt;li&gt;Make the schedule self-reporting. Mine now computes and prints &lt;code&gt;runs_per_day&lt;/code&gt; at startup, so every deploy logs a number I can disagree with. That one line has caught two bad schedules since, before they shipped.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The whole fix is about forty lines: a primary key, a unique constraint, a lock, a jitter sleep, and a heartbeat that complains when it stops being touched. What it replaced was my belief that a schedule I hadn't measured was a schedule I understood.&lt;/p&gt;

&lt;p&gt;Go count your runs. &lt;code&gt;grep -c&lt;/code&gt; your own log right now. If the number surprises you, the duplicates are already in someone's inbox and you're just waiting for the forward.&lt;/p&gt;




&lt;p&gt;I write up the specific playbooks in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/0ce2371c-c75d-423c-b64d-685a00445048" rel="noopener noreferrer"&gt;The Solo Operator's AI Agent Playbook&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.90. Enter it in the discount field on the checkout page; the link won't apply it for you. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>programming</category>
      <category>productivity</category>
    </item>
    <item>
      <title>My Agent Wrote 1,842 'Success' Logs to a Full Disk. Python's Logging Module Never Complained Once.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Fri, 06 Nov 2026 01:01:10 +0000</pubDate>
      <link>https://dev.to/ulnit/my-agent-wrote-1842-success-logs-to-a-full-disk-pythons-logging-module-never-complained-once-2aej</link>
      <guid>https://dev.to/ulnit/my-agent-wrote-1842-success-logs-to-a-full-disk-pythons-logging-module-never-complained-once-2aej</guid>
      <description>&lt;p&gt;I found out my agent's observability was fake on a Tuesday afternoon, while trying to answer a boring customer question: "did the nightly report run on the 14th?"&lt;/p&gt;

&lt;p&gt;I grepped the log file. Nothing after the 9th.&lt;/p&gt;

&lt;p&gt;The dashboard said 100% success. The agent had posted &lt;code&gt;done&lt;/code&gt; in Slack every single night. The queue table said the job completed. And there was not one byte of evidence for six days of work.&lt;/p&gt;

&lt;p&gt;The disk was full. It had been full since the previous Wednesday.&lt;/p&gt;

&lt;h2&gt;
  
  
  The numbers, because the numbers are the point
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;df -h&lt;/code&gt; on the Pi:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/dev/root        29G   29G     0  100% /
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then &lt;code&gt;du -xh --max-depth=2 / | sort -rh | head&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;5.1G  /var/log
4.2G  /home/agent/logs
1.8G  /home/agent/.cache
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two log directories. 9.3GB of logs on a 29GB card. The agent's own file was &lt;code&gt;agent.log&lt;/code&gt;, and next to it sat &lt;code&gt;agent.log.1&lt;/code&gt;, &lt;code&gt;agent.log.2.gz&lt;/code&gt;... except &lt;code&gt;logrotate&lt;/code&gt; had never been installed, so there was no rotation at all. Just one file, 4.2GB, opened with &lt;code&gt;logging.FileHandler&lt;/code&gt; in append mode since August.&lt;/p&gt;

&lt;p&gt;Why so big? Because I had turned on &lt;code&gt;DEBUG&lt;/code&gt; in March to chase a bug and never turned it off. Every HTTP request logged headers. Every prompt logged in full. Every model response logged in full, twice — once on receipt, once in the summary line that included the whole body "for convenience."&lt;/p&gt;

&lt;p&gt;That's roughly 700MB a day of a language model explaining itself to me in a file nobody reads.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why nothing screamed
&lt;/h2&gt;

&lt;p&gt;This is the part that actually scared me. Four separate things failed to tell me the truth, and each one is a normal, defensible default.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Python's &lt;code&gt;logging&lt;/code&gt; module swallows handler errors by design.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;

&lt;span class="n"&gt;log&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getLogger&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;agent&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;addHandler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;FileHandler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/home/agent/logs/agent.log&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setLevel&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;DEBUG&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;When the disk fills, &lt;code&gt;FileHandler.emit()&lt;/code&gt; raises &lt;code&gt;OSError: [Errno 28] No space left on device&lt;/code&gt;. &lt;code&gt;Handler.handleError()&lt;/code&gt; catches it. If &lt;code&gt;logging.raiseExceptions&lt;/code&gt; is &lt;code&gt;True&lt;/code&gt; — the default — it prints a traceback to &lt;code&gt;sys.stderr&lt;/code&gt; and returns. The logging call itself does not raise. Your &lt;code&gt;log.info(...)&lt;/code&gt; returns &lt;code&gt;None&lt;/code&gt;. Your agent continues, cheerfully, into the void.&lt;/p&gt;

&lt;p&gt;And my &lt;code&gt;stderr&lt;/code&gt;? It was redirected to &lt;code&gt;agent.err&lt;/code&gt;, on the same full disk. So the traceback about the disk being full could not be written to the disk that was full.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. My success markers were in the same place as the logs.&lt;/strong&gt; The "job done" record was a JSON line appended to a file. The append failed. The code didn't check the return value, because &lt;code&gt;open(..., "a").write()&lt;/code&gt; doesn't return one. The Slack ping fired anyway, because the Slack call came &lt;em&gt;before&lt;/em&gt; the state write in my code, not after.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The queue thought the job never finished.&lt;/strong&gt; My worker marked tasks complete by writing to SQLite. &lt;code&gt;sqlite3.OperationalError: database or disk is full&lt;/code&gt; — caught by my broad &lt;code&gt;except Exception:&lt;/code&gt; and logged at &lt;code&gt;WARNING&lt;/code&gt;. Which, per point 1, went nowhere. The task stayed &lt;code&gt;pending&lt;/code&gt;. It ran again the next night. And the next. Five nights of duplicate runs of a job that emails a summary.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. My uptime monitor checked the process, not the outputs.&lt;/strong&gt; &lt;code&gt;systemctl is-active agent.service&lt;/code&gt; returned &lt;code&gt;active&lt;/code&gt; the entire time. Of course it did. The process was healthy. It was just blind, mute, and repeating itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  The near-miss
&lt;/h2&gt;

&lt;p&gt;The duplicate runs were the dangerous part, not the disk. The nightly job compiles a per-customer usage summary and mails it. On the fifth night I happened to be awake when the send started, noticed the same subject line twice in my own inbox, and killed it.&lt;/p&gt;

&lt;p&gt;I checked the send-log table. Nothing duplicate — because that table lived in the same SQLite file that couldn't be written. The dedupe key I was so proud of was &lt;code&gt;INSERT OR IGNORE&lt;/code&gt; into a table that was failing every insert. Deduplication that depends on a write, on a disk that can't write, is not deduplication. It's a coin flip.&lt;/p&gt;

&lt;p&gt;If I hadn't been awake, forty-one customers would have gotten the same email twice, on a Saturday, with numbers that were five days stale.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I changed, in order of how much it mattered
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. A canary write at the top of every run.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;preflight&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;probe&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/home/agent/state/.canary&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;probe&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;write_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;str&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;time&lt;/span&gt;&lt;span class="p"&gt;()))&lt;/span&gt;
    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;OSError&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;notify_critical&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;agent cannot write state: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;SystemExit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;78&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# fail closed, not open
&lt;/span&gt;    &lt;span class="n"&gt;free_gb&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;shutil&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;disk_usage&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;free&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;1e9&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;free_gb&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="nf"&gt;notify_critical&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;disk low: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;free_gb&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;GB free&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;raise&lt;/span&gt; &lt;span class="nc"&gt;SystemExit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;78&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The critical detail is &lt;code&gt;SystemExit&lt;/code&gt;. My first version logged a warning and continued. A warning on a broken logging system is a thought you had and didn't say out loud.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. &lt;code&gt;logging.raiseExceptions&lt;/code&gt; is not enough — use a handler that escalates.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;LoudFileHandler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;FileHandler&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;handleError&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="c1"&gt;# Do not swallow. Route to a place that outlives this disk.
&lt;/span&gt;        &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;urllib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;request&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;https://ntfy.sh/my-agent-alerts&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;log write failed: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;exc_info&lt;/span&gt;&lt;span class="p"&gt;()[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
                &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
            &lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;finally&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;raise&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;An out-of-band channel is the whole trick. Anything that reports failure &lt;em&gt;through the system that is failing&lt;/em&gt; is decoration.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Rotation, and a cap on verbosity that isn't a comment in a file.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight conf"&gt;&lt;code&gt;/&lt;span class="n"&gt;home&lt;/span&gt;/&lt;span class="n"&gt;agent&lt;/span&gt;/&lt;span class="n"&gt;logs&lt;/span&gt;/*.&lt;span class="n"&gt;log&lt;/span&gt; {
    &lt;span class="n"&gt;daily&lt;/span&gt;
    &lt;span class="n"&gt;rotate&lt;/span&gt; &lt;span class="m"&gt;7&lt;/span&gt;
    &lt;span class="n"&gt;maxsize&lt;/span&gt; &lt;span class="m"&gt;200&lt;/span&gt;&lt;span class="n"&gt;M&lt;/span&gt;
    &lt;span class="n"&gt;compress&lt;/span&gt;
    &lt;span class="n"&gt;delaycompress&lt;/span&gt;
    &lt;span class="n"&gt;missingok&lt;/span&gt;
    &lt;span class="n"&gt;notifempty&lt;/span&gt;
    &lt;span class="n"&gt;copytruncate&lt;/span&gt;
}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;copytruncate&lt;/code&gt; because my agent holds the file descriptor open and I didn't want to restart it to rotate. Yes, it can lose lines written during the copy. For debug logs, that's an acceptable trade. For the audit trail, I moved to a separate append-only file with &lt;code&gt;O_APPEND&lt;/code&gt; and its own budget.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Success is a row, not a message.&lt;/strong&gt; The Slack ping now fires &lt;em&gt;after&lt;/em&gt; a committed write, and it includes the row id. If the write fails, there is no ping. Silence is the alarm.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. &lt;code&gt;df&lt;/code&gt; is a metric, not a command I run when things feel weird.&lt;/strong&gt; One line in the existing health check, exporting free bytes. It's the cheapest monitoring I have and it was the one thing I wasn't doing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The honest part
&lt;/h2&gt;

&lt;p&gt;Two confessions, because the fixes above are the easy half.&lt;/p&gt;

&lt;p&gt;First: I had a disk-space warning for three weeks before this. It was a &lt;code&gt;WARNING&lt;/code&gt; line in a log I wasn't reading, emitted by a check I'd written as a placeholder. I wrote the check, felt productive, and never wired the output anywhere. I have now done this enough times to state the rule plainly — a check that reports into the system it's checking is not a check. It's a diary entry.&lt;/p&gt;

&lt;p&gt;Second, and worse: during the six-day window, a customer reported that the agent had given them a wrong number. I spent an afternoon writing about context windows and model drift, and I posted a careful internal note about "increased hallucination rates on long prompts." There were no logs for that window. There was no prompt to inspect. I had invented a theory to explain an absence of evidence, and I'd started to believe it. The actual cause was a stale input file — the upstream fetch had been failing silently since the 9th for the same disk reason — and the model had done something reasonable with five-day-old data.&lt;/p&gt;

&lt;p&gt;I blamed the model for my plumbing. That's the most common mistake I make, and this time I have a 4.2GB log file with a hole in it to prove it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The checklist I now actually run
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;df&lt;/code&gt; free bytes exported as a metric, alert at 15%&lt;/li&gt;
&lt;li&gt;canary write before every job, fail closed&lt;/li&gt;
&lt;li&gt;log write failures escalate out-of-band&lt;/li&gt;
&lt;li&gt;rotation configured, &lt;code&gt;DEBUG&lt;/code&gt; behind an env var, never a code edit&lt;/li&gt;
&lt;li&gt;success notification sent only after a committed state write&lt;/li&gt;
&lt;li&gt;dedupe keys stored somewhere with a different failure mode than the send path&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of it is clever. All of it took an afternoon. The six days of missing evidence cost me a week of debugging and one customer's trust, which is a much worse exchange rate.&lt;/p&gt;

&lt;p&gt;If you run an agent unattended on a Pi, go check &lt;code&gt;df -h&lt;/code&gt; right now. Then check whether your logger can tell you it's broken. Mine couldn't, for six days, while reporting success 1,842 times.&lt;/p&gt;




&lt;p&gt;I keep the operational patterns that came out of incidents like this in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/af4c3237-d411-4fbd-87b9-d5a562e55e4c" rel="noopener noreferrer"&gt;The Agent Prompt Vault&lt;/a&gt; — $3, lifetime updates. Steal the ones that fit your workflow.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>raspberrypi</category>
      <category>automation</category>
      <category>programming</category>
    </item>
    <item>
      <title>I chmod 600'd My Agent's .env and Called It Done. The Key Was in Four Other Places, Starting With My Shell History.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Thu, 05 Nov 2026 01:04:01 +0000</pubDate>
      <link>https://dev.to/ulnit/i-chmod-600d-my-agents-env-and-called-it-done-the-key-was-in-four-other-places-including-a-56jd</link>
      <guid>https://dev.to/ulnit/i-chmod-600d-my-agents-env-and-called-it-done-the-key-was-in-four-other-places-including-a-56jd</guid>
      <description>&lt;h2&gt;
  
  
  The audit started because a reader asked me a question I couldn't answer
&lt;/h2&gt;

&lt;p&gt;A reader emailed me in October: "How do you rotate your agent's API key?"&lt;/p&gt;

&lt;p&gt;My honest answer was: I open &lt;code&gt;.env&lt;/code&gt;, paste a new key, restart the service. That's not a rotation, it's a replacement. And I had no idea whether the old key still worked anywhere.&lt;/p&gt;

&lt;p&gt;So I did the laziest possible audit. One grep, across my home directory and &lt;code&gt;/etc&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rIl&lt;/span&gt; &lt;span class="nt"&gt;-E&lt;/span&gt; &lt;span class="s1"&gt;'sk-[A-Za-z0-9]{20,}|ghp_[A-Za-z0-9]{20,}|AKIA[0-9A-Z]{16}'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  ~ /etc/systemd/system 2&amp;gt;/dev/null
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nine files. My &lt;code&gt;.env&lt;/code&gt; was one of them. The other eight were the interesting part, and none of them were a file I would have called a secrets file.&lt;/p&gt;

&lt;p&gt;This is the post I wish I'd read seven months ago, when I thought protecting a secret meant protecting one file.&lt;/p&gt;

&lt;h2&gt;
  
  
  What "chmod 600" actually buys you
&lt;/h2&gt;

&lt;p&gt;Here's the threat model I thought I had: &lt;em&gt;another user on the box reads my key.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;That's a real threat and &lt;code&gt;600&lt;/code&gt; closes it. What it does not close is: &lt;strong&gt;anything running as the same user reads my key.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;My agent runs as a dedicated user, &lt;code&gt;agent&lt;/code&gt;. Fine. But it also has a shell tool: it can run &lt;code&gt;bash&lt;/code&gt;, &lt;code&gt;printenv&lt;/code&gt;, &lt;code&gt;python3 -c "import os; print(os.environ)"&lt;/code&gt;, and read &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/environ&lt;/code&gt; for any sibling process — same UID, and &lt;code&gt;/proc&lt;/code&gt; doesn't stop you.&lt;/p&gt;

&lt;p&gt;So &lt;code&gt;600&lt;/code&gt; is not a lock. It's a fence between users. If the thing you're most afraid of is your own agent being manipulated into exfiltrating something — a prompt injection, a malicious dependency, a tool call you didn't intend — then &lt;code&gt;chmod 600&lt;/code&gt; protects you from exactly nothing, because the attacker is already inside the fence. It's running as &lt;code&gt;agent&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;I knew this in the abstract. I did not know it in the way that makes you change a systemd unit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Place 1: my shell history, at &lt;code&gt;644&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;In August I spent three hours debugging a 401. At some point I ran something like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;export &lt;/span&gt;&lt;span class="nv"&gt;OPENAI_API_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;sk-proj-XXXXXXXXXXXX python3 test_call.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It worked. I moved on. &lt;code&gt;~/.bash_history&lt;/code&gt; still had that line, and unlike my &lt;code&gt;.env&lt;/code&gt;, it was mode &lt;code&gt;644&lt;/code&gt; — world-readable on a box where I'd once given a friend an SSH account to look at a Grafana dashboard.&lt;/p&gt;

&lt;p&gt;The irony is specific: I had spent an afternoon hardening a file, then left the same string in plaintext in a file I never thought of as containing secrets. A history file is a log of everything I've ever typed, which makes it a worse secrets file than &lt;code&gt;.env&lt;/code&gt; by every measure.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I do now:&lt;/strong&gt; a &lt;code&gt;HISTCONTROL=ignoreboth&lt;/code&gt; plus &lt;code&gt;HISTIGNORE='*KEY*:*TOKEN*:*sk-*:*password*'&lt;/code&gt; in my shell rc, and if I need to test a key interactively I read it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;read&lt;/span&gt; &lt;span class="nt"&gt;-rs&lt;/span&gt; KEY &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; curl &lt;span class="nt"&gt;-H&lt;/span&gt; &lt;span class="s2"&gt;"Authorization: Bearer &lt;/span&gt;&lt;span class="nv"&gt;$KEY&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; ...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's four extra seconds and keeps the string out of the file that syncs to every backup I own.&lt;/p&gt;

&lt;h2&gt;
  
  
  Place 2: the systemd unit file, which is world-readable by default
&lt;/h2&gt;

&lt;p&gt;For a while my unit had this in it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;Environment&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;OPENAI_API_KEY=sk-proj-XXXXXXXXXXXX&lt;/span&gt;
&lt;span class="py"&gt;Environment&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;AGENT_DB=/home/agent/state.db&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;/etc/systemd/system/agent.service&lt;/code&gt; is created &lt;code&gt;644&lt;/code&gt;, and &lt;code&gt;systemctl cat agent.service&lt;/code&gt; prints the whole thing to any local user without a password prompt. I had put the secret in the one place on the machine designed to be readable.&lt;/p&gt;

&lt;p&gt;The migration was small:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;User&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;agent&lt;/span&gt;
&lt;span class="py"&gt;EnvironmentFile&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/agent/secrets.env&lt;/span&gt;
&lt;span class="c"&gt;# secrets.env: root:root, 0600, and systemd reads it as root
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's better. It is not perfect — the process still has the key in its environment, and the process is what I'm worried about. So for the key that matters most I went further: Debian 12 ships systemd 252, so &lt;code&gt;LoadCredential=&lt;/code&gt; exists:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="py"&gt;LoadCredential&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;openai_key:/etc/agent/openai.key&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The agent reads &lt;code&gt;$CREDENTIALS_DIRECTORY/openai_key&lt;/code&gt;, a &lt;code&gt;tmpfs&lt;/code&gt; path scoped to the unit, and the value never enters the environment block at all. &lt;code&gt;systemctl show -p Environment&lt;/code&gt; returns nothing useful. It took twenty minutes, and it's the single change I'd keep if I had to throw the rest away.&lt;/p&gt;

&lt;h2&gt;
  
  
  Place 3: a crash dump that printed my headers
&lt;/h2&gt;

&lt;p&gt;In September a &lt;code&gt;requests&lt;/code&gt; call in my worker raised on a bad gateway. My handler logged it with &lt;code&gt;exc_info=True&lt;/code&gt;, and the exception repr included the &lt;code&gt;PreparedRequest&lt;/code&gt; — whose repr includes headers. Authorization header, in full, into journald.&lt;/p&gt;

&lt;p&gt;Survivable, except for two settings I'd made months earlier and forgotten:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# journald.conf
&lt;/span&gt;&lt;span class="py"&gt;Storage&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;persistent&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and, worse, I had added &lt;code&gt;agent&lt;/code&gt; to the &lt;code&gt;adm&lt;/code&gt; group back in June to read logs without sudo, and never removed it. So the box had a persistent, on-disk copy of the key in &lt;code&gt;/var/log/journal&lt;/code&gt;, in a file the compromised process could read.&lt;/p&gt;

&lt;p&gt;You can't stop a library printing a header in a traceback. You can stop it at the formatter:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;

&lt;span class="n"&gt;SECRET_RE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;compile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;(sk-[A-Za-z0-9_\-]{8})[A-Za-z0-9_\-]+&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="nc"&gt;Redact&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Filter&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getMessage&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;SECRET_RE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;Authorization&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;msg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;SECRET_RE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;sub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;\1…REDACTED&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;True&lt;/span&gt;

&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;getLogger&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;addFilter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Redact&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It's not clever and won't catch everything. It catches the case that actually happened, which is how I judge every control on this box.&lt;/p&gt;

&lt;h2&gt;
  
  
  Place 4: the backup tarball on a share with guest read
&lt;/h2&gt;

&lt;p&gt;My nightly backup was a tar of &lt;code&gt;/home/agent&lt;/code&gt; into &lt;code&gt;/home/agent/backups&lt;/code&gt;, &lt;code&gt;rsync&lt;/code&gt;'d to a NAS whose share had guest read because I set it up in 2024 to move photos and never revisited it. The tarball inherited the default umask, &lt;code&gt;022&lt;/code&gt;, so it was &lt;code&gt;644&lt;/code&gt; on the NAS too. And of course it contained &lt;code&gt;.env&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The whole hardening exercise sat downstream of a pipeline that mailed the key to a share my TV can browse.&lt;/p&gt;

&lt;p&gt;The fix that mattered wasn't file modes, it was exclusion plus encryption:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;umask &lt;/span&gt;077
&lt;span class="nb"&gt;tar&lt;/span&gt; &lt;span class="nt"&gt;--exclude&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'.env'&lt;/span&gt; &lt;span class="nt"&gt;--exclude&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'*.key'&lt;/span&gt; &lt;span class="nt"&gt;--exclude&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'backups'&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
    &lt;span class="nt"&gt;-czf&lt;/span&gt; - /home/agent | age &lt;span class="nt"&gt;-r&lt;/span&gt; &lt;span class="nv"&gt;$AGE_PUBKEY&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; agent-&lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;date&lt;/span&gt; +%F&lt;span class="si"&gt;)&lt;/span&gt;.tar.gz.age
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Excluding &lt;code&gt;.env&lt;/code&gt; means a restore needs one manual step, which I've written down. That's a real tradeoff and I'd rather have it than a searchable copy of my keys on a consumer NAS.&lt;/p&gt;

&lt;h2&gt;
  
  
  The honest part: I fixed this and then broke it again
&lt;/h2&gt;

&lt;p&gt;Here's the failure I'm least proud of.&lt;/p&gt;

&lt;p&gt;After the audit I rotated the key properly. New key created, old key revoked via the provider's API, &lt;code&gt;.env&lt;/code&gt; updated, service restarted, a note in the changelog with the date. I felt genuinely good about it.&lt;/p&gt;

&lt;p&gt;Eleven days later staging broke. Staging had its own &lt;code&gt;.env&lt;/code&gt;, in a directory I'd forgotten existed, still holding the revoked key. My fix — I want to be precise about this — was to create a new key and paste it into &lt;code&gt;staging/.env&lt;/code&gt;, &lt;code&gt;prod/.env&lt;/code&gt;, the unit file's &lt;code&gt;Environment=&lt;/code&gt;, and a Postman environment. I recreated the exact problem I had just spent a weekend solving, because the pressure was "make staging work in the next ten minutes," and the correct fix (one key per environment, one source of truth) was a two-hour job I did not have two hours for.&lt;/p&gt;

&lt;p&gt;Two weeks later I found the new key in a &lt;code&gt;.env.bak&lt;/code&gt; I'd created while editing. Which is how secrets actually escape: not through one dramatic leak, but through a series of reasonable ten-minute decisions made by a tired person who has already solved this problem once and resents solving it again.&lt;/p&gt;

&lt;p&gt;The thing that broke the loop wasn't discipline. It was making rotation a single script that writes the file, restarts the unit, and prints the revocation command — so the fast path and the correct path became the same path.&lt;/p&gt;

&lt;h2&gt;
  
  
  The takeaways, in the order I'd apply them
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Write down your threat model before your controls.&lt;/strong&gt; Mine was "another user reads the file." Every control I had was correct for that threat and useless for the real one: my own agent, running as me.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grep for the key shape, not the filename.&lt;/strong&gt; &lt;code&gt;sk-&lt;/code&gt;, &lt;code&gt;ghp_&lt;/code&gt;, &lt;code&gt;AKIA&lt;/code&gt;, plus your provider's prefix. Secrets live in history, unit files, logs, backups, and &lt;code&gt;.bak&lt;/code&gt; files, and none of those look like secrets.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Default to &lt;code&gt;umask 077&lt;/code&gt;&lt;/strong&gt; in anything that writes files: services, cron jobs, backup scripts. One line, closes a whole class.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Get keys out of &lt;code&gt;Environment=&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;EnvironmentFile=&lt;/code&gt; with root-owned &lt;code&gt;0600&lt;/code&gt; is the ten-minute version; &lt;code&gt;LoadCredential=&lt;/code&gt; is the twenty-minute version and strictly better.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Redact at the formatter&lt;/strong&gt; — you cannot control what a library puts in an exception string.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat group memberships as secret surface.&lt;/strong&gt; &lt;code&gt;adm&lt;/code&gt; can read persistent journald. I added myself for convenience and never subtracted.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The uncomfortable summary is that &lt;code&gt;chmod 600&lt;/code&gt; made me feel done, and feeling done is what stopped me looking for seven months. A permission bit answers a question about &lt;em&gt;other users&lt;/em&gt;. It says nothing about the process you're trying to contain, and that process is usually the one with the shell.&lt;/p&gt;




&lt;p&gt;The full checklist + scripts are in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/763b023d-bfb5-475d-ab28-9ba0e9ba142d" rel="noopener noreferrer"&gt;Ship Safe — The Launch-Day Security Kit&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.50. Enter it in the discount field on the checkout page; the link won't apply it for you.&lt;/p&gt;

&lt;p&gt;If you've run this audit on your own box, I'd like to know where your key turned up. Mine was shell history.&lt;/p&gt;

</description>
      <category>security</category>
      <category>ai</category>
      <category>automation</category>
      <category>raspberrypi</category>
    </item>
    <item>
      <title>My Agent Spawned 400 Workers Overnight. The OOM Killer Killed My Database and Left the Agent Running.</title>
      <dc:creator>ULNIT</dc:creator>
      <pubDate>Wed, 04 Nov 2026 01:01:30 +0000</pubDate>
      <link>https://dev.to/ulnit/my-agent-spawned-400-workers-overnight-the-oom-killer-killed-my-database-and-left-the-agent-315l</link>
      <guid>https://dev.to/ulnit/my-agent-spawned-400-workers-overnight-the-oom-killer-killed-my-database-and-left-the-agent-315l</guid>
      <description>&lt;p&gt;My agent had been running fine for eleven weeks when the Raspberry Pi stopped answering. Not crashed — &lt;em&gt;answering&lt;/em&gt;, but slowly, in that way that makes you think the network is bad. SSH took forty seconds to give me a password prompt. Then it dumped me into a shell and nothing responded to anything.&lt;/p&gt;

&lt;p&gt;At 3:14 AM I pulled the power, because that's what you do at 3:14 AM. When it came back, the database file was intact but the last six hours of writes were gone, and my queue had 412 rows stuck in &lt;code&gt;running&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;The agent process itself was still alive and had been the whole time. That was the part that took me a week to understand: &lt;strong&gt;the thing that failed was the innocent bystander, and the thing that caused the outage was the only thing that survived.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I actually built
&lt;/h2&gt;

&lt;p&gt;The Pi runs a small document-processing agent. It watches a table for pending jobs, picks one up, and shells out to a Python subprocess that renders a PDF, extracts text, and writes structured results back. Each subprocess uses a headless browser, so it's ~200-350 MB of RSS while it's working.&lt;/p&gt;

&lt;p&gt;There was a concurrency limit. I want to be precise about this, because "concurrency limit" is the phrase I used for two months and it was a lie:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;MAX_WORKERS&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;4&lt;/span&gt;   &lt;span class="c1"&gt;# I set this
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That constant was read in exactly one place: a log line. &lt;code&gt;logger.info(f"starting, max_workers={MAX_WORKERS}")&lt;/code&gt;. Every code path that actually started a worker did so from a &lt;code&gt;for row in pending:&lt;/code&gt; loop with no check at all, because the loop was written before the constant existed and I never went back.&lt;/p&gt;

&lt;p&gt;So when a bulk import dropped 400 documents in at 9 PM, the agent did what it was told: it walked the list and spawned. Not 4 at a time. All of them, as fast as &lt;code&gt;subprocess.Popen&lt;/code&gt; could return.&lt;/p&gt;

&lt;h2&gt;
  
  
  The kernel did exactly what it's designed to do
&lt;/h2&gt;

&lt;p&gt;Here's the part I got wrong in my head for months. I assumed the OOM killer would target the memory hog — my agent — and I'd lose some jobs. Instead the Pi didn't cleanly OOM-kill anything for a long time. With a swapfile on the SD card, 400 browser processes pushed the machine into swap thrash: the kernel spent its time paging, not running, and every process slowed to a crawl, including the ones that were almost done.&lt;/p&gt;

&lt;p&gt;When it finally did start killing, it picked by &lt;code&gt;oom_score&lt;/code&gt;, which is roughly proportional to resident memory. My agent is a 60 MB Python process. Postgres was the largest single allocation on the box. So the kernel killed Postgres — repeatedly, as it got restarted by systemd — and left the 400 browsers alone, because killing one of them only freed 300 MB and the scoring is about the process, not the herd.&lt;/p&gt;

&lt;p&gt;The lesson generalizes further than my Pi: &lt;strong&gt;the OOM killer is not a referee. It's a scorer.&lt;/strong&gt; It has no idea which process is the problem. It knows which process is biggest.&lt;/p&gt;

&lt;h2&gt;
  
  
  The three things I got wrong
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. I put the limit in the wrong layer.&lt;/strong&gt; Application-level limits only cover the code paths you remember. My agent had four ways to start work — the cron poller, an HTTP endpoint, a retry sweeper, and a manual CLI I used for debugging. I patched the poller. Three entry points still bypassed it. A limit that lives in one function isn't a limit; it's a habit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. My monitoring measured the wrong thing.&lt;/strong&gt; My alert was a heartbeat: the agent wrote a timestamp to Redis every 30 seconds, and if the timestamp went stale I got a push notification. That check passed for six hours while the box was unusable, because the agent was never blocked — it was &lt;em&gt;thriving&lt;/em&gt;. It had 400 children and was having the best night of its life. Liveness monitoring answers "is the process running," which is almost never the question you actually have.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. My load test used three files.&lt;/strong&gt; Three documents, 400 KB each, once, on a machine with nothing else happening. It passed. It could not have failed. A load test that doesn't approach the real input volume tests your code's syntax, not its behavior.&lt;/p&gt;

&lt;h2&gt;
  
  
  What fixed it, in order of how much it mattered
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Hard limits at the OS level.&lt;/strong&gt; The fix that actually holds is one the application can't bypass because it isn't in the application. A systemd drop-in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;MemoryMax&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;2G&lt;/span&gt;
&lt;span class="py"&gt;MemoryHigh&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;1.5G&lt;/span&gt;
&lt;span class="py"&gt;TasksMax&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;64&lt;/span&gt;
&lt;span class="py"&gt;OOMPolicy&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;stop&lt;/span&gt;
&lt;span class="py"&gt;Restart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;on-failure&lt;/span&gt;
&lt;span class="py"&gt;RestartSec&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;10&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;TasksMax=64&lt;/code&gt; alone would have prevented the outage. Sixty-four tasks is roughly fifteen browsers plus overhead, and when the cap is hit, &lt;code&gt;Popen&lt;/code&gt; fails with &lt;code&gt;EAGAIN&lt;/code&gt; instead of eating the machine. My agent now gets an exception it can log instead of a corpse to discover.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Protect the bystanders.&lt;/strong&gt; Postgres gets a lower OOM score so the kernel reaches for something else first:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;OOMScoreAdjust&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;-800&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This isn't a guarantee — nothing at the kernel level is — but it changes which process dies first, and "the database dies last" is the property I actually wanted.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;A queue that can only be drained by N workers.&lt;/strong&gt; Instead of a list I iterate over, jobs are claimed inside a transaction, and the number of claimers is fixed by systemd template units:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# agent-worker@.service
&lt;/span&gt;&lt;span class="nn"&gt;[Service]&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="py"&gt;ExecStart&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/agent/worker.py&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;systemctl enable --now agent-worker@{1..4}&lt;/code&gt; gives me four workers. The fifth doesn't exist. There's no variable to forget to check, because the number of loops in the system is the number of unit files.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Alert on symptoms, not heartbeats.&lt;/strong&gt; The three numbers I actually watch now: queue depth (is it growing faster than it drains), per-worker RSS (is one job leaking), and time-in-&lt;code&gt;running&lt;/code&gt; per job (is something wedged). All three would have fired by 9:30 PM.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Turn off swap on the SD card, or use zram.&lt;/strong&gt; Thrash is worse than death. A clean OOM kill loses one job; thrashing loses six hours and possibly your SD card. &lt;code&gt;vm.swappiness=10&lt;/code&gt; plus zram gets you a machine that fails fast and loud.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cost, honestly
&lt;/h2&gt;

&lt;p&gt;Six hours of writes and 412 re-run jobs. Two customers got their documents twice because I hadn't made the pipeline idempotent — the output was written, then written again on retry, and the second pass appended instead of replacing. That's the bug that actually reached a human, and it had nothing to do with memory. It was waiting there the whole time for any retry to expose it.&lt;/p&gt;

&lt;p&gt;Total cost: about $40 in my own time refunded, one corrupted afternoon, and a rule I now apply to every service I run: &lt;strong&gt;if the failure mode is "the machine dies slowly," you have to bound it outside the thing that's dying.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The eleven good weeks weren't evidence the design was sound. They were evidence the input volume stayed low. That's the trap with automation you run yourself — nothing pages you when the only thing that changed is that it got popular.&lt;/p&gt;

&lt;p&gt;If you're running anything agent-shaped on a Pi, put &lt;code&gt;TasksMax&lt;/code&gt; and &lt;code&gt;MemoryMax&lt;/code&gt; on it tonight. It takes four minutes and it's the difference between an error in your logs and a machine you have to unplug.&lt;/p&gt;




&lt;p&gt;I write up the specific playbooks in &lt;a href="https://uln.lemonsqueezy.com/checkout/buy/0ce2371c-c75d-423c-b64d-685a00445048" rel="noopener noreferrer"&gt;The Solo Operator's AI Agent Playbook&lt;/a&gt; — code LAUNCH90 at checkout makes it $1.90. If it doesn't save you 5 hours in week one, reply to the receipt for a refund.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>automation</category>
      <category>raspberrypi</category>
      <category>programming</category>
    </item>
  </channel>
</rss>
