<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: dav123</title>
    <description>The latest articles on DEV Community by dav123 (@david_sevin_6e1630a985613).</description>
    <link>https://dev.to/david_sevin_6e1630a985613</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4214713%2Fbc28fabc-e26f-4b9b-98e0-0a441f40764a.jpg</url>
      <title>DEV Community: dav123</title>
      <link>https://dev.to/david_sevin_6e1630a985613</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/david_sevin_6e1630a985613"/>
    <language>en</language>
    <item>
      <title>I wanted to know how they got GTA V into a browser. I ended up with an arcade and a monkey in a tutu</title>
      <dc:creator>dav123</dc:creator>
      <pubDate>Fri, 30 Oct 2026 16:58:47 +0000</pubDate>
      <link>https://dev.to/david_sevin_6e1630a985613/i-wanted-to-know-how-they-got-gta-v-into-a-browser-i-ended-up-with-an-arcade-and-a-monkey-in-a-tutu-5cpl</link>
      <guid>https://dev.to/david_sevin_6e1630a985613/i-wanted-to-know-how-they-got-gta-v-into-a-browser-i-ended-up-with-an-arcade-and-a-monkey-in-a-tutu-5cpl</guid>
      <description>&lt;p&gt;Earlier this month, videos started going around of GTA V running in a web browser. The site went offline shortly after. GTA V came out in 2013 and is still the latest GTA you can actually play, since GTA VI isn't out until November. And the way I understood the story, the people behind it had got there with the help of an LLM.&lt;/p&gt;

&lt;p&gt;I wanted to understand how. So I did what any reasonable person does when faced with a huge open-world game ported to the web: I asked an AI to make me a bubble shooter.&lt;/p&gt;

&lt;p&gt;This is the story of what happened next. Spoiler: I still don't know how they did GTA.&lt;/p&gt;

&lt;h2&gt;
  
  
  A bubble shooter and some postage stamps
&lt;/h2&gt;

&lt;p&gt;I had two coding agents to try out. Codex would write the code. Claude would orchestrate and, above all, refuse to believe Codex when it said "all green". I've written about that review loop &lt;a href="https://medium.com/p/96e75583d88e" rel="noopener noreferrer"&gt;elsewhere&lt;/a&gt;, so I won't repeat it here. The short version: the ticket pinned the stack (strict TypeScript, Phaser 3, Vite, Vitest, Playwright), and each game engine had to be pure, seeded, and testable without a browser.&lt;/p&gt;

&lt;p&gt;Less than an hour after the first ticket, the bubble shooter worked. My first request (quotes from my tickets are translated from French) was the obvious one: "let me play it on my phone". By the evening I had a review, and it wasn't glowing:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"the phone display needs work, it feels like playing on postage stamps"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;So the canvas got the whole screen and everything else moved into a ☰ menu, with the touch controls in a single row under your thumbs.&lt;/p&gt;

&lt;p&gt;Then I got greedy. I asked for a game picker on the home screen and more arcade games, 30 levels each, and I listed my childhood: Tetris, Arkanoid, the snake from the Nokia 3210, Metal Slug, Galaga, Space Invaders, Pac-Man. What came out were Serpentin, Briqueterie, Empilo, Orbite, Essaim, Dédale and Ferraille. Original names, original levels, everything drawn in code, and no brand anywhere in the code. The ticket was blunt about why: some rights holders go after clones, even free ones.&lt;/p&gt;

&lt;p&gt;The next morning I said "go for Opencade, state of the art", and it went public: MIT license, a CI that runs lint, unit tests, build and end-to-end tests, and a demo on GitHub Pages.&lt;/p&gt;

&lt;h2&gt;
  
  
  The backlog that wouldn't stop growing
&lt;/h2&gt;

&lt;p&gt;Every time I played, I came back with a ticket.&lt;/p&gt;

&lt;p&gt;Between the first bubble shooter ticket and the last card I opened, two and a half days went by. My backlog gained 40 numbers, 33 of them about games, and the game tickets all started the same way: I'd just played something.&lt;/p&gt;

&lt;p&gt;"Put a picture on the home page for each game so you can see what it is." That one became animated thumbnails, rendered by each game's own engine. A lunar lander, a crate-pushing puzzle, a road-and-river crossing game and a kart racer all landed on the same day. Then, one evening: "Among the games I really liked, there was Zuma." Ticket. "I really loved Lemmings." Ticket. "I really loved Worms." Ticket. "And Boulder Dash." Ticket, one minute later.&lt;/p&gt;

&lt;p&gt;Five releases, from 0.1.0 to 0.3.2. I'm not proud. I'm a little proud.&lt;/p&gt;

&lt;h2&gt;
  
  
  The kart that couldn't be driven
&lt;/h2&gt;

&lt;p&gt;Dérapage is a top-down kart racer. My first review listed four things: the controls weren't practical on a phone, the kart should be much slower on grass, the track needed boost strips, and there should be trees. My second review was shorter: "in Dérapage, it's always the same circuit."&lt;/p&gt;

&lt;p&gt;So I asked for tracks inspired by Formula 1: a street circuit in the spirit of Monaco, one that loops around Les Invalides in Paris, a fast, hilly one through the forest like Spa. Thirty tracks, chequered finish lines, a lap timer and best lap. Version 0.3.1 shipped with 527 green tests, 299 unit and 228 end-to-end.&lt;/p&gt;

&lt;p&gt;Before six the next morning, I sent my verdict as a player: "Dérapage is unplayable."&lt;/p&gt;

&lt;p&gt;Claude didn't argue. It wrote a simulated thumb that reads the game state every 150 ms and steers by its heading error. That's a driver with perfect information, a much better one than me. On the first circuit, in 40 seconds, it reached gate 2, hit things 121 times, spent 32% of its time off the track, and sat at a standstill for 23 samples, stuck against the rail near the start line.&lt;/p&gt;

&lt;p&gt;So it wasn't just me. The causes were design choices: a tight street circuit as level one, steering relative to a camera that rotates with the kart, and rails that stuck to you. Version 0.3.2 switched to an absolute 360° joystick with a fixed camera, north always up, rails that push you back onto the track, and a wide, easy first circuit. Same simulated thumb, this time with ±0.35 rad of random noise on every gesture to simulate a clumsy one:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Circuit&lt;/th&gt;
&lt;th&gt;Gate reached&lt;/th&gt;
&lt;th&gt;Standstills&lt;/th&gt;
&lt;th&gt;Off track&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;9, lap 2 at 24 s&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;4%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The old first circuit, now fifth&lt;/td&gt;
&lt;td&gt;11, lap 2 at 25 s&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;I'd love to end the kart story there. I can't: a later run of the same clumsy thumb got stuck on lap 1 of the first circuit after a minute and a half, so the kart isn't officially declared playable again yet. 527 green tests, and unplayable. No test tells you a game isn't fun. You need someone to play it, even badly.&lt;/p&gt;

&lt;h2&gt;
  
  
  "Ferraille is super repetitive"
&lt;/h2&gt;

&lt;p&gt;Ferraille is the run-and-gun one. After a few levels, I wrote: "Ferraille is super repetitive."&lt;/p&gt;

&lt;p&gt;Again, Claude didn't argue, it measured. It built a repetitiveness score from 0 to 100 for all twelve games. A "novelty" is a game element that appears for the first time (changing a number doesn't count), a "clone" is a level too close to the previous one, and a "plateau" is a run of levels with nothing new:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;repetitiveness = 100 × [ 0.50 × share of clones
                       + 0.25 × (1 − novelties / (n − 1))
                       + 0.25 × longest stretch without novelty / (n − 1) ]
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ferraille scored 84.5. Four new things, all before level 7. After that, only numbers changed. And it wasn't even the worst: the maze game scored a perfect 100.&lt;/p&gt;

&lt;p&gt;Ferraille has since gained six enemy behaviours, three weapons, wagons, hostages, guardians and a three-phase reactor. It's oddly satisfying to see your thumbs proven right in a table.&lt;/p&gt;

&lt;h2&gt;
  
  
  My pencils, then ChatGPT, then Codex
&lt;/h2&gt;

&lt;p&gt;Next I wanted recurring characters across all the games: a panda, a red panda, a dragon, a monkey in a tutu and a unicorn. They're called Boubou, CaliRoux, Pestouille, Tutu and Lumi. The rule I set: they're redrawn in code, vector style, and no raster image goes into the public repo.&lt;/p&gt;

&lt;p&gt;I also made a generous offer: "if you need graphics or visuals, put the request in a dedicated card, I'll get my pencils out."&lt;/p&gt;

&lt;p&gt;The card exists. It has received zero requests so far. In the end I sent eight reference sheets anyway, without being asked. In my own words: "I made the sheets with ChatGPT, starting from hand drawings." So the characters went from my pencils to ChatGPT, and from ChatGPT to Codex, which redraws them in TypeScript. One more AI in the chain. I'm starting to lose count.&lt;/p&gt;

&lt;p&gt;My favourite design debate was about Tutu. There's a chain-shooter game on the way, the genre where a line of coloured balls rolls towards a hole and you fire balls into it. My first idea: the chain is the monkey's tail. Then I hesitated, and the alternative that came back made a fair point: popping beads is a lot less strange than popping bits of tail. Final decision: Tutu dances along the track dragging a necklace of coloured beads, the beads pop, the necklace pulls itself back together, and as a bonus the red panda grabs the necklace and pulls it backwards. That game is still being written as I type.&lt;/p&gt;

&lt;h2&gt;
  
  
  Built for airplane mode
&lt;/h2&gt;

&lt;p&gt;I wanted to play with no connection at all. Version 0.2.0 turned Opencade into an installable web app. A service worker precaches the whole build, Phaser included, so there's nothing left to fetch once it's installed. Saves live in &lt;code&gt;localStorage&lt;/code&gt;, so they work offline too. I still owe it the real test, phone in airplane mode; so far only the end-to-end test has done that.&lt;/p&gt;

&lt;p&gt;One morning I was convinced I was playing an old version. Maybe I was: service workers are very good at keeping things. Version 0.3.2 now shows the version and build date at the bottom of the home screen, checks for updates on launch, every 30 minutes and when the app comes back to the foreground, and has a "Force update" button in the menu. I'll admit the button exists mainly for me.&lt;/p&gt;

&lt;h2&gt;
  
  
  "No ads, no trackers (I hope that's true)"
&lt;/h2&gt;

&lt;p&gt;When I asked for this article, the sentence I wrote was: I had a great time on my phone, with no ads and no trackers, and I hope that's true. I never wrote the code. My ticket said "no requests to third parties", which is not the same thing as checking.&lt;/p&gt;

&lt;p&gt;So I had Claude check.&lt;/p&gt;

&lt;p&gt;First, the code. In the public repository's &lt;code&gt;src/&lt;/code&gt;, there is exactly one &lt;code&gt;fetch&lt;/code&gt;: the update check, which asks the app's own &lt;code&gt;version.json&lt;/code&gt;. No XHR of its own, no &lt;code&gt;sendBeacon&lt;/code&gt;, no WebSocket, no cookie. The only production dependency is Phaser, which brings in one tiny event library and nothing else. The CSS uses &lt;code&gt;system-ui&lt;/code&gt;, so there's no web font to download either.&lt;/p&gt;

&lt;p&gt;Second, what GitHub Pages actually serves. The built bundle mentions two outside URLs: &lt;code&gt;phaser.io&lt;/code&gt;, in Phaser's default config and its console banner, and a &lt;code&gt;bit.ly&lt;/code&gt; link inside a Workbox warning message. Neither is ever requested.&lt;/p&gt;

&lt;p&gt;Third, a real session. In a Playwright container on a small VPS, on a simulated phone, Claude recorded every request the browser made, service worker included, while playing three games with taps, then reloading offline:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;on&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;request&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="nx"&gt;reqs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;url&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="na"&gt;sw&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="o"&gt;!!&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;serviceWorker&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="p"&gt;}));&lt;/span&gt;
&lt;span class="c1"&gt;// ... play Dérapage, Alunir, Orbite, then ctx.setOffline(true) and reload&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;third&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;reqs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;new&lt;/span&gt; &lt;span class="nc"&gt;URL&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;url&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nx"&gt;origin&lt;/span&gt; &lt;span class="o"&gt;!==&lt;/span&gt; &lt;span class="nx"&gt;origin&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The result: 30 requests, all 30 to the Pages origin, 11 distinct files, zero to anyone else. Offline, the twelve games were still there. In storage, a single &lt;code&gt;localStorage&lt;/code&gt; entry, the lunar lander save: &lt;code&gt;{"unlocked":1,"best":0}&lt;/code&gt;. No cookies, no IndexedDB, one cache, the app's own.&lt;/p&gt;

&lt;p&gt;So, no ads and no trackers in Opencade. Two honest caveats. GitHub itself says that when a Pages site is visited, "the visitor's IP address is logged and stored for security purposes", signed in or not, and I can't see or change that. And the page has no Content-Security-Policy, so nothing in the browser would stop a future commit from adding a third-party call. Only an audit like this one would catch it. I checked Chromium and three games, not every browser.&lt;/p&gt;

&lt;h2&gt;
  
  
  So, how did they do GTA?
&lt;/h2&gt;

&lt;p&gt;I still don't know. I started out wanting to understand how people got a huge open-world game running in a browser tab, and I ended up choosing between a monkey's tail and a bead necklace. And typing "unplayable" with one thumb before six in the morning turned out to be a perfectly valid bug report.&lt;/p&gt;

&lt;p&gt;If you know how they actually got GTA V into a browser, tell me in the comments. I clearly need a hint before I start asking for a pseudo-3D kart view. Which, by the way, is already in the backlog.&lt;/p&gt;

&lt;p&gt;You can play Opencade in your browser, or install it on your phone: &lt;a href="https://davy45270.github.io/Opencade/" rel="noopener noreferrer"&gt;the demo&lt;/a&gt;. The code is on &lt;a href="https://github.com/Davy45270/Opencade" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>gamedev</category>
      <category>ai</category>
      <category>webdev</category>
      <category>javascript</category>
    </item>
    <item>
      <title>My self-healing homelab saved 17 seconds. Here's why I kept it</title>
      <dc:creator>dav123</dc:creator>
      <pubDate>Fri, 30 Oct 2026 13:38:35 +0000</pubDate>
      <link>https://dev.to/david_sevin_6e1630a985613/my-self-healing-homelab-saved-17-seconds-heres-why-i-kept-it-45m8</link>
      <guid>https://dev.to/david_sevin_6e1630a985613/my-self-healing-homelab-saved-17-seconds-heres-why-i-kept-it-45m8</guid>
      <description>&lt;p&gt;A power cut rebooted my home servers one June morning, and the reverse proxy didn't come back. Every site I run goes through it, and I only found out that evening. The alert had fired on time, but it reached my phone through a public URL, and that URL went through the proxy that was down.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt; I built an auto-remediation chain for my homelab: Prometheus → Alertmanager → n8n → a policy gate → an executor that can only run actions from a closed list. Restarting a service is one line. The real work was teaching the bot when to refuse. Then I timed it against a human, and it won by 17 seconds.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;A watchdog now restarts the proxy if it dies. It can't see a frozen one: a paused container stays "running" with its port open while no traffic gets through. That quiet kind of failure is the one worth automating.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a ten-line restart script is dangerous
&lt;/h2&gt;

&lt;p&gt;The obvious fix is a script that restarts a service when it stops responding. Then the NAS goes down, and it restarts the thirty services that depend on it, in a loop, all night. It can't fix anything because the cause is elsewhere, and by morning one diagnosable failure is buried under a pile of restarts. A remediation bot that can't say no turns small outages into big ones. Everything below is about keeping the bot from acting when it shouldn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  The architecture: four roles, strictly separated
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Prometheus ─► Alertmanager ─► n8n ─────────► hsd/gate ─────────► rb-exec
 detects       routes          filters,        decides: act or     runs ONE action
                               asks one        refuse, where,      from a closed
                               question        with which action   per-host catalogue
                                                    │
                                                    └─► re-checks recovery in Prometheus
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://n8n.io/" rel="noopener noreferrer"&gt;n8n&lt;/a&gt; runs on its own small VM and does very little on purpose. Alertmanager pushes it an alert, not a command. n8n drops anything no longer current (a grouped webhook payload also carries alerts that just resolved), asks the gate "here's the alert and its labels, what do we do?", and logs the answer.&lt;/p&gt;

&lt;p&gt;Handing n8n the executor's credentials would be the classic &lt;a href="https://en.wikipedia.org/wiki/Confused_deputy_problem" rel="noopener noreferrer"&gt;confused deputy&lt;/a&gt;. So the secrets vault gives n8n its own credential and the one for calling the gate, and refuses it the executor's. Only the gate gets that one. Someone who takes over n8n can ask for a restart, not run one.&lt;/p&gt;

&lt;p&gt;The gate answers with one of nine verdicts. One means "act". The other eight are refusals: frozen by a deeper cause, circuit breaker open, unknown alert, ambiguous match, host not resolved, not armable, undecidable (Prometheus is unreachable, so it can't tell whether something bigger is going on), and maintenance. Every refusal pages me except the two that are expected: frozen and maintenance.&lt;/p&gt;

&lt;h2&gt;
  
  
  What a catalogue entry looks like
&lt;/h2&gt;

&lt;p&gt;Here's the first service I armed, my ebook library (Calibre-Web), lightly sanitized. The executor's catalogue declares the action and the only container it may touch:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;restart-container&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;args&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;enum&lt;/span&gt;
      &lt;span class="na"&gt;values&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;calibre-web&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
  &lt;span class="c1"&gt;# "Is it ALREADY in the desired state?" Exit 0 means nothing to do (noop).&lt;/span&gt;
  &lt;span class="na"&gt;precondition&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;test&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s"&gt;$(docker&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;inspect&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;-f&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;'{{.State.Health.Status}}'&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{name}&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;2&amp;gt;/dev/null)&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;=&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;healthy"&lt;/span&gt;
  &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;docker&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;restart&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{name}"&lt;/span&gt;
  &lt;span class="na"&gt;proof&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;test&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="s"&gt;$(docker&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;inspect&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;-f&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;'{{.State.Health.Status}}'&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{name}&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;2&amp;gt;/dev/null)&lt;/span&gt;&lt;span class="se"&gt;\"&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;=&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;healthy"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the gate side, the service's runbook maps the alert to that action and gives the PromQL proof of recovery:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;### S1 — UI frozen while the container is Up&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; alert: CalibreWebDown
&lt;span class="p"&gt;-&lt;/span&gt; proof: &lt;span class="sb"&gt;`probe_success{instance="&amp;lt;calibre-login-url&amp;gt;"} == 1`&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; fix_ref: F1   # F1: rb-exec restart-container name=calibre-web
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The gate re-checks that proof for up to 120 seconds after the action. If it doesn't hold by then, the attempt is logged as a failure and I'm paged. The entry can run at most three times an hour, and two failures in a row disarm it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Guardrail 1: don't treat the symptom of a deeper cause
&lt;/h2&gt;

&lt;p&gt;When the NAS goes down, five services alert at once and none of them is broken. So the gate keeps a cause hierarchy: NAS down freezes the five services that depend on it, a server down freezes everything on it, the internet box down freezes everything. Frozen means the alert is real, the gate knows it, and it doesn't act, because the alert for the real cause is already firing.&lt;/p&gt;

&lt;p&gt;One entry makes me smile: if the machine hosting the gate goes down, everything freezes, because the gate's own view of the world can't be trusted anymore.&lt;/p&gt;

&lt;h2&gt;
  
  
  Guardrail 2: a circuit breaker that survives restarts
&lt;/h2&gt;

&lt;p&gt;A wrong bot is wrong fast, so it has hard ceilings:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Limit&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;th&gt;Prevents&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Actions per service per hour&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;A service going round in circles&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Actions across the fleet per hour&lt;/td&gt;
&lt;td&gt;10&lt;/td&gt;
&lt;td&gt;A fleet-wide runaway&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;24 h failure rate that disarms everything&lt;/td&gt;
&lt;td&gt;30%&lt;/td&gt;
&lt;td&gt;A bot that has started getting things wrong&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consecutive failures that disarm an entry&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Retrying all night&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Actions per day on a critical component&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Trial and error on, say, the proxy&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The counters live in SQLite on disk: in memory they'd reset right after the crash a runaway remediation caused. There's no reset button either. A &lt;code&gt;noop&lt;/code&gt; ("nothing to do") counts toward the hourly budgets but never toward the failure rate, so 38 noops and 2 failures is two failures out of two, not 5%.&lt;/p&gt;

&lt;p&gt;The breaker tripped on its own on day one, after four attempts in the same hour. A better test than any I'd have written.&lt;/p&gt;

&lt;h2&gt;
  
  
  Guardrail 3: verify recovery, don't trust the exit code
&lt;/h2&gt;

&lt;p&gt;"Restart done" only means exit code 0, so the gate asks Prometheus whether the service's probe answers again. The proxy needs two proofs: its admin endpoint, and a public site served through it with its real hostname and certificate.&lt;/p&gt;

&lt;p&gt;Subtler: restarting a paused container unpauses it first, and the old process answers for a few seconds while shutting down. In four of my first eight real failures, the gate closed on a sample taken &lt;em&gt;before&lt;/em&gt; the restart finished. Now every sample a proof reads must postdate the action.&lt;/p&gt;

&lt;p&gt;And watch the wording. "Is the symptom present?" and "has the service recovered?" sound alike, but one is true when things are broken and the other when they're fine. Swap them and the chain reports success no matter what.&lt;/p&gt;

&lt;h2&gt;
  
  
  The canary: proving the chain is still alive
&lt;/h2&gt;

&lt;p&gt;How do you know a safety mechanism works when it almost never fires? One metric holds the timestamp of the last successful end-to-end run (&lt;code&gt;remediation_canary_dernier_passage&lt;/code&gt;; &lt;em&gt;dernier passage&lt;/em&gt; is French for "last run"). Past 20 hours, a "wake up" alert goes into the chain itself, which runs a harmless &lt;code&gt;canary-touch&lt;/code&gt; action that refreshes the timestamp, and the alert resolves. The chain proves it works by clearing the alert that called it. Past 26 hours, a second alert goes to my phone.&lt;/p&gt;

&lt;p&gt;My first version of that second rule had a hole: "older than 26 h" never fires if the series disappears, because comparing against a missing series returns nothing rather than false. The real rule:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;RemediationCanaryStale&lt;/span&gt;
  &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;&amp;gt;-&lt;/span&gt;
    &lt;span class="s"&gt;(time() - remediation_canary_dernier_passage &amp;gt; 93600)&lt;/span&gt;
    &lt;span class="s"&gt;or absent_over_time(remediation_canary_dernier_passage[2h])&lt;/span&gt;
  &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;15m&lt;/span&gt;
  &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;critical&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  Breaking it for real
&lt;/h2&gt;

&lt;p&gt;I started with the ebook library, where downtime costs nothing and which had once frozen unnoticed for twenty-five minutes. I reproduced that (process paused, container up) and touched nothing.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Freeze induced&lt;/td&gt;
&lt;td&gt;14:06:34&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Alert fires (after its 3 min &lt;code&gt;for:&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;14:09:49&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service answers again&lt;/td&gt;
&lt;td&gt;14:10:22&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Alert to recovery&lt;/td&gt;
&lt;td&gt;33 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Total outage&lt;/td&gt;
&lt;td&gt;3 min 48 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Look at the precondition above: the executor won't act while the container reports healthy, and right after the freeze it still does. Its healthcheck takes 95 seconds to flag the problem, the alert 180 to fire. That's 85 seconds of margin, luckily in the right direction. Reversed, the chain would have found a frozen service claiming to be healthy and answered "nothing to do". I now time that gap for every new service.&lt;/p&gt;

&lt;h2&gt;
  
  
  Seventeen seconds
&lt;/h2&gt;

&lt;p&gt;How long would a human have taken? Eight minutes earlier I'd replayed the same failure with nothing armed and an operator at the screen, reacting within a second. That's the best case for a human, not a realistic one. It's also a single run on each side, so read it as an order of magnitude, not a benchmark.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Alerted human&lt;/th&gt;
&lt;th&gt;Automated chain&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Alert to recovery&lt;/td&gt;
&lt;td&gt;50 s&lt;/td&gt;
&lt;td&gt;33 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Total outage&lt;/td&gt;
&lt;td&gt;4 min 13 s&lt;/td&gt;
&lt;td&gt;3 min 48 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Seventeen seconds. I'd hoped for much more. But my long outages came from nobody knowing the service was down: in June because the alert never reached me, for the library because there was no probe at all. Most of the gain arrived the day I added the probe and the alert. The value was in the smoke detector, not the fire truck.&lt;/p&gt;

&lt;p&gt;The argument for the chain is different, then. Those 33 seconds are the same at 3 a.m. on a Sunday or while I'm on holiday, so the fair comparison is against a human who isn't there, and I can't measure that without a real night-time outage. Every repair also leaves a record of when, which action and which proof.&lt;/p&gt;

&lt;h2&gt;
  
  
  Traps I nearly shipped
&lt;/h2&gt;

&lt;p&gt;Some mechanisms only looked alive. The monitoring registration tool printed &lt;code&gt;registered n8n (6 VMs total)&lt;/code&gt; into a directory Prometheus no longer read, and n8n's health endpoint answers before its workflows are active, so a deployment check passed while nothing listened.&lt;/p&gt;

&lt;p&gt;Planned stops were another gap. When I stop a service to upgrade it, the chain would restart it three minutes later, maybe mid-migration. Docker doesn't expose whether a stop was intended, so I declare it. A maintenance mode covers a whole host, carries an end time rather than a switch I'd forget to flip back, and holds even if I forgot to silence the alert. If its file becomes unreadable, maintenance is treated as active, for at most 24 hours.&lt;/p&gt;

&lt;p&gt;The password vault (Vaultwarden) taught me the most. First, something was already repairing it: an older tool restarts any container that reports unhealthy, after three and a half minutes, while the vault's alert waits ten. The new chain would have arrived six and a half minutes late with nothing to do. "What already repairs this?" is now a step in the procedure.&lt;/p&gt;

&lt;p&gt;Second, the generic precondition read the container's internal healthcheck, while the alert's probe hits the vault from outside. When those disagree is exactly when a restart would help, and the precondition would have said "nothing to do". The vault's precondition is now the same HTTP request as its proof. My check for any new entry: break what the proof measures, and confirm the precondition turns false.&lt;/p&gt;

&lt;p&gt;Third, "it answers" isn't "it works". A vault with a corrupted database still answers and can't decrypt a single password. An endpoint returning an account's key-derivation parameters looked like a good proof, until a non-existent address got the same response, by design, so accounts can't be enumerated. The real proof is a full end-to-end login. Building it showed that nothing had been watching this at all. That new detection may be the biggest win of the project.&lt;/p&gt;

&lt;h2&gt;
  
  
  The front door, frozen on purpose
&lt;/h2&gt;

&lt;p&gt;Last came the frozen proxy, where a botched restart takes every site down. The watchdog handles a dead proxy; the chain covers a frozen one. The executor got two reasons to refuse. It validates the config in a throwaway container with no network first, because restarting on an invalid file would leave everything down. Sanitized:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker run &lt;span class="nt"&gt;--rm&lt;/span&gt; &lt;span class="nt"&gt;--network&lt;/span&gt; none &lt;span class="nt"&gt;--entrypoint&lt;/span&gt; caddy &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$CADDYFILE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;:/etc/caddy/Caddyfile:ro &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$IMAGE&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  adapt &lt;span class="nt"&gt;--config&lt;/span&gt; /etc/caddy/Caddyfile &lt;span class="nt"&gt;--adapter&lt;/span&gt; caddyfile &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null 2&amp;gt;&amp;amp;1 &lt;span class="se"&gt;\&lt;/span&gt;
  &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="o"&gt;{&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s1"&gt;'invalid Caddyfile: restart refused'&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;exit &lt;/span&gt;3&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It also refuses if a network address the proxy binds to has disappeared, since Docker would then fail to start it. And it gets one action per day.&lt;/p&gt;

&lt;p&gt;I froze the proxy for real one late morning while polling six public sites about every five seconds.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Proxy frozen&lt;/td&gt;
&lt;td&gt;11:55:17&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Alert fires (after its 2 min &lt;code&gt;for:&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;around 11:57:45&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Proxy restarted by the chain&lt;/td&gt;
&lt;td&gt;11:58:07&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;All six sites answer again, usual status codes&lt;/td&gt;
&lt;td&gt;11:58:11&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Outage seen from outside&lt;/td&gt;
&lt;td&gt;about 2 min 50 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In June the front door stayed shut until evening. This time it was under three minutes, almost all of it the alert's confirmation delay. The restart itself took about ten seconds.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where it stands
&lt;/h2&gt;

&lt;p&gt;About twenty services are armed now, from the canary to the proxy, including the vault, the wiki and the code forge. Each went through a full acceptance run with an induced failure first.&lt;/p&gt;

&lt;p&gt;I expected to fix things much faster, and the stopwatch said otherwise. What I got is more modest: things get fixed even when I'm not around.&lt;/p&gt;

&lt;p&gt;Do you let automation restart things in your homelab? Where do you draw the line, and how do you stop a dead NAS from turning into thirty pointless restarts? I'd like to hear it in the comments.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Prefer the story without the config? There's a shorter non-technical version on Medium: &lt;a href="https://medium.com/@davidsevin/i-taught-my-house-to-fix-itself-the-hard-part-was-teaching-it-when-not-to-3727fa0bc491" rel="noopener noreferrer"&gt;I taught my house to fix itself. The hard part was teaching it when not to.&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>homelab</category>
      <category>selfhosted</category>
      <category>devops</category>
      <category>automation</category>
    </item>
  </channel>
</rss>
