DEV Community

Cover image for A loop repaired a live HTTP service in 2m 30s. Nobody pressed start.
Alex
Alex

Posted on Originally published at github.com

A loop repaired a live HTTP service in 2m 30s. Nobody pressed start.

The self-heal loop — MOMUS finds, SKOPOS conducts, Factory writes a diff, node agent ships by digest, two gates

Short answer: yes, we have a real self-heal-with-redeploy loop. It is not “the AI merged to main and saved production overnight.” It is documented, gated, and deliberately boring about what it does not prove.

On 2026-08-29, 10:51:29 → 10:53:59, the loop re-healed a regression with nobody in it:

find → signed ticket → Factory diff → Gitea branch → node agent build → MOMUS gates the candidate → DeployOrder → live recreate → MOMUS gates live.

Wall clock: two minutes thirty seconds. The container id changed. The previous image was journalled. A fresh probe returned findings: 0.

The first end-to-end ship was two days earlier (2026-08-27, 5m 02s, human-dispatched). Same pipe. Same honesty.

Primary record: first-self-heal.md · Ops: self-healing-operations.md


Honesty first (read before the wow)

Claim people hear What is actually true
“The factory fixed production” It fixed momus-canary — a fixture that must violate its own contract so the pipe can be seen firing
“Fully autonomous” No human in the 2m 30s run — and still no self-merge to main. The branch is the review artifact
“Magic redeploy” A node agent claims a signed DeployOrder, builds from a named commit, promotes by digest, health-gates, journals the previous image
“Treasury paid real money” Not this story. Older find→verify→pay cycles used simulated settlement. Scanner key ≠ Treasury key

The finding was real (ordinary probes, real signatures). The target is a canary. That distinction is the product, not a footnote.

MOMUS live scanner console — English UI, crypto off, signed findings including canary
MOMUS: tolerated adversary. Signs findings. Cannot pay itself. Crypto off, fail-closed.


Who is on the wire

Role Component Job
Finder + deploy gate MOMUS Probe, sign finding / fix verdict. Retest candidate and live
Conductor SKOPOS remediation One job: Factory → git → BuildOrder → DeployOrder → rollback. Not the fleet dashboard
Patch author AI-Factory (private instance) Unified diff only — never an image. Route absent on the public factory
The hand SKOPOS node agent Polls. Never opens inbound. Builds/deploys only allowlisted services
The patient momus-canary Breaks on purpose until the loop heals it

SKOPOS the name is also a traffic globe. In this loop it is the conductor. The hand that rebuilds the box is the node agent.

SKOPOS landing — English UI, EN language selected
Watcher and conductor share a name. They are not the same deployment.

Factory live HUD — EN
Public factory never mounts remediation. Private instance writes under a ticket.


The bug was ordinary

The canary declared a free-tier ceiling of 100, then served unpaid n=500 with 200. Finding mom-31eb7bc4971644ba (free_tier_ceiling_bypass).

Before the heal — n=500 returned 200
Ceiling said 100. Response was 200. Should have refused.

After the heal:

After the heal — n=500 → 402, n=5 → 200
Ceiling holds. Normal use still served.


Nine lines. Not an image.

The Factory authored a unified diff on momus/canary/canary.pynine lines in, eight out. It removed the conditional bypass, not the probe’s input. That is the instruction the route gives the model: fix the root cause; a change that only makes the probe pass is worse than no patch.

Factory patch — nine lines in, eight out
Never an image. Diff only. The public factory does not mount this route.


Two gates, two subjects

A gate that examines the box you just replaced is a ceremony.

  • gated=candidate — MOMUS probes the build, before promotion
  • gated=live — MOMUS probes the running box, after recreate

fixed=true is not a vibe. It names which box.

Two gates, two subjects — gated=candidate then gated=live

The hand promotes by digest, then verifies the running container is that digest. A recreate of the old image used to count as deploy. It does not anymore.

Container identity changed — previous image journalled
Before 5bdeae2bf93c / after 0009b9ae5e77. Undo is a journalled digest, not a prayer.


2m 30s, nobody pressed start

The 27 Aug run was dispatched by a human. This one was not: a scanner on a timer found the defect, a policy decided it was worth fixing, and the ticket opened without anybody asking.

The canary had been rebuilt from unpatched source — a genuine regression against a remediation that had already shipped.

time what
before container 5bdeae2bf93c, image 73205c15575a
10:51:29 Factory asked for a patch — job re-opened as a regression
10:52:11 patch on momus/fix-mom-31eb7bc4971644ba-1
10:52:15 node agent builds commit 64a05d389ee7
10:53:17 MOMUS gates candidate — fixed · DeployOrder signed
10:53:59 agent deploys · MOMUS retests live — done
after container 0009b9ae5e77, image c1e3e12a121b

Verified outside the loop’s own done: container id changed, created during the run, previous_image = the broken build, probe findings: 0.


What dry-run never caught

Enabling the live path surfaced seven defects. Shared shape: a guard that is written but never exercised reads exactly like a guard that works.

Among them: deploy recreated the old image; DeployOrder.image was unread; the gate ran before the agent had moved; there was no rollback; a dry-run DONE swallowed the first live ticket.

Containment now lives on the agent’s host, not on the order’s say-so: allowlist (canary, not MOMUS, not Treasury), branch prefix momus/fix-…, promote only images this agent built, refuse if the verdict examined live instead of candidate.

Findings against the security core never take this path. An auditor that patches itself is the failure mode.


What this does not prove

  • One component, one finding, one probe. The patient is a fixture. Nobody depends on it.
  • A fixed verdict proves the finding stopped reproducing. It does not prove the patch is good, does not read the diff for a backdoor, and cannot notice breakage the probe never tested. That is why the branch exists and why merging stays human.
  • The gate re-runs the named probe. Everything outside it is unexamined. Rollback exists for that day.

Soft star / live

Soft ⭐ Why
github.com/alexar76/momus Finder + gate + canary + the checkable records
github.com/alexar76/skopos Conductor + node agent + fleet watch
github.com/alexar76/aicom Factory remediation instance lives in the monorepo

Live: momus.modelmarket.dev · skopos.modelmarket.dev

Season 47 of the ecosystem series walks the same loop: The canary was live.


We did not ship “agents that fix the company while you sleep.”

We shipped a loop that can refuse: separate keys, candidate before live, digest before recreate, allowlist before build, branch before merge, human before main.

The canary is still a canary. The two minutes thirty are still two minutes thirty.

Verify, don’t trust — starting with first-self-heal.md.

Top comments (0)