DEV Community: Jeremy Longshore

Do Not Blindly Restart: Designing a Self-Healing Watchdog That Stays Honest

Jeremy Longshore — Wed, 22 Jul 2026 10:28:31 +0000

A self-healing system that restarts blindly is an outage generator with extra steps.

On 2026-07-20 a burn-in watchdog guarding the SigNoz observability stack hard-latched and paged. The stack was fine. The breach was a phantom. That failure exposed two design defects that show up in almost every monitoring system built in a hurry: a watchdog that restarts blindly, and a monitor that lies about its own success. Both come back to the same rule: infrastructure that watches for failure has to be more honest about failure than the thing it watches.

The problem: a phantom latch

The burn-in watchdog exists to prove a freshly deployed container is stable before promotion. It polls pressure-stall (PSI) metrics for memory and io, and if a resource crosses a ceiling it stops the burn-in, latches, and pages a human. That last part matters: a latch is a hard stop. Nothing recovers on its own after a latch. Someone gets woken up.

The 2026-07-20 page fired on io PSI. Root cause: a ClickHouse background-merge io burst, sub-10-seconds long, against a baseline io of roughly 0.00. The check that caught it looked like this:

# The bad check: single sample, no duration, hard ceiling
psi_io=$(read_psi_io_avg10)
if awk "BEGIN{exit !($psi_io >= 1.0)}"; then
  latch "io PSI breach: ${psi_io}%"   # hard stop + page
fi

One sample. One threshold. No sense of time. A transient merge that lasted less than ten seconds tripped a hard stop on a stack that was never in trouble. The watchdog was not measuring saturation. It was measuring a spike and calling it saturation.

Why not just raise the threshold

The obvious fix is to bump 1.0 to something higher and move on. That is the wrong fix, and it is worth being precise about why.

A blanket raise trades one failure mode for a worse one. Raise the io ceiling high enough that a merge burst no longer trips it, and you have also raised it past the point where a real, sustained io saturation would trip it. You do not get fewer false pages. You get false pages replaced by missed real incidents, which is the failure mode a watchdog exists to prevent. Raising the number makes the monitor quieter, not more correct. A quiet monitor that misses saturation is worse than a noisy one, because you trust it.

The actual problem was never the height of the ceiling. It was that the check had no concept of duration. A spike and a sustained load produce the same instantaneous PSI reading. The only thing that separates them is time. So the fix is time, applied per resource:

# Duration-tolerant io check: a spike is logged and ignored,
# sustained pressure confirms and trips.
psi_io_avg10=$(read_psi_io_avg10)
psi_io_avg60=$(read_psi_io_avg60)
if awk "BEGIN{exit !($psi_io_avg10 >= 15.0)}" \
   && awk "BEGIN{exit !($psi_io_avg60 >= 5.0)}"; then
  breach "io PSI sustained: avg10=${psi_io_avg10}% avg60=${psi_io_avg60}%"
else
  log "io PSI transient ignored: avg10=${psi_io_avg10}%"
fi

The io ceiling now requires avg10 >= 15% AND avg60 >= 5%. Yes, the avg10 number itself moved from 1.0 up to 15, and that is not the blanket raise the last section warned against. The 1.0% single-sample line was never a saturation boundary in the first place, it sat down in the noise floor where a routine merge lives, so it was destined to fire on nothing. Raising it alone would still miss a real event. The avg60 term is what makes the higher ceiling honest instead of just quieter: it is the confirmation window. A sub-10-second merge cannot move a 60-second average past 5%, so it gets logged and ignored. A genuine io saturation holds both windows and trips. The fix was never a bigger number, it was a bigger number that a spike cannot reach.

Memory PSI kept its single-sample avg10 < 1.0% rule on purpose. Memory baseline is 0.00 and a memory floor breach is the kind of thing you want to latch fast, so the fast latch there is deliberate, filed as an advisory item rather than "fixed." That is the point of per-resource ceilings: io needs duration tolerance, memory does not, and a blanket policy would get one of them wrong.

The redesign: classify, then bound, then latch

Raising thresholds was never going to be enough, because the deeper defect was structural. The old watchdog had exactly one response to any breach: stop and page. It could not tell a recoverable hiccup apart from a permanent fault, so it treated everything as permanent. The redesign gives it a decision: classify the breach, restart if it is on a known-safe allowlist, latch if it is anything else, and stop restarting before restarting becomes the outage.

Four conditions landed, each closing a specific way a self-healer can hurt you.

C1: Fail-closed classifier

RECOVERABLE is an allowlist, not a denylist. Exactly two reasons are recoverable: c8-unhealthy and staging-restart-anomaly. Everything else latches immediately: a RAM floor, disk, the data tree, a memory-PSI breach, a prod-unhealthy signal, an unexpected listener (that is a security event), and critically any reason the classifier has never seen.

classify_breach() {
  case "$1" in
    c8-unhealthy|staging-restart-anomaly) echo "RECOVERABLE" ;;
    *)                                    echo "NON_RECOVERABLE" ;;
  esac
}

The default arm is the whole design. A denylist would restart on anything it forgot to name. This allowlist latches on anything it does not explicitly bless. When someone adds a new breach reason next quarter and forgets to classify it, the watchdog latches and pages instead of restarting into an unknown state. That is fail-closed: the unknown path is the safe path. Do not blindly restart.

C2: Slow-flap guard

A container that restarts, looks healthy, then breaches again an hour later can loop for days, each cycle a small outage. BI_MAX_LIFETIME_RECOVERIES=3 caps lifetime recoveries per burn-in. After the cap, a recoverable breach latches instead of restarting forever. Recovery is bounded, not infinite.

C3: Health poll, not snapshot

A restarted container is not healthy the instant it comes up. Verifying health with a single snapshot can latch a container that is still starting. verify_healthy polls up to BI_HEALTH_TIMEOUT=120s, requiring staging active AND C8 healthy, before it decides anything.

verify_healthy() {
  local deadline=$((SECONDS + BI_HEALTH_TIMEOUT))
  while (( SECONDS < deadline )); do
    if staging_active && c8_healthy; then return 0; fi
    sleep 5
  done
  return 1   # never latched on a single still-starting snapshot
}

C4: Reset hygiene and honest outcome

A new burn-in-reset.sh clears recovery.json and incident.json. RECOVERED is logged only after a verified-healthy poll passes, never optimistically at restart time. The first breach is preserved exactly once in incident.json, so the original signal survives even after a successful recovery. recovery_count is surfaced in status.json, and recovery pages carry severity info, because a recovery is not an incident and should not read like one.

Verification is the product

A self-healer is only as trustworthy as the failures you can prove it survives. The redesign shipped with four planted scenarios, each run 20 times, 20 for 20:

transient-recovery: breach, restart, verify healthy, log RECOVERED.
persistent-failure: breach latches immediately, zero restarts.
notification-failure: recovery proceeds, logs an honest "receipt non-zero," fires no false page.
retry-exhaustion: exactly MAX_RESTARTS attempts, then latch, no infinite loop.

The two that matter most are the ones that assert the watchdog does nothing. Persistent-failure must latch with zero restarts, because a fault that restarting cannot fix must never be restarted. Retry-exhaustion must stop at the cap and latch, because the whole point of a bound is that it holds under a fault that never clears. A test that only proves the happy path (breach, restart, healthy) proves nothing about safety. The proof that a self-healer is safe is the set of scenarios where it correctly refuses to act.

All three scripts are shellcheck clean. The scenarios are wired into validate:signoz-staging, and pnpm check exits 0. Guardian review returned approve-with-conditions, and the four conditions above are those blocking conditions, landed.

The parallel honesty story

The same day, a second system taught the same lesson from a different angle. The governed alert chain went live on both hosts, and the trap was sitting in plain sight: notify.sh exits 0 unconditionally. It always reports success. Reuse it as the alert transport and you recreate a prior defect where 108 alerts were false-delivered: reported sent, never sent.

An alert transport that always exits 0 is not a monitor. It is a monitor-shaped object that lies. The fix was a new vps-slack-transport.sh that does an honest webhook POST, classifying the HTTP status and treating a non-2xx as a real failure instead of trusting the exit code. notify.sh was left untouched for its other callers, so the honesty lives in the transport that needs it.

That verification was as ruthless as the watchdog's: six drills, a bidirectional rollback rehearsal that succeeded in each direction, and a 26-case proof covering 22 failure modes plus 4 non-delivered outcomes, with live green sweeps captured on both hosts. The dev box also got the migrated scorecardecho uptime producer, replacing a stale pre-adoption copy that cron had been quietly running since 07-16, a real docs-versus-reality gap, plus an external dead-man's-switch ping. The guardian loop stayed honest too: round one blocked on an untracked liveness-bypass exclusion, which became a filed follow-up, and round two approved.

Both systems converge on one rule. A monitor that always reports success is worse than no monitor, because no monitor at least tells you nothing is watching. A lying monitor tells you everything is fine. The watchdog's fail-closed classifier and the transport's 2xx check are the same idea wearing different clothes: make the honest outcome the default outcome.

Also shipped

Outside the watchdog incident, three smaller things landed the same day.

diagnostic-pro (PR #23): added the in-repo MiniMax advisory PR-review workflow (defect lane plus adversarial-claims lane), the same reviewer already running in claude-code-plugins and intent-os. Runs on pull_request (never pull_request_target), same-repo head guard so fork PRs never see the API key, gated behind an ENABLE_MINIMAX_REVIEW kill switch, advisory only. Enforcement travels with the code.
claude-code-plugins: fixed a heredoc bug in the MiniMax A-grade-coach lane. A head -c 16000 byte cut landed mid-line, leaving no trailing newline, so the closing __MINIMAX_EOF__ delimiter fused onto the last content line and GitHub Actions aborted the step with "Matching delimiter not found." Fix: force a trailing newline plus a per-run random delimiter. Only triggered on PRs big enough to fill the 16KB cap.
learn-intent-solutions-hub: ported the CPN cohort hub off Cloudflare onto Intent Solutions infra. Four runtime seams swapped: Workers export default app to @hono/node-server; D1 drizzle(c.env.DB) to a better-sqlite3 file; c.env.ASSETS.fetch() to Caddy file_server; c.env.EXPORT_TOKEN to process.env. Added /api/health for a real post-deploy smoke. Product behavior, schema, and routes unchanged.

Passing Is Not Validating: A Green Check With No Teeth

Jeremy Longshore — Wed, 22 Jul 2026 10:28:29 +0000

A validator gate returned green on every check. Nineteen rule checks across four rules, all passing. And most of them were exercising nothing at all.

The Contract

Intent Solutions built a repository-identity contract to canonically identify GitHub repositories in a registry. The question: what is the single source of truth for a repo's identity when repos get renamed and transferred between owners and orgs?

The design decision was sound. Use the immutable GitHub database ID (the numeric source_repository_id), never the mutable owner/name pair. Owner changes on transfer. Name changes on rename. Those belong in an identity_history list, not the primary key.

The schema enforced this intent.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["source_repository_id"],
  "properties": {
    "source_repository_id": {
      "type": "integer",
      "minimum": 1,
      "description": "The ONLY identity key. A string here is an invalid record by construction."
    },
    "current_owner": {
      "type": "string"
    },
    "current_name": {
      "type": "string"
    },
    "identity_history": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "owner": {"type": "string"},
          "name": {"type": "string"},
          "changed_at": {"type": "string", "format": "date-time"}
        }
      }
    }
  },
  "additionalProperties": false
}

A separate validator rule-gate existed to prove the identity rules held in practice. Four rules. Nineteen check statements total. All green.

The Symptom

The code-review lane weakened the schema as a regression test. It changed source_repository_id from {"type": "integer"} to {"type": ["integer", "string"]}. That should break the whole identity model. A string could now be the identity key. The validation rules should go red.

They stayed green.

That is the proof: a check has no teeth. You sabotage the exact thing it exists to reject, and it does not notice.

The Bug

The first rule, R1, was supposed to validate rename and transfer survival. This is what it looked like.

def test_r1_rename_transfer_survival():
    # Construct test records
    renamed = {
        "source_repository_id": 12345,
        "current_owner": "new-owner",
        "current_name": "new-name"
    }
    transferred = {
        "source_repository_id": 12345,
        "current_owner": "other-org",
        "current_name": "old-name"
    }

    # Arithmetic check (never validates against schema)
    assert renamed["source_repository_id"] == transferred["source_repository_id"]
    assert renamed["current_owner"] != "original-owner"

    # GREEN. Tests nothing.

It constructed its own data. It did arithmetic on that data. It never called the actual jsonschema validator. It was testing the test, not the system.

This is a tautological test: it builds its own input, asserts something about that input, and never runs the system under test. It cannot fail for the reason it exists, because the thing it guards is never exercised.

The Fix

The teeth check: deliberately weaken the invariant and assert the rule goes red.

def test_r1_rename_transfer_survival():
    # Valid snapshot 1: rename
    renamed = {
        "source_repository_id": 12345,
        "current_owner": "new-owner",
        "current_name": "new-name"
    }

    # Valid snapshot 2: transfer
    transferred = {
        "source_repository_id": 12345,
        "current_owner": "other-org",
        "current_name": "old-name"
    }

    # Positive control: both snapshots must pass the real schema
    validator = jsonschema.Draft7Validator(schema)
    validator.validate(renamed)
    validator.validate(transferred)
    assert renamed["source_repository_id"] == transferred["source_repository_id"]

    # Teeth check: weaken the schema (allow strings as source_repository_id)
    weakened = copy.deepcopy(schema)
    weakened["properties"]["source_repository_id"]["type"] = ["integer", "string"]

    # With the weakened schema, a malformed record using owner as identity should be REJECTED
    invalid_by_string_id = {
        "source_repository_id": "original-owner",  # Wrong: string, not integer
        "current_owner": "new-owner"
    }

    # TEETH CHECK: the original validator rejects the string, the weakened one accepts it
    weak_validator = jsonschema.Draft7Validator(weakened)
    weak_validator.validate(invalid_by_string_id)  # Passes with weakened schema

    # Original schema must reject it
    with pytest.raises(jsonschema.ValidationError):
        validator.validate(invalid_by_string_id)

Now R1 fails when you sabotage the schema. It has teeth.

The other gap: the schema used annotation-only format for timestamps instead of validating pattern.

{
  "changed_at": {
    "type": "string",
    "format": "date-time"
  }
}

JSON Schema draft-07 does not enforce format by default. You have to opt in. The fix added explicit regex.

{
  "changed_at": {
    "type": "string",
    "pattern": "^\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}Z$"
  }
}

The rewrite grew the suite while it was at it. The same four rules now drive twenty-eight valid and twenty-five invalid fixtures through the real validator, up from nineteen hollow checks. Each invalid one now genuinely fails for its specific planted reason, not an accidental different one.

The Lesson

A test that constructs its own input and does arithmetic on it is testing the test. The system never sees the validator.

The fix is the teeth check. To prove a check has teeth:

Construct valid data that passes the real validator.
Deliberately break the invariant the check exists to protect (weaken the type, drop the constraint, plant the exact invalid case).
Assert the sabotaged case now slips through the weakened version.
Assert the original, unbroken validator still rejects it.
If step 4 does not go red, the check has no teeth.

This is the same failure mode as a mock that mocks the function under test, or a snapshot test that snapshots the bug. Mutation testing is the industrial version of the same idea.

Green checks are not proof of anything. Proof is: you sabotage the invariant and the check goes red.

Also Shipped

Bob's Big Brain registrar added a seam firewall so governance scores are branded and retrieval scores cannot cross into them. L4 integration test now gates every PR.

Diagnostic Pro converted to Capacitor iOS and Android mobile apps, and onboarded to VPS auto-deploy with Umami analytics.

Mission Control operator kit built, proven, and wired.

A Green Recovery Drill Can Still Be Lying

Jeremy Longshore — Mon, 20 Jul 2026 10:14:39 +0000

We had a backup mechanism for a SigNoz staging stack. What we did not have was a proven restore. Those are different claims, and the gap between them is where outages become permanent.

SigNoz is open-source observability backed by ClickHouse. The staging deployment on our self-hosted VPS ran the usual set: ClickHouse for telemetry, clickhouse-keeper for coordination, and Postgres for stack metadata. Snapshots existed. Nobody had ever restored one and confirmed the data came back queryable. A backup you have never restored is a hypothesis, not a safety net.

So the job was to build a real drill: seed known telemetry, back the stack up hot, restore it somewhere safe, and verify the exact seeded data survives. The interesting part is not that we built it. The interesting part is what stood between a drill that runs and a drill that means something: four traps, found in the order below. Two failed loudly and honestly, and fixing them taught us the setup. The other two went green while proving nothing, and one of those would have lied silently forever. The loud ones you catch by running the drill. The silent ones you catch only by auditing the proof.

The shape of the work

Three scripts, each with one job.

Hot backup. Tar the ClickHouse, clickhouse-keeper, and Postgres state trees while the stack keeps running. Read-only, zero downtime. Staging never stops serving.

Isolated restore. Unpack the backup into a disposable, network-isolated ClickHouse plus keeper pair. Guaranteed teardown at the end. It never shares a network with live and never touches the production containers.

Recovery drill. Seed synthetic telemetry through the live ingest gateway, run the backup, run the restore, and verify the seeded signals come back by run id. Every signal we push goes through a disclosure-filter gateway (we call it C8) that strips prohibited content before anything reaches storage. That filter is a governance requirement, and as you will see, it is also the source of two of the four traps.

The target scorecard was concrete: a specific seeded run recovers as a trace and a metric and a log, a planted prohibited phrase is absent after restore, all six SigNoz databases come back queryable, and live staging stays healthy the entire time.

Green on that scorecard sounds like proof. Getting there honestly took four corrections.

Trap 1: the macros trap

The first failure was honest and loud, which is the good kind.

ClickHouse ReplicatedMergeTree tables encode their coordination path using {shard} and {replica} macros. Those macros resolve against server config. On restore into the disposable container, the tables refused to attach:

Code: 36. DB::Exception: No macro 'shard' in config

The macros live in the config.yaml we mounted into the container. So why were they missing? Because ClickHouse does not read every config file you hand it. It selects one primary config file, and the image default config.xml won that selection over our mounted config.yaml. The macros were present in a file the server never treated as primary.

You can fight ClickHouse over which primary file wins. Or you can stop fighting. ClickHouse always merges everything under /etc/clickhouse-server/config.d/, regardless of which primary file it picked. So the fix was a tiny drop-in:

<!-- /etc/clickhouse-server/config.d/macros.xml -->
<clickhouse>
    <macros>
        <shard>01</shard>
        <replica>replica-restore</replica>
    </macros>
</clickhouse>

Mount that into config.d/, and the macros are guaranteed present no matter which primary config the server elects to load. Tables attached on the next run.

Lesson: for configuration that must be present, use the always-merged drop-in directory, not the primary file that some other layer might override. Do not depend on winning a precedence fight you do not control.

Trap 2: the vacuous canary

With tables attaching, the disclosure filter needed proving. The assertion is negative: after restore, a prohibited string must be absent, showing the filter's clean state survived the round trip.

To prove a thing is absent, you first have to put a candidate in. So the drill seeds a canary through the gateway, then checks it is gone after restore. The first canary was a random marker string, something like CANARY-7f3a91.

It passed. It also proved nothing.

The gateway's filter does not strip arbitrary text. It matches a specific set of prohibited phrases. A random marker is not on that list, so the gateway correctly passed it straight through to storage. The canary was present in the backup, present in the restore, and the "absent after restore" check was reading the wrong signal entirely. Worse, if the check had been written to expect absence, it would have failed for the right-looking wrong reason, and we would have chased a filter bug that did not exist.

The canary has to be a phrase the filter actually targets. Only then does its absence mean the filter did its job through backup and restore.

Lesson: a negative-space test, proving something is absent, is only meaningful if the thing would otherwise be present. An absence assertion over data that was never going to be there is vacuous. It is green by construction and tells you nothing.

Trap 3: the poisoned run

Fixing the canary broke a different check, and the way it broke is instructive.

The synthetic producer stamps the canary into three places in a run: the span name, the run's one metric name, and its one log body. Once the canary became a real prohibited phrase, the gateway did exactly what it is built to do. It dropped the whole signals that carried it. The span survived in altered form, but the metric and the log were filtered out completely.

Now look at what the full-signal verifier needs: a trace and a metric and a log, all sharing the run id. A canary-carrying run cannot supply that, because the metric and log it needs are precisely the signals the filter destroys. The two goals are in direct conflict. Proving the filter strips prohibited content requires data the filter will delete. Proving full-signal recovery requires data that survives untouched. One fixture cannot be both.

The fix is to stop asking one run to do two contradictory jobs. Seed two runs per drill:

A clean run with no prohibited content, used to prove run-id recovery across trace, metric, and log.
A canary run carrying a real prohibited phrase, used to prove the filter's round-trip fidelity by confirming the phrase is absent after restore.

Each run answers exactly one question, and neither question undermines the other.

Lesson: one synthetic fixture cannot serve two assertions that require opposite data conditions. When two checks demand contradictory inputs, that is a signal to split the fixture, not to weaken a check.

Trap 4: the false-pass by default

The drill was green. Every check passing: containers up, queries answered, all databases back, clean run recovered by run id, canary absent. It looked done.

Then we ran an adversarial code review over the harness itself, and it found the trap that matters most, because this one would have lied silently forever.

The verifier script queries ClickHouse for the seeded run id. Its query target, the container it talks to, defaulted to the live container when an environment variable was unset. Read that again in context. In step one, the synthetic run is seeded through the live gateway. That means the run genuinely exists in live storage. So if any future call, any refactor, any copy-pasted invocation forgot to set the target variable, the verifier would query live, find the run that live legitimately holds, and report PASS.

The restore could recover absolutely nothing. The disposable container could be empty. The verifier would still go green, because it was quietly asking the wrong database, and the wrong database happened to have the answer. A broken restore would pass every single time, and the drill would keep congratulating itself.

The fix is to make the target impossible to omit:

# before: target falls back to live when unset
CH_CONTAINER="${CH_CONTAINER:-signoz-clickhouse}"

# after: explicit, required, no default to a happy source
restore-verify --container "$DISPOSABLE_CH" --run-id "$RUN_ID"
# the script exits non-zero if --container is not provided

Every restore-verify call now names its disposable target in code. There is no fallback, so there is no path where a forgotten variable routes the query to live. If you do not say where to look, the verifier refuses to run.

Lesson: a verifier that defaults to the happy source is a false-pass generator. If the thing you are trying to prove recovered is also present somewhere the verifier can silently reach, an unset default will find it there and lie. Make the target explicit and make omitting it a hard error.

The collaboration beat

Worth being honest about how the fourth trap surfaced. The build ran on Claude Opus 4.8. The macros trap, the vacuous canary, and the poisoned run all came out through iteration: a check failed or passed suspiciously, we traced it, we fixed it. Normal engineering.

The false-pass-by-default was different. It did not fail. The drill was already green when we ran a dedicated code-reviewer subagent adversarially against the harness, with a narrow charge: can any verify step false-pass, and can any path touch live? That framing, aimed at the proof rather than the data, is what caught the live-default fallback. The drill had proved the data recovers. The adversarial review proved that the proof itself was not yet trustworthy. Those are two different audits, and passing the first does not grant the second.

Result

The final drill passes eight of eight checks, and now each one means what it says. The eight, in order:

The disposable keeper container starts.
The disposable ClickHouse container starts.
The restored ClickHouse accepts queries.
All six SigNoz databases are present.
The seeded signal rows are recovered and queryable.
The metadata tables are recovered.
The clean run recovers by run id, as a trace, a metric, and a log.
The prohibited canary is absent in the restore, confirming filter round-trip fidelity.

Two measurements come out of the same run, RTO about 30 seconds and RPO about 2 minutes, and two invariants hold every time: the disposable stack tears down, and live staging stays healthy and untouched. RTO and RPO are now numbers from an exercised drill, not aspirations from a runbook nobody has run.

Takeaway

Green is a claim, not evidence. A recovery drill is a program, and like any program it can be correct-looking and wrong. Two of these traps failed loudly, the macros error and the poisoned run, and that is the honest kind: the drill told us the setup was broken before it ever went green. The other two are the dangerous kind. The vacuous canary went green by construction, and the live-default verifier would have gone green over a completely empty restore, forever. You find loud failures by running the drill. You find silent false-passes only by auditing the proof itself.

For any recovery or verification harness, the question to ask is not "does it pass." It is "what would make this pass when it should fail," enumerated concretely, then tested one by one. Would an absent-by-default value satisfy an absence check? Does one fixture quietly serve two contradictory assertions? Can the verifier reach a source that holds the answer for the wrong reason?

None of these four are specific to ClickHouse or SigNoz. A config drop-in that must always merge, a negative test that needs its subject to actually exist, a fixture pulled between two goals, and a verifier that defaults to the happy source: those generalize to any recovery drill you will ever build. The stack was observability. The lesson is about trusting your own green.

Let the Model Judge. Make the Code Decide.

Jeremy Longshore — Sat, 18 Jul 2026 14:07:31 +0000

This post was written by the pipeline it describes. The transcript analyzer that runs on every build day looked at the work that produced this system and reported the receipts: four different models did the building (Claude Opus 4.8, Claude Fable 5, Grok 4.5, and GPT-5.6 Sol), across 459 minutes, with 91 moments where something failed and got fixed, and 5 places a human stepped in to change direction. That is not a brag. It is the point. No single model built this, no model committed a line of it, and the record of who did what is machine-readable because deterministic code wrote it down, not a model's memory.

The rule the whole day came down to is short. When you put a language model in a production pipeline, split the work. The model owns judgment: writing, scoring, proposing. Code owns the decisions: what ships, what gets enforced, what gets recorded. Keep them strictly apart. Call it produce then land. Yesterday it got applied across an entire writing system in one pass, and every fix was the same fix wearing a different shirt.

The pipeline had let the model creep

This blog publishes a post most days with no human in the loop. A cron job wakes up, a Claude Code skill writes the day's post, classifies it into a tier, and it goes live. That part worked. What had rotted was everywhere the model had quietly taken over a job that code should own.

Five symptoms, one disease:

The model was in the git-commit path. It wrote the post and it also decided the post was good enough to ship, then pushed. A model having a bad morning could push a bad morning straight to production.
The voice rules lived in four files. The deny-list of banned phrases was restated in the skill, in a doc, in an agent prompt, and in a checklist. Four copies means four truths, and they had already started to disagree.
A "learned patterns" file that nothing read. Every tier misclassification was supposed to teach the system. The lessons got written to a file. The file was never loaded. times_applied sat at zero.
The persona's own facts had drifted. The star count for the open-source project showed up as 1,300 in one place, 2,000-plus in another, 2,100-plus in a third, and 2,519 in a fourth. Same day. Four homes, four numbers.
The transcript scanner threw away the actual work. It was supposed to measure how each day got built. Instead it dumped truncated chat and called it analysis.

Every one of those is the same mistake. A model was trusted to be the source of truth for something that has an exact, checkable answer. Judgment is the model's job. Facts, gates, and records are not judgment. They are code's job. The rest of the day was moving each thing back to the side of the line it belonged on.

I ran restaurants for twenty years before I wrote production code. A line cook has enormous judgment: seasoning, timing, when a plate is wrong and goes back. The line cook does not decide whether the kitchen passes the health inspection. That is a checklist somebody else signs. You do not let the person cooking the food also be the person who certifies the kitchen is clean. Same rule here.

Produce then land: the model can write junk, but it can never ship it

The first and biggest fix was pulling the model out of the git-commit path entirely.

Before, the skill did everything: wrote the post, judged it, committed, pushed. After, the skill writes two things and stops. It writes the post. It writes a small readiness sentinel, a JSON file that says "I believe this is ready, and here is why." Then it does no git at all.

{
  "date": "2026-07-16",
  "ready": true,
  "tier": 3,
  "gates_passed": ["hugo_build", "consistency_audit", "fact_check"],
  "post_path": "content/posts/let-the-model-judge-make-the-code-decide.md"
}

A separate script, blog-land.sh, does the landing. It is pure bash. No model runs inside it. It reads the sentinel, then it re-checks every precondition itself, from scratch, because a model claiming a gate passed is a claim, not a fact:

# blog-land.sh (shape, trimmed)
sentinel="$STAGING/$DATE.intent.json"

ready=$(jq -r '.ready' "$sentinel")
[ "$ready" = "true" ] || quarantine "sentinel not ready"

classifier_record_exists "$DATE"        || quarantine "no classifier record"
audit_addendum_present "$DATE"          || quarantine "no step-8 audit addendum"
hugo --buildFuture --gc --minify        || quarantine "hugo build failed"

# only now does anything touch git
git add "$post_path" && git commit -m "$msg" && git push
publish_to_tonsofskills "$post_path"
queue_crosspost "$post_path"

If any check fails, the post gets quarantined. The script moves the file aside, restores a clean working tree, and fires an alert. It does not publish something half-baked, and it does not leave a wrecked repo for tomorrow's run to trip over.

That last part matters more than it sounds. The old failure mode was a stuck run. A gate would hang, a model would time out mid-commit, and the tree would be left dirty. The next day's job would start in a broken state and fail too. One bad morning became a bad week. Now a bad produce step just means one quarantined post and a clean tree. Tomorrow runs like nothing happened.

The model can write a bad file. It cannot ship one. The commit path has no model in it. That is the entire idea, and it is worth saying plainly because it is the thing most agent pipelines get wrong: they let the smart component also be the trusted component. Smart and trusted are different properties. Code is dumb and trustworthy. Use it for the part where trust is the requirement.

Single source of truth, applied twice

Two of the five symptoms were the same bug: a fact copied into many places instead of referenced from one. Both got the same fix, and the fix has a name. Reference, do not copy.

The voice deny-list

The list of banned phrases (the AI-slop tells: delve, seamless, game-changer, and two dozen more) had been pasted into four files. When I added a phrase to the linter, the doc still had the old list. The agent prompt had a third version. Nothing was wrong exactly, but nothing agreed either, and drift is just being wrong on a delay.

It collapsed to one file:

// voice-denylist.json (26 phrases, the one true copy)
{
  "banned_phrases": ["delve", "dive into", "seamless", "supercharge",
    "leverage", "game-changer", "revolutionize", "comprehensive",
    "it's worth noting", "at its core", "in conclusion", "..."],
  "banned_chars": ["em-dash", "en-dash"]
}

The linter reads it. The docs reference it by path instead of restating it. Add a phrase in one place and every consumer sees it on the next run. There is exactly one answer to "is this phrase banned," and it lives in one file.

The persona star count

The other copy-drift was uglier because it was public. The open-source project's star count was quoted in the writing persona, in the site's own about copy, in a partner doc, and in the blog's byline material. Four numbers: 1,300, 2,000-plus, 2,100-plus, 2,519. All stale except by accident, because a hand-typed number is stale the moment the counter ticks.

The fix was a canonical persona source of truth in the intent-os repo (persona/master.md), with the number stated once and a live-verify command sitting right next to it so nobody ever guesses again:

# live-verify the OSS numbers instead of quoting from memory
gh api repos/jeremylongshore/claude-code-plugins \
  --jq '"stars=\(.stargazers_count) forks=\(.forks_count)"'
# stars=2519 forks=362

The canonical text now reads "2,500-plus stars, 360-plus forks, 45,000-plus npm downloads." Rounded down, honest, and every other file points at that one instead of carrying its own copy. Two merged pull requests did the cleanup: one stood up the canonical persona (master.md and voices.md), one corrected the star count on the personal site. A model does not own a fact that a command can check. That is not a judgment call. It is a lookup, and lookups belong to code.

Deterministic enforcement of learned judgment

Here is the one that was quietly the most broken, because it looked like it worked.

The tier classifier scores each post and assigns a tier: field note, deep-dive, case study, distinguished paper. When it got a tier wrong, a feedback step recorded the correction as a "learned pattern" in patterns.jsonl. The intent was a system that gets sharper over time. A pattern looks like this:

{
  "id": "tch-anchor-cap",
  "condition": {"tier_claimed": ">=2", "named_artifact_count": 0},
  "action": {"cap_tier": 1},
  "evidence": "Tier 2+ requires a named artifact; 0 present downgrades to 1",
  "times_applied": 0
}

See the times_applied: 0. That was the tell. The patterns were being written and never read. The "learning loop" was a human hand-copying each learned rule back into the prompt by hand, which nobody had done in a while. The file was a diary, not a control.

The fix was a deterministic engine, apply-patterns.py, that loads the rules and evaluates each one against the actual scores, then caps the tier when a rule matches. The model still scores the post. It reads the artifacts, weighs the evidence, proposes a tier. That is judgment, and it stays with the model. But the enforcement (does this proposed tier survive the learned rules) is code now:

def enforce(scores, patterns):
    tier = scores["proposed_tier"]
    for p in patterns:
        if matches(p["condition"], scores):
            tier = min(tier, p["action"]["cap_tier"])
            p["times_applied"] += 1
    return tier

The model scores. The code enforces. And the moment I ran the engine against history, it earned its keep: one active rule matched 7 of 238 historical decisions, and 4 of those were real downgrades, exactly matching the evidence the rule had recorded about itself. The system had been right about its own blind spot the whole time. It just had no hands.

Wiring it up also surfaced two silent import bugs. The analytics index that was supposed to hold every feedback entry and every pattern had zero rows in both tables. Zero. Two import paths had been quietly failing, so the loop had been writing to a void for who knows how long. This is the tax you pay for a "learning" component nobody checks: it can be fully disconnected and still look alive, because the diary still fills up. Only when code started reading the diary did the disconnection show.

Measure how the work actually happened

The transcript analyzer got a full rewrite, and it is the clearest example of the split, so it is worth walking.

The old scanner grabbed chat logs, truncated them, and pasted them into a summary. Useless. It threw away the one thing worth keeping, which is what actually happened during the build, and kept the one thing worth throwing away, which is raw model chatter.

The new scan-session-transcripts.py reads every AI-coding CLI used on a given day: Claude Code, Grok, Codex. It names each model exactly, because "an AI helped" is not a fact and "Claude Opus 4.8, Claude Fable 5, Grok 4.5, and GPT-5.6 Sol" is. Exact model names are also just good SEO; people search the model, not the vibe. Then it extracts the collaboration deterministically:

Tool activity: how many edits, reads, bash runs, which files.
Failure-to-fix arcs: a command failed, then a later command succeeded on the same target. That is a debugging loop, and counting them is arithmetic, not opinion.
Human course-corrections: the human redirected the work. Counting them is arithmetic too.

Run on its own build day, the analyzer reported the numbers this post opened with: four models, 91 failure-to-fix arcs, 5 course-corrections, 459 minutes. The model in this pipeline does exactly one thing with all that: it narrates it into a paragraph. Every number is extracted by code. The model writes the sentence around the numbers. It never invents one.

The rewrite also does secret redaction on any code it quotes, and that guard caught a live one. A real API token had leaked into a "command not found" error string in a transcript. The redactor scrubbed it before it could reach the summary. If the model had been trusted to "just summarize the logs," that token ships in a blog post. Deterministic redaction on the extraction path is the difference between a near miss and an incident.

There is a second loop in the same family worth naming: next-topics.py. It reads Umami content-performance data (which posts actually got read) and turns it into a ranked queue of what to write next. Real reader behavior in, ranked topics out. The model writes the posts; the data decides the order. Same split, different surface.

The honest calibration call

Not every decision resolved cleanly into "code decides." One was a genuine judgment tradeoff, and pretending otherwise would be dishonest.

The transcript analyzer produces a "session signal" that feeds tier classification. The idea: a day with heavy, hard engineering probably produced a meatier post. Reasonable. But the raw signal is volume, and the volume is almost always "strong," because a normal day here runs 1,600 to 2,500 tool calls. If I wired a mechanical floor ("strong signal equals bump the tier"), every post would inflate to case study on day one. The tripwire that guards against tier creep would scream every week, correctly, because I would have built the creep myself.

So the call was: set the signal from real data, but do not let it mechanically set the tier. The signal is an input the model weighs against the specific arcs of the day (did something genuinely break and get solved, was there a real design decision), not a volume threshold that auto-promotes. Code owns the measurement. The judgment about what the measurement means stays judgment. That is the honest version. The split is a principle, not a religion, and the one place it bent was the one place the quantity was real but the meaning was not.

The silence bug

One more, because it is the operator lesson in miniature.

There is a morning packet that emails the day's syndication material out for a human to post. One morning it did not send. And here is the thing: silence looked exactly like a working pipeline. No error. No alert. Nothing in the inbox is what "all quiet, nothing to post today" looks like, and it is also what "the whole thing is broken" looks like. The two states were identical from the outside.

The fix was a positive heartbeat. Now, when there is genuinely nothing to send, the pipeline sends a one-line "NO PACKET TODAY" note on purpose. An expected message. The absence of that message is now itself an alarm, because the only way to get silence is a real failure.

I checked kitchens the same way. An empty prep list is not "we are ready." An empty prep list means nobody did prep. You do not get to assume the quiet means good. You make good produce a signal, so the quiet can only mean bad. Silence was the actual defect. The missing email was not the bug's symptom; the missing email was the bug, and the fix was to make silence impossible.

Why deterministic-owns-the-gates beats letting the model do it all

Step back from the five fixes and the shape is one design decision made five times. The gate, the fact, the enforcement, the measurement, the record: all pulled out of the model and given to code. Here is why that is not just tidiness.

Reliability. A model is a probabilistic component. It has good mornings and bad mornings, and you cannot schedule which. Code is deterministic. When the trustworthy part of your pipeline is the deterministic part, the model's bad morning costs you one quarantined post, not a corrupted repo.

Auditability. "Four models, 91 failure-to-fix, 5 course-corrections, 459 minutes" is a fact I can stand behind because code counted it. If a model had summarized the day from memory, I would have a vibe, not a receipt. Receipts are what let this post open with real numbers instead of a hand-wave.

Reproducibility. The learned patterns went from a diary nobody read to a rule engine that produces the same downgrade every time the same condition appears. 7 of 238 matches, 4 real, repeatable on the next run. A model re-judging from scratch each time is not reproducible. A rule is.

It survives a flaky model. This is not hypothetical. During the build, one model hit its weekly usage limit mid-stream. A fallback model took over. The pipeline still shipped, because the model was never the thing that ships. The commit path is bash. It does not care which model wrote the file, or whether the model that started the post is the model that finished it. When code owns the commit, the model layer can fail over, rate-limit, or swap entirely, and the output does not corrupt. You cannot get that property if the model is holding the git handle.

Across two days this was 34 commits on the blog repo, two merged pull requests on the persona and voice system, and one on the personal site to kill the star-count drift. Every one of them was the same move: find a place a model was trusted with a decision, and give the decision to code.

The lesson

If you ship agents in production, the model is the worker on the line, not the manager who signs off. It has real skill and real judgment, and you should use all of it: let it write, let it score, let it propose. Then take its work to a gate it does not control. Let the model judge. Make the code decide.

The tell that you have it backwards is comfort. If your pipeline feels clean because the model "handles everything end to end," you have handed the trust property to the component that has bad mornings. Find the git handle, the fact, the gate, the record. Move each one to code. What is left for the model is the part it is actually good at, and the part where a mistake costs one quarantined file instead of a shipped incident.

Related posts:

Copying Files Is Not Installing

Jeremy Longshore — Sat, 18 Jul 2026 14:07:28 +0000

A marketplace install copies your plugin's files into place. It does not run npm install in them. Those are two different operations, and the gap between them is where "works on my machine" lives.

We hit that gap shipping Bob's Big Brain, a Claude Code and Cowork plugin that runs a local stdio MCP server to give a team a governed knowledge brain. In the author's checkout it worked. For every teammate installing from the marketplace, one of its two modes died on start. The teammates did nothing wrong. The install path had an assumption baked into it, and the assumption only held on the machine where the code was written.

Two modes, one of them has native code

The server runs in one of two modes, decided at startup.

Team mode talks to a remote API. It reads TEAMKB_API_URL from the environment or a ~/.teamkb/team.json file, sends requests over HTTP, and holds no local state. It is pure JavaScript with zero native dependencies. Nothing to compile, nothing to install.

Local mode runs an embedded store on the user's own machine, backed by SQLite. That mode needs two native Node modules: better-sqlite3 for the database and fs-ext for file locking. Native means compiled C++ bindings, not pure JS. They do not exist until something runs npm install and the build step produces a .node binary for the current platform.

So the dependency surface is lopsided. Team users need nothing. Local users need a compile step to have happened. That asymmetry is the whole story.

The exact failure

Here is the sequence for a clean marketplace install of local mode:

The marketplace copies the plugin directory into place. Source files, package.json, package-lock.json, the .mcp.json manifest. Everything except node_modules.
Claude launches the MCP server named in the manifest.
The server enters local mode and calls require('better-sqlite3').
There is no node_modules. The require throws MODULE_NOT_FOUND.
The server crashes on boot. Local mode is dead before it does any work.

In the dev checkout, step 4 never happened, because npm install had been run there by hand months earlier and node_modules was sitting on disk. The code was correct. The environment was not reproducible. A marketplace install is a file copy, and a file copy carries no build artifacts.

The fix is not "tell users to run npm install." A plugin that requires a manual build step after install is a plugin that fails for most people. The fix is to make the server provision its own native deps, on first start, only when it actually needs them, and to do it safely when several copies of the server start at once.

The bootstrap launcher

The manifest now points at a small launcher instead of the server directly. plugin-runtime/bootstrap.cjs is about 125 lines and uses only Node builtins. That constraint is load-bearing: the launcher has to run before any dependency exists, so it cannot import a single thing from node_modules. fs, path, child_process, module. Nothing else.

On start it makes one decision: does this launch need native deps at all?

// A value counts as "configured" only if it is real and expanded.
// An unexpanded ${VAR} placeholder is NOT configured.
function isConfigured(value) {
  return typeof value === 'string'
    && value.length > 0
    && !value.startsWith('${');
}

function teamModeRequested() {
  if (isConfigured(process.env.TEAMKB_API_URL)) return true;
  const base = process.env.TEAMKB_BASE_PATH
    || process.env.TEAMKB_HOME
    || path.join(os.homedir(), '.teamkb');
  return fs.existsSync(path.join(base, 'team.json'));
}

If team mode is requested, the launcher does nothing and executes the bundled server immediately. Team users pay zero install cost, which is correct: they never touch SQLite. The ${ guard matters more than it looks. When a manifest ships "TEAMKB_API_URL": "${TEAMKB_API_URL}" and the variable is unset, the literal string ${TEAMKB_API_URL} arrives in the environment. Treating that as "configured" would route a local user into team mode and fail differently. So an unexpanded placeholder counts as absent.

If team mode is not requested, the launcher checks whether the native deps are already present before doing any work:

const { createRequire } = require('module');

function nativeDependenciesReady() {
  const requireFromBundle = createRequire(BUNDLE);
  try {
    requireFromBundle('better-sqlite3');
    requireFromBundle('fs-ext');
    return true;
  } catch {
    return false;
  }
}

createRequire(BUNDLE) resolves from the bundle's own root, not the launcher's, so the check asks the exact question that matters: can the server load these when it runs? If yes, provisioning is skipped entirely. Idempotent by construction. If no, the launcher provisions:

npm ci --omit=dev --no-audit --no-fund

npm ci, not npm install. ci installs the exact versions pinned in package-lock.json and errors if the lockfile and manifest disagree. That makes provisioning reproducible: every teammate gets the same better-sqlite3 build the author tested, not whatever "latest" resolves to that day. After the install the launcher re-runs nativeDependenciesReady() and throws if the deps are still missing. A silent partial install is worse than a loud failure.

The concurrency protocol

The hard part is not installing. It is installing exactly once when the server can start several times at once.

An MCP server gets launched per client. Open three Claude windows, or run a few agents, and you get three concurrent bootstraps racing to npm ci the same directory. Two npm ci runs on one directory step on each other and corrupt the tree. A naive "check if node_modules exists, if not install" has a check-then-act race: both processes check, both see nothing, both install.

The lock is an atomic mkdirSync of a .native-install.lock directory. mkdir is the right primitive because it either creates the directory or fails with EEXIST, atomically, at the filesystem layer. There is no window between check and create. A check-then-write lock reintroduces the race; an atomic create, whether mkdir or an exclusive wx open, does not. mkdir is just the simplest primitive that gives it.

function acquireInstallLock(lockDir) {
  const STALE_MS = 5 * 60 * 1000;   // 5 minutes
  const WAIT_STEP_MS = 250;
  const MAX_WAIT_MS = 2 * 60 * 1000; // 2 minutes
  let waited = 0;

  for (;;) {
    try {
      fs.mkdirSync(lockDir);        // atomic: create or EEXIST
      return;                        // we hold the lock
    } catch (err) {
      if (err.code !== 'EEXIST') throw err;

      // Someone else holds it. Three ways out:
      if (nativeDependenciesReady()) return;  // 1. deps already landed
      if (lockIsStale(lockDir, STALE_MS)) {   // 2. crashed installer
        fs.rmSync(lockDir, { recursive: true, force: true });
        continue;                              //    reclaim and retry
      }
      if (waited >= MAX_WAIT_MS) {             // 3. give up loudly
        throw new Error('native install lock timed out after 2m');
      }
      sleep(WAIT_STEP_MS);
      waited += WAIT_STEP_MS;
    }
  }
}

Three exits when the lock is already held, in priority order:

Deps are ready. Another process finished the install while we waited. Return, do nothing, let the server start. This is the common happy path for the second and third windows.
The lock is stale. lockIsStale compares the lock directory's mtime against a 5-minute ceiling. A process that crashed mid-install leaves the lock dir behind forever; without reclamation, one crash wedges the whole team permanently. Older than 5 minutes means the holder is presumed dead, so we remove the lock and retry.
Otherwise wait, bounded. Sleep 250ms and re-loop, up to a 2-minute ceiling, then throw. A live installer gets the time it needs; a pathological state fails with a clear message instead of hanging forever.

The sleep is a real blocking sleep, not a busy-spin:

function sleep(ms) {
  Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, ms);
}

Atomics.wait on a throwaway SharedArrayBuffer blocks the thread for the duration without burning CPU. And the lock directory is removed in a finally around the install, so a successful holder always releases it.

The whole launcher fails closed. Any error, missing bundle, failed install, lock timeout, prints one line to stderr and exits non-zero:

[governed-brain] REFUSING TO START: <reason>

A server that cannot guarantee its own dependencies should not come up half-initialized and answer queries against a store that is not there. Refuse loudly. The operator sees the reason immediately.

Proving it stays fixed

A fix you cannot regression-test is a fix with a shelf life. The environment assumption was invisible precisely because the author's machine always had node_modules. CI had the same problem: it installed deps before running tests, so it never exercised a clean install either.

The new smoke test removes that blind spot. smoke/bootstrap-clean-install.mjs copies the shipped plugin into a scratch directory with no node_modules, launches the actual MCP artifact through the bootstrap, and drives a full disposable round trip against local mode: capture a memory, govern it, read status, verify the audit chain, then clean up. It is wired into .github/workflows/smoke.yml, so CI now boots the plugin the way a teammate does. If a future change breaks the clean-install path, the smoke goes red before the release ships. It also exercises the lock path, which is what test(plugin): harden bootstrap concurrency checks covers.

The general lesson, and two smaller ones

If you ship a plugin with native dependencies to any marketplace that installs by copying files, assume the install runs no build step. Provision deps yourself, at runtime, gated on a readiness check so you do nothing when there is nothing to do, and protected by a real lock because your process can start concurrently. Pin with npm ci so every install is the build you tested. Fail closed when you cannot.

Two hygiene fixes rode along in the same release, both small, both the difference between an install that reads clean to a new user and one that does not. The MCP manifest's description had command-shaped text in it, so it got rewritten as plain prose. A description is metadata a human reads, not a script, and shell syntax in a schema field is just noise that trips linters and confuses readers. And a team-onboarding README example pointed at an invalid API URL, corrected to a valid local example, so the first thing a new team user copies actually works.

This shipped as v1.1.2. The one-line version: copying files is not installing, and a plugin that assumes otherwise works only on the machine where the files were already installed.

Producer Fallback: When Claude Hits the Weekly Limit, the Pipeline Still Ships

Jeremy Longshore — Thu, 16 Jul 2026 16:18:30 +0000

A daily content pipeline that dies because one model returns a rate-limit error is measuring the wrong success criterion. The job is not "Claude ran." The job is "yesterday has a post on the live site."

On 2026-07-15 that distinction stopped being theoretical.

What broke at 04:00

Two failures, one after the other:

Dirty tree preflight. Closing a bead the night before left one uncommitted line in .beads/interactions.jsonl. preflight_branch_normalize correctly refused to run on a dirty master. No produce step. No post for 2026-07-14.
Claude weekly limit. After the tree was cleaned and the run re-fired, claude -p /blog-backfill exited in three seconds with: weekly limit, resets Jul 19. Land saw no post file and reported NO-POST (rc=20).

That is the same shape as automation assurance on Intent-OS: exit codes and "the scheduled job ran" are not outcome verification. The outcome is a live URL.

Fix one: do not brick the cron on beads interaction noise

.beads/interactions.jsonl is an append-only session audit log. Any bd close dirties it without committing. Treating that as "human has uncommitted feature work" is wrong; treating real content dirt the same as always is right.

preflight_branch_normalize now:

Still FATAL on any tracked dirt that is not interactions-only.
If the only dirty path is .beads/interactions.jsonl, auto-commit it with a mechanical message and continue.

That keeps the clean-tree invariant for posts and methodology files without letting a late-night bead close silence the morning publish.

Fix two: hard voice lint (already on master)

Separately, the produce path now fails closed on AI voice fingerprints: em dash / en dash hard ban, expanded slop phrase list, deterministic lint-post-voice.py in the skill and again in blog-land.sh. Phrase checks mask fenced code and URLs so repo names do not false-positive. Historical posts are not bulk-rewritten; only the post being landed is gated.

That gate is what made the recovered 2026-07-14 post ship without the em-dash density that had become the house default.

Fix three: Grok as producer fallback

Claude remains the primary producer. It is not the only producer.

blog-backfill-daily.sh now:

Tries claude -p /blog-backfill (same as before).
If Claude fails and no post exists for the target date, runs a Grok headless producer with the same contract: write the post, append decisions.jsonl, write .blog-staging/DATE.intent.json with ready:true only after gates (including voice lint). No git.
Always runs blog-land.sh afterward (verify → commit → push → dual-publish → queue, or quarantine).

Env knobs:

Variable	Values	Meaning
`BLOG_PRODUCER`	`auto` (default), `claude`, `grok`	Which producer(s) to run
`GROK_BIN`	path	Defaults to `~/.grok/bin/grok`
`BLOG_GROK_MAX_TURNS`	int	Headless turn cap (default 120)

auto is the production setting until Claude is healthy again, and it stays useful after: a weekly limit or API blip should not equal a missed day.

What actually shipped for 2026-07-14

While Claude was dark, the 2026-07-14 post was produced manually under the same land contract:

Exit 0 Is Not Success: Automation Assurance That Verifies Outcomes
Voice lint PASS, Hugo PASS, dual-publish to tonsofskills + field-notes, live liveness OK
Ezekiel packet re-sent with real social copy after the first send degraded (voice-gen still hits Claude)

The content thesis of that post and the ops lesson of this one are the same sentence: exit 0 is not success; verify the outcome.

Also on the board today

Release bumps and README/changelog sync after the voice-lint and preflight commits landed on master.
IAM Bob packaging release noise (v2.1.6) from the overnight auto-release path: docs-only, not the story of the day.

Takeaway

Producer outages will happen. Rate limits, model downtime, auth glitches. The pipeline should treat them like any other dependency failure:

Do not fail open (publish half-baked).
Do not fail silent (alert + log).
Do not single-home the produce step on one vendor when a second local producer can honor the same artifact contract.
Keep land deterministic and dumb: it does not care which model wrote the markdown.

Tomorrow's 04:00 run can try Claude, fall back to Grok, and still land. That is the bar.

Exit 0 Is Not Success: Automation Assurance That Verifies Outcomes

Jeremy Longshore — Thu, 16 Jul 2026 16:18:27 +0000

A cron that exits zero and produces nothing is not healthy. It is a silent failure wearing a green badge.

That distinction drove most of 2026-07-14 on Intent-OS. Phase-2 landed a stack of fail-closed controls, and the load-bearing one for operations was automation execution assurance: a registry, an engine, and active drills that refuse to treat "process finished cleanly" as "the work got done."

The lie exit codes tell

Schedulers are good at answering a narrow question: did the process start and did it return 0?

They are bad at the question operators actually care about:

Did the backup finish with a restorable artifact?
Did the heartbeat file get a fresh timestamp?
Did the export write the expected tables, not an empty stub?
Did the watchdog that was supposed to catch all of the above still run?

If your only gate is the process exit code, every one of those can fail while the dashboard stays green. We already ship that lesson in other forms (empty data that passed as clean, liveness without health, gates that fail open). Automation was the next place the same shape showed up.

What we shipped: independent outcome verification

PR I on Intent-OS (feat(automation): execution assurance) makes the rule explicit:

An exit code of 0 is not business success.

The design is deliberately boring and mechanical:

registry.json lists each automation's full lifecycle policy: schedule, lateness, max runtime, heartbeat, retry/backoff, catch-up, overlap, idempotency, expected output, outcome verification, owner, and runbook.
assure.py runs a fail-closed status ladder. A completed exit-0 run is classified exit0-no-outcome unless an independent check verifies the expected output.
Watchdog + watchdog-of-watchdog: catching dead jobs is not enough if the catcher itself dies. Escalation covers the uncovered gap.
drills.sh injects the minimum failure set on disposable/synthetic targets and asserts detect → receipt → recover.

The first-run scorecard for 2026-07-14 recorded 34 predicates, 0 failures, including a real process kill (stuck heartbeat), a real exit-0 with no output (exit0-no-outcome), timer disabled, scheduler unavailable, credential missing, stale output, and watchdog failure. The scorecard field that matters for humans is simple: no_false_green=True. Exit-0 is healthy only with a verified outcome.

That is the same discipline as two-marker liveness on the blog fleet (.beat vs .ok): ran is not succeeded. The automation package just names the failure modes and drills them.

Why not trust the job?

The tempting alternative is "the job already knows if it failed; just check its exit code harder." That collapses three different things into one bit:

Signal	What it proves	What it hides
Process started	Scheduler fired	Outcome quality
Exit 0	No crash path	Empty success, skipped body, wrong target
Heartbeat fresh	Something is looping	Correct work product
Outcome check	Expected artifact exists and is plausible	(this is the bar)

Independent verification is not cynicism about the job author. It is an admission that the success path is where silent failure hides. Crashes are loud. Empty green is quiet.

We also fixed a kill 0 process-group hazard in the drill trap while we were there: drills that accidentally widen their blast radius are worse than no drills. Disposable targets only; no production automation was touched in the first-run pack.

Fail-closed neighbors on the same day

Assurance did not land alone. The Phase-2 foundation batch on Intent-OS put the same fail-closed posture in several adjacent places:

Disclosure policy (PR A / C8): canonical fail-closed enforcement at every sink. The irreversible control is the one that refuses to emit when the policy cannot be evaluated cleanly.
Observability failure-detection spine (PR H): registry + engine + disposable drills for detection, not just dashboards.
Agent gateway (PR J): narrow tools, default-deny, signed receipts. Governed agentic ops with an allowlist, not an open shell.
Alert floor (PR C): one authoritative source plus drift enforcement so routing cannot silently rot.
Watchtower scan-gate (PR D): resolve + scan + gate before promote.
Supply-chain CI baseline (PR G): normalize + policy gate + schemas.

The audit trail for the day closed with a Phase-2 re-audit after D88 residuals: conformant with explicit gaps, not "green because we stopped looking."

Also shipped

claude-code-plugins / Freshie inventory. Several production-truth fixes: refuse to export incomplete discovery runs to Dolt; gate public exports behind a table allowlist; stop --fix-agents from stripping required color; floor curated wipes; propagate populate-db failures; harden external-sync supply chain after an ops review. Incomplete data no longer pretends it is a finished inventory snapshot.

DiagnosticPro. Instrument-grade frontend redesign (less template, quieter nav), real Open Graph images, whiteglove PDF customer reports, and live e2e journeys with free-coupon seed cases. Public surface stopped looking like a stock SaaS shell.

Bob's Big Brain rename. Umbrella and plugin directories aligned to bobs-big-brain-* on disk and remote; skill docs tighten the search ladder (keywords first, then wider scope) and marketplace author schema so install stops failing on a CLI type check.

Blog pipeline (this site). Hard ban on em dashes and AI-slop phrases via lint-post-voice.py, skill write rules, and a land-step gate. Phrase checks mask fenced code and URLs so repo names do not false-positive. That work merged after the dirty-tree preflight story below, but the lesson matches the day: soft style guidance without a hard gate is theater.

The dirty-tree lesson (same failure shape)

This post itself almost did not exist. The 04:00 backfill aborted because .beads/interactions.jsonl had one uncommitted line from a late bead close. Preflight correctly refused a dirty master. The content pipeline never ran.

That is exit0-no-outcome in human form: the machine "did something" (bd wrote a log line), then left the system in a state where the real job (publish a post) could not start. Fix was twofold: commit the audit line, and teach preflight to auto-commit only when interactions.jsonl is the sole tracked dirt. Any other dirt still fails closed.

What is still honest ABSENT

Automation assurance is proven on disposable drills. Wiring live schedulers (systemd, cron, GitHub Actions receipts, borg/R2 lifecycle, the real ops/verify stream) as observation sources, and deploying the off-host watchdog-of-watchdog, is still change-control-gated production work. Shipping the engine without claiming the fleet is fully under it is part of the same honesty rule: do not paint partial coverage as complete.

Takeaway

If your automation dashboard only counts exit codes, you are measuring crashes, not success. Put the success criterion in a separate check of the artifact you actually needed. Drill the failure modes on disposable targets until no_false_green is true. Then wire production under change control.

Exit 0 means the process finished. Outcome verification means the work finished. Only one of those is worth a green light.

Empty Is Not Clean: Five Fail-Open Bugs in an AI Agent

Jeremy Longshore — Wed, 15 Jul 2026 13:07:49 +0000

A policy said deny anything under /etc. A Bash call read /etc/shadow and came back {allow, rule: 'trust-bash'}. No human in the loop. The deny rule was still there, still first in the list, still scoped exactly the way an operator would write it. It just never fired.

Two cold-contributor reviewers reproduced it independently on this policy:

const policy = [
  { effect: 'deny', tool: 'Bash', pathPrefix: '/etc' },
  { effect: 'auto_approve', tool: 'Bash', rule: 'trust-bash' },
];
// Bash → /etc/shadow  ⇒  { allow: true, rule: 'trust-bash' }

The gate is the policy engine inside claude-code-slack-channel — an MCP-mediated Slack agent whose whole reason to exist is that it can prove what a tool did: a signed, offline-verifiable audit journal plus a per-tool-call policy engine. If the deny rule can be walked past, none of that matters. This is the story of that bug, and the four other bugs that turned out to be the same bug wearing different clothes.

The pattern: absence read as success

Every one of these had the same shape — the same fail-open bug. When the data a check needs to make its decision was absent, the check defaulted to the permissive answer — allow, pass, clean, covers-all — instead of the safe one. A missing argument. An empty log. A property that isn't there. A chain link that got sheared off. A scope field left unset.

The principle that closes all five: when a check has nothing to check, it must fail closed. Default-deny, not default-allow. Absence is not a green light. Absence is the alarm.

Disguise 1 — empty args made the deny rule vanish

The engine's evaluate() takes a ToolCall and walks the rules. matchApplies() decides whether a rule's scope — pathPrefix, argEquals — matches the call. The exploit lived one level up, in the only production caller.

The MCP permission_request handler in server.ts never receives structured tool arguments. It gets an input_preview string for display and nothing else. So it built the call like this:

// server.ts — the sole production caller
const call: ToolCall = {
  tool: req.tool_name,
  input: {},                      // ← no structured args available here
};
const decision = policy.evaluate(call);

A deny scoped by pathPrefix: '/etc' asks: does input.path start with /etc? Against input: {} the answer is no. matchApplies() returns no-match, the deny is silently skipped, and the next broad auto_approve for Bash swallows the call. The scoped rule the operator wrote was invisible precisely because the engine couldn't see what it was scoped on.

The fix does not try to reconstruct the missing args. It can't — they aren't there. Instead, when the gate cannot see the input a scoped deny/require_approval rule needs, it stops trusting the fall-through:

// If a scoped deny/require rule COULD apply but we lack the input
// to know, escalate to a human instead of falling through to auto_approve.
if (scopedRuleNeedsInput && inputIsUnavailable(call)) {
  return { allow: false, escalate: 'require_approval', reason: 'input-unavailable' };
}

Not knowing whether a deny applies is treated as "it might" — and "it might deny" routes to a human, never to auto-approve.

Disguise 2 — a wiped audit log verified clean

The journal's verifier walks the event chain and reports broken links. Zero events means zero broken links. So an empty audit.log verified clean. An attacker who truncates the entire log to nothing gets a green check — the strongest possible signal from the weakest possible file.

The fix is a floor. --min-events N makes verification fail when the log holds fewer events than expected:

$ verify audit.log --min-events 500
FAIL: expected ≥500 events, found 0

Zero is no longer "clean." Zero is "someone deleted the evidence." The flag parser is deliberately fail-closed too: a malformed value throws rather than silently disabling the floor, because a floor you can turn off by fat-fingering it is not a floor. (A later fix routed a space-form negative like --v2-floor-seq -5 to the value validator so it throws the accurate "must be a non-negative integer" instead of a misleading "missing value.")

Disguise 3 — a prototype key past a truthiness gate, and the near-miss

Channel-policy reads (access.channels[id]) were scattered across seven call sites with inconsistent guards. A bare index read inherits Object.prototype members: ask for the channel 'constructor' or 'toString' and you get back a truthy function that never belonged to anyone's config — enough to slip past a truthiness-based gate.

Not exploitable today; Slack channel ids match [CD][A-Z0-9]+ and never take prototype-key forms. But it's a footgun class, and seven inconsistent guards is seven chances for one to drift wrong. The fix routes every read through one chokepoint that returns a policy only for an own property:

export function getChannelPolicy(access: Access, id: string): ChannelPolicy | undefined {
  return Object.hasOwn(access.channels, id) ? access.channels[id] : undefined;
}

One helper, seven callers, no drift. And this is where a fail-closed refactor nearly shipped a fail-open crash.

access.channels is typed as always-present. It isn't. An Access loaded from disk can arrive without it, and Object.hasOwn(undefined, id) throws. A Gemini audit flagged it HIGH. The old scattered code used access.channels?.[id] — optional chaining that safely returns undefined when the object is missing. The refactor, in the name of hardening, would have converted a defined-safe absence into a thrown exception on every channel read. Whether an unhandled throw at the gate ends up allowing or denying depends on which handler catches it — and "depends on the handler" is exactly the uncertainty a security gate must not have. Defined behavior became undefined behavior at the one point that has to stay defined.

The guard belongs inside the one helper, so it can't be forgotten at any of the seven sites:

export function getChannelPolicy(access: Access, id: string): ChannelPolicy | undefined {
  const channels = access.channels;
  return channels && Object.hasOwn(channels, id) ? channels[id] : undefined;
}

A fail-closed accessor must fail closed. It must not throw. Same discipline, one recursion deeper — the fix for a fail-open bug has to itself fail closed on its missing input.

Why one chokepoint instead of hardening seven sites?

Hardening each of the seven reads in place would have worked on the day it was written. It also would have created seven independent copies of a subtle guard, and the eighth read — the one added six months from now by someone who copied the simplest nearby example — would skip it. A chokepoint is not just less code. It's the only version where the correct behavior is the default behavior. You can't call the channel-policy lookup wrong because there's only one way to call it.

Disguise 4 — a sheared, renumbered chain recomputes clean

The genesis-seq pin from an earlier hardening pass was documented as defeating head-truncation. It does — but only on signed, key-verified (v2) chains. On an unsigned v1 chain the links are bare keyless SHA-256:

hash = sha256(prevHash + canonicalJson(event))

A writer-capable attacker shears the head off, renumbers the survivors so the new first event is seq=1, and recomputes every hash forward. No secret is required — the formula is public and the file contains everything it needs. The sheared file verifies clean because it is internally consistent. It's a smaller, valid-looking journal that never happened.

The close is two optional verification anchors:

verify(log, {
  pinnedGenesisHash: 'a1b2…',  // event[0].prevHash MUST equal this operator-held value
  v2FloorSeq: 1,               // every event at seq ≥ N MUST be v2/signed
});

pinnedGenesisHash is the key move — and it's a different mechanism from the older genesis-seq pin. That earlier pin fixed the sequence number, which a renumbering attacker just recomputes. pinnedGenesisHash fixes the first event's prevHash against a value the operator captured outside the file. When an attacker renumbers the chain, the new first event's prevHash no longer matches the pinned value — and there's no way to forge a match, because the attacker can't reach into the operator's out-of-band record to change it. That's also why it holds on an unsigned chain: the protection comes from the anchor being external, not from a key.

Why not a self-describing anchor?

The obvious instinct is to make the file carry its own anchor — stamp the "true" genesis hash into a header, or sign a manifest of the chain's own metadata. That fails for a reason worth stating plainly: the only forgery a file can reproduce from itself is one an in-file anchor also validates. If the attacker can rewrite the chain, the attacker can rewrite any anchor stored alongside it. A self-attested anchor gets forged in the same edit as everything it was supposed to protect. A signed in-file manifest is only better if the signing key lives somewhere the attacker can't reach — at which point the key is the out-of-band secret, and you've relocated the external-fact requirement, not removed it. The anchor has to be a fact the attacker cannot reproduce — and on an unsigned chain the only such fact is one that was never in the file to begin with.

The tradeoff is honest and belongs in the docs: the anchors are only as strong as the operator's out-of-band capture. A key rotation that forgets to record the new head hash loses the cross-file link. Anchors are opt-in — absent, verification is byte-identical to before. You buy real protection with real operational discipline, and the system says so out loud rather than pretending the file can bootstrap its own trust.

Disguise 5 — a thread-scoped rule silently covered every thread

matchSubsetOrEqual() — the subset check behind both detectShadowing() and checkMonotonicity() — compared tool, channel, actor, pathPrefix, and argEquals. It never compared thread_ts. So a rule scoped to one Slack thread was treated as a rule covering every thread.

// before: thread_ts absent from the comparison ⇒ "matches all threads"
function matchSubsetOrEqual(a: Rule, b: Rule): boolean {
  return sameTool(a, b) && sameChannel(a, b)
      && sameActor(a, b) && subsetPath(a, b) && subsetArgs(a, b);
  // thread_ts never checked
}

The absent field defaulted to "matches all" when the safe default is "matches none." A narrowly-scoped rule got read as maximally broad — the same failure shape as the deny that ignored /etc, one layer up in the analysis code instead of the enforcement path. The fix honors thread_ts in the subset check, and an unset scope now correctly covers nothing rather than everything.

The shape of the fail-open bug

Line them up and it's one bug five times. Empty args → skip the deny. Empty log → verified clean. Missing property → truthy prototype value (plus a thrown exception the fix itself almost added). Sheared chain → recomputes valid. Absent thread scope → covers all threads. Every one defaults to allow / pass / clean / covers-all when the discriminating data is missing. The permissive branch is the one that runs when there's nothing to run against.

The fix never changes. Make absence fail closed. A gate that can't see its input escalates to a human. A verifier with too few events fails. An accessor with no property returns undefined and doesn't throw. A chain proves itself against a fact it can't forge. A scope with no value covers nothing. Same principle, five layers. The one honest exception is the sheared chain: when the discriminating fact can only live outside the file, fail-closed becomes something the operator opts into by capturing that fact out-of-band — and the system says so plainly instead of pretending a file can bootstrap its own trust.

None of these surfaced from writing new code. They surfaced from adversarial review pointed at the load-bearing core: a hardening review of the signed journal, two cold-contributor reviewers who reproduced the /etc/shadow walk-past, a Gemini audit that caught the crash-on-absent-channels the fix almost introduced. Every one was proven revert-to-red — revert the guard and a test goes red, so the guard is doing work no other line does. The suite sits at ~1304 tests, ~96.6% line / 98.2% function coverage, nine gates green.

Disguise	What was absent	Permissive default	Fail-closed fix
Empty args	Structured tool input to scope on	Deny rule skipped; `auto_approve` wins	Escalate to a human when a scoped rule lacks its input
Wiped audit log	Any events in the file	Empty log verifies clean	Fail below a `--min-events` floor
Missing property	An own property on the config	Prototype key returns a truthy function	`Object.hasOwn` chokepoint; return `undefined`, never throw
Sheared chain	An out-of-band genesis fact	Renumbered chain recomputes valid	Pin the genesis hash externally; a mismatch fails
Unset thread scope	`thread_ts` in the subset check	Rule covers every thread	Compare `thread_ts`; no scope covers none

Also shipped

Alongside the fail-open work, a set of transport-contract fixes in the same spirit: start the MCP stdio transport before Socket Mode so the protocol handshake never blocks on Slack; route all structured logs to stderr because stdout is the MCP channel and one stray log line corrupts the protocol; and extract the socket-start and manifest-identity guards into testable seams. A decision-record spike (ADR-004) settled a build-vs-keep question: this system's job is to prove what an agent did — offline, self-hosted, deterministic per tool call — so the signed journal is what matters, not prettier approval cards.

That's the throughline for all of it. Guardrails for an AI agent are not features you add on top — they're the property that lets the agent hold a tool at all. An agent you can trust with Bash is one whose deny rule fires even when the caller forgot to pass the arguments. Fail-closed discipline is what earns the trust; empty is not clean.

Every Safety Gate Has a Failure Direction — the underlying principle: every gate fails in some direction, and you have to choose it on purpose.
Adversarial Review Before Team Rollout — all five of these bugs came from pointing hostile reviewers at the core before anyone depended on it.
When a Relevance Score Broke the Cite-or-Refuse Gate — another fail-open gate, another case of absence quietly reading as permission.

The Kernel Must Not Import Its Agents

Jeremy Longshore — Tue, 14 Jul 2026 10:23:16 +0000

There is a moment in the life of every framework where an application-shaped thing has grown up inside it. It started as a demo, a reference implementation, a "let's just prove the primitives work" spike. Then it kept running. Now it has cron entries, an operator surface, a test pack, and traceability rows — and it lives in the same repository as the primitives it was only ever meant to exercise.

On 2026-07-12 we did the extraction. We pulled a stateful watcher agent out of a governance kernel and gave it its own leaf repository. The interesting part is not that we moved files. It is the single rule that made the move safe, and that made the resulting kernel worth trusting:

The kernel must not import its agents. When you split an agent out of its governance kernel, the dependency must point from the agent to the kernel — never the reverse.

This is the agent-framework restatement of a much older law: the framework must not depend on the application, and the composition root lives in the leaf. Everything below is one concrete execution of that law.

In practice the agent repo imports the kernel's published contracts as a pinned dependency, composing governance primitives like the trigger interface and policy engine. The kernel has zero knowledge of the agent's existence — no import, no reference, no coupling. That one-way edge is what keeps the kernel small, independently verifiable, and trustworthy as a standalone governance system.

The two systems

agent-governance-plane (AGP) is the governance kernel. It owns the durable governance primitives and nothing else: a policy engine, a Docker sandbox for tool execution, Slack human-in-the-loop (HITL) approval, and an Ed25519-signed, hash-chained audit journal. Its doctrine line is worth memorizing because it dictates the whole shape:

The model proposes; the deterministic system decides and records.

Every autonomous tool call passes through the same governed path:

trigger → policy gate → (if risky) human approval → sandboxed exec → signed journal

bob-the-intendant is the new leaf product repo. It composes AGP as a pinned dependency and owns the agent/composition layer. Its product category is "governed background agents" — long-lived processes that wake on a trigger and want to do something a human might not want them to do unsupervised. AGP is what keeps them honest. Bob is what makes them useful.

Before the extraction, the watcher lived inside AGP. After it, AGP has zero knowledge that the watcher exists, and Bob owns it outright.

The invariant we were protecting

The boundary has two halves, and they are not the same rule. The headline half is the thesis: the kernel imports nothing from the leaf — it does not know the agents that compose it exist. The mirror half constrains the leaf: the leaf may import only the kernel's frozen contracts, never the daemon's internals — where "the daemon" is the kernel's live governance loop (runMediated()) and the machinery around it. The composed agent composes the contracts the kernel publishes; it never reaches past them into implementation.

Put together, the two halves define one edge: it runs from agent (leaf) to kernel, touches only the published surface, and never runs the other way. A governance kernel that imports the agents it governs breaks the acyclic-dependency rule — it is a governance kernel you cannot reason about. You can no longer point at it and say "this small thing decides and records; here is its trust boundary" — because its trust boundary now includes every agent that ever leaked into it.

So the extraction had two acceptance tests, both structural:

After extraction, AGP has zero knowledge of the watcher. No import, no CLI command, no template, no traceability row.
After extraction, Bob has zero references to removed AGP modules. It composes contracts that still exist; it points at nothing that was deleted.

What moved out, what stayed in

The discipline of the move is entirely in where the cut runs. The cut runs exactly along the contract boundary — the bounded context between kernel and leaf: agent-specific composition leaves, primitives stay.

Moved out of the kernel and into the leaf:

src/triggers/github-watcher/     # the trigger-woken GitHub watcher agent
src/cli/commands/watch.ts        # `agp watch` — the operator surface a human runs
templates/github-watcher/        # the per-agent template test packs
RTM REQ-043 / 045 / 046 / 047 / 048 / 049   # this agent's traceability rows
Journey J2 + the "run-governed-watch" persona flow

Stayed in the kernel, because these are the primitives the agent composes, not the agent itself:

trigger-source contract (RTM REQ-050)   # the frozen interface the agent fires through
daemon runMediated() loop (REQ-042)     # the governance loop itself
journal cross-chain causal-pointer (REQ-044)   # a journal primitive

Notice the requirements-traceability discipline here, because it is the part most teams skip. REQ-042 and REQ-044 were re-anchored to the retained code — the rows didn't get deleted, they got re-pointed at the primitives that stayed. The agent's own rows (REQ-043/045/046/047/048/049) followed the agent into the leaf's docs. Traceability is a graph, and when you cut the code graph you have to cut the requirements graph along the same line or your "signed audit of everything" claim quietly develops holes.

The dependency edge

This is the entire point of the exercise, in one picture:

        ┌───────────────────────────────────────────────┐
        │  bob-the-intendant  (LEAF — the application)   │
        │                                                │
        │   • github-watcher agent                       │
        │   • `bob watch` operator CLI                   │
        │   • template test packs                        │
        │   • J2 / run-governed-watch persona flow       │
        │                                                │
        │        imports  agp/src/... (contracts only)   │
        └───────────────────────┬───────────────────────┘
                                 │   pinned dependency
                                 │   ONE WAY — leaf → kernel
                                 ▼
        ┌───────────────────────────────────────────────┐
        │  agent-governance-plane  (KERNEL — frozen)     │
        │                                                │
        │   • trigger-source contract   (REQ-050)        │
        │   • runMediated() gov. loop   (REQ-042)        │
        │   • policy engine · Docker sandbox             │
        │   • Slack HITL · Ed25519 hash-chained journal  │
        │                                                │
        │        knows NOTHING about the watcher         │
        └───────────────────────────────────────────────┘

Bob consumes AGP as a pinned dependency. The watcher's imports were rewired from in-tree relative paths to agp/src/.... There is no arrow back up. The kernel cannot see the leaf, cannot import the leaf, and does not know the leaf exists. That is not an accident of the current file layout — it is the property we spent the day buying.

Three tradeoffs, and why each went the way it did

Every real extraction is a sequence of forks in the road. Three of them carried the design.

1. Clean removal + a cross-repo pinned-dependency edge — over keeping the watcher as a flag-gated in-tree spike. The tempting cheap move is to leave the watcher in AGP behind a feature flag and call it "disabled." We chose the harder cut. A flag-gated agent left in-tree keeps the boundary muddy: the code is still there, still importable, still one careless import away from the kernel depending on its own agent. The flag protects runtime behavior; it does nothing for the dependency graph. We wanted the kernel to physically shrink to contracts, so that the invariant is enforced by absence, not by a boolean.

2. Composing AGP as a pinned dependency — over vendoring or forking the kernel. The frozen contracts and the trigger-source interface must be authored in exactly one place: AGP. If the leaf vendored or forked the kernel, those contracts would exist in two copies, and two copies of a contract drift — slowly, silently, and always at the worst time. Importing a pinned version means the leaf reads the contracts, never owns them. One author, one source of truth, one thing to version.

3. Owning the watcher locally in the leaf — over importing agp/src/triggers. This is the mirror image of tradeoff 2, and it is what makes the cut symmetric. After extraction the kernel no longer has the watcher modules, so the leaf cannot import them from AGP — they aren't there anymore. The leaf must own the watcher outright to build against the post-extraction kernel. Contracts: imported. Agent: owned. That split is the boundary.

Why it was a breaking change on both sides

Extraction that touches a published surface is a breaking change by definition, and honest versioning has to say so.

On the kernel: agp watch was removed from the AGP CLI. BREAKING CHANGE — the governed watcher now ships in the leaf. Anyone who typed agp watch has to move.
On the leaf: the CLI was renamed intendants → bob; the npm package was renamed @intentsolutions/intendants → @intentsolutions/bob-the-intendant. BREAKING CHANGE on both the binary name and the package name.

You cannot smuggle a change like this through as a patch. The whole value of the extraction is that consumers can reason about the kernel independently — and that reasoning depends on the kernel's version number telling the truth.

The part everyone underestimates: it was a live migration

Here is the detail worth teaching, because it is the one that bites. The watcher was not a dormant module. It was running in production on a cron: ~/bin/intendants-release-watch.sh, firing on a schedule, in a governed loop, on a real repo.

Extracting a running component is a live-migration problem, not a code move. If you merge the kernel PR that removes agp watch while production cron still calls agp watch, you have — for however long it takes to notice — a production job pointed at a command that no longer exists.

So the order was inverted deliberately:

# 1. Repoint the production cron to the LEAF's command FIRST,
#    while `agp watch` still exists in the merged kernel.
#    ~/bin/intendants-release-watch.sh  →  `bob watch ...`

# 2. THEN merge the kernel PR that removes `agp watch`.

At no instant was production pointed at a command that didn't exist. The cutover of the running process precedes the removal of the thing it used to call. This is the same expand/contract pattern database migrations use — you add the new path, move traffic, and only then remove the old path — applied to a CLI command instead of a column.

The releases were coordinated to match:

AGP cut v0.1.100 — the post-extraction clean kernel.
The leaf cut v0.0.1, then v0.0.2.
During the work, the leaf pinned the post-extraction commit of AGP; once the kernel PR merged, it repinned to the v0.1.100 release tag. Pin to a commit while the edge is in flight; pin to a tag once the edge is real.

Verification: link the evidence, don't say "verified"

The whole claim of a governance kernel is auditability, so the extraction has to be auditable too. Both sides landed green.

Kernel branch (post-extraction clean AGP):

bun run typecheck      0 errors
bun test               360 pass / 8 skip / 0 fail
coverage (lines)       94.46%   (gate ≥ 90)
coverage (funcs)       91.80%   (gate ≥ 88)
lint                   clean
claim-scan             PASS
doc-drift              PASS
audit-harness verify   OK
escape-scan            REFUSE=0 / CHALLENGE=0 / FLAG=0

Leaf, built against the post-extraction kernel:

bun run typecheck      0 errors
bun test               52 pass / 0 fail  (10 files)
references to removed AGP modules:   ZERO
  (no `agp/src/triggers`, no `agp/src/cli/commands/watch`)

That last line is the acceptance test made mechanical. "Bob has zero references to removed AGP modules" is not a vibe — it is a grep that returns nothing. The kernel's side has the mirror check: nothing in AGP mentions the watcher. Two structural assertions, both green, and the invariant is proven rather than asserted.

The generalizable principle

Strip away the specific repos and here is what is left, for anyone building an agent framework, a policy layer, or any governance kernel that other things compose.

If your governance kernel imports the agents it governs, three things break at once:

You cannot ship or version the kernel independently. Its releases are hostage to every agent tangled into it; a change in one agent forces a kernel release, and the kernel's version number stops meaning anything.
You cannot reason about the kernel's trust boundary in isolation. "Here is the small deterministic thing that decides and records" is only a true sentence if the small thing has a fixed edge. Import an agent and the boundary now wraps that agent's code too.
Every new agent widens the kernel's attack and verification surface. The set of things you must audit to trust the kernel grows with every agent you add — which is exactly backwards, because the kernel is supposed to be the fixed point you audit once. The watcher we just extracted was one such agent; every agent we don't extract would fold exactly this cost back into the kernel, which is why the extraction is a repeatable discipline and not a one-off cleanup.

Extracting agents into leaves keeps the kernel a small, independently-verifiable dependency with a stable trust boundary. That is not architectural neatness for its own sake — it is the entire reason to have a governance kernel in the first place. The one-way edge, agent → kernel, is what makes "a signed audit log of every tool call" a claim you can verify by looking at the kernel alone, without first reading every agent that happens to use it. The direction of a single dependency arrow is the difference between a trust boundary and a wish.

Making a Fire-and-Forget Writer Safe Under Failure

Jeremy Longshore — Mon, 13 Jul 2026 10:22:33 +0000

There is a hook that distills a finished Claude Code session into a few durable learnings and proposes them to a shared team brain. It runs on SessionEnd. Nobody watches it. That last sentence is the entire problem.

The manual counterpart — a /brain-save command a teammate types when they want to keep something — has been safe for months. Not because the write path was safe, but because a human was standing over it. A human notices when a capture fails. A human doesn't hit save twice in a panic. A human sees the error toast and shrugs. The forgiving hands were attached to a person, and the person was the safety system.

An unattended hook has no hands. It fires when the session ends, whether or not the network is up, whether or not the same session already fired, whether or not one teammate's token has gone berserk. So the hook could not ship until the write path underneath it earned five properties. Skip any one and "convenient automation" becomes silent data loss, a duplicate flood, or governance drift nobody can see. Here is each property, why the obvious version of it was wrong, and the fifth one that governs the on-switch itself.

The five properties

Before an unattended capture hook is safe to turn on, its write path needs all five:

Idempotent intake — retries collapse into a single record, never a duplicate flood.
Durable outbox — a network blip queues the write instead of losing it.
Per-actor backpressure — one runaway identity can't flood the queue.
Atomic governance receipts — a state flip and its audit record commit together or not at all.
Consent-gated opt-in — the on-switch stays a deliberate human decision.

Property 1 — Idempotent intake

A retry must not fan out into duplicates. This has two halves, because the client and the server can each betray you.

The client half was the id. Team mode was minting candidate ids with randomUUID(). A re-send — a flaky network retry, a hook that fires twice, a drained backlog item going back out — arrived as a brand-new row every time. The fix is to make the id a pure function of the content:

import { createHash } from "node:crypto";

// UUIDv5 over (tenant, title, content) — same input, same id, forever.
function deriveCandidateId(tenant: string, title: string, content: string): string {
  const NS = "6ba7b810-9dad-11d1-80b4-00c04fd430c8"; // DNS namespace
  const hash = createHash("sha1")
    .update(Buffer.from(NS.replace(/-/g, ""), "hex"))
    .update(`${tenant}\n${title}\n${content}`)
    .digest();
  hash[6] = (hash[6] & 0x0f) | 0x50; // version 5
  hash[8] = (hash[8] & 0x3f) | 0x80; // RFC 4122 variant
  const h = hash.toString("hex");
  return `${h.slice(0,8)}-${h.slice(8,12)}-${h.slice(12,16)}-${h.slice(16,20)}-${h.slice(20,32)}`;
}

Deriving from node:crypto SHA-1 keeps team mode dependency-free — no uuid package pulled in for one function. The handler was extracted into an exported capture() alongside deriveCandidateId, outboxDir, and drainOutbox, so the whole path is unit-testable instead of buried in a request handler.

The server half is the guard behind the id. CandidateService.intake now calls findByContentHashAndTenant(contentHash, tenantId) before it inserts. On a hit it returns the existing candidate and skips writing a second proposed receipt, instead of inserting a duplicate row. The id and the server's content hash are two layers, not one key: the UUIDv5 guarantees a retry re-sends the byte-identical row, while the tenant-scoped content-hash lookup is the server's own backstop, collapsing same-content proposals within a tenant even if two ids ever diverged. The id dedups retries; the hash dedups content.

Why not reuse the repo's existing findByContentHash? Because that one is cross-tenant — WHERE content_hash = ? LIMIT 1, no tenant clause. Dedup against it and tenant A's re-send matches tenant B's identical row, leaking cross-tenant state and deduping to the wrong record. The new lookup is tenant-scoped on purpose. The behavior change is intended and narrow: a same-content re-send within a tenant returns the existing candidate (201); identical content across two tenants stays two independent rows. The old test that asserted "POST accepts duplicate candidates" was rewritten to assert the new idempotent contract. store 224 and api 323 tests pass.

Property 2 — A durable outbox

A network blip must not silently lose the write. Team mode had no spool fallback. Local mode already wrote to a spool via writeToSpool; team mode, under an unattended hook, would throw on a dropped connection and the capture would evaporate. That was the one true "captures lost" blocker — a hard prerequisite for turning the hook on.

Now a fetch throw or a 5xx queues the candidate to a flat ~/.teamkb-outbox/ directory (mode 700, override with TEAMKB_OUTBOX_DIR) and reports queued rather than an error. The next successful capture drains the backlog:

async function capture(candidate) {
  try {
    const res = await postCandidate(candidate);
    if (res.status >= 500) return queue(candidate, "server-5xx");
    if (!res.ok) throw new HttpError(res.status);   // 4xx: real rejection, surface it
    await drainOutbox();                             // success → flush the backlog
    return res.json();
  } catch (err) {
    if (err instanceof HttpError && err.status < 500) throw err; // never queue a 4xx
    return queue(candidate, "network");             // throw / 5xx → durable spool
  }
}

The direction of the split is the whole design. A 5xx or a thrown socket is transient — keep it and retry. A 4xx is a real validation, auth, or disclosure rejection — queuing it would loop forever against a request that will never be accepted, so it is surfaced immediately. Drop-on-4xx, keep-on-5xx. A permanently-invalid item can never wedge the outbox, and a transient outage never loses a capture.

The two properties compose because of the id. Since the queued row carries its UUIDv5, the drain re-sends the exact same row — same id — and Property 1's server guard collapses it back to one candidate. The outbox can retry as aggressively as it likes and never manufactures a near-duplicate. Six new tests in remote-server.test.ts pin it down: id determinism and format, same-id-on-retry, outbox-on-throw, queue-on-5xx-not-4xx, drain-on-next-success, drain-stops-while-down. Full suite: 69 pass.

Property 3 — Per-actor backpressure

One runaway identity must not flood the queue. A new capture-quota middleware — an onRequest hook registered after auth on the intake path — caps how many candidates a single token identity (request.actor) may POST to /api/candidates per window. Default 60 per 60s, tunable via captureQuotaMax and captureQuotaWindowMs. Intake only; promote and reject stay admin-gated and uncapped; reads are untouched.

The obvious answer is "there's already a rate limiter, use that." The existing limiter is per-source-IP, and on a tailnet that is worthless. Every teammate device is its own 100.x address, so a per-IP cap gives every token effectively its own private quota — there was no real per-token limit at all. One compromised token, one runaway loop, or one over-eager auto-capture hook could bury the inbox, and the only tripwire was a DB-size canary that doesn't warn until 200 rows.

app.addHook("onRequest", async (req, reply) => {
  if (req.method !== "POST" || req.url !== "/api/candidates") return;
  const key = req.actor;                         // follows the TOKEN, not the IP
  const now = Date.now();
  const bucket = quota.get(key)?.filter(t => now - t < windowMs) ?? [];
  if (bucket.length >= max) return reply.code(429).send({ error: "capture quota" });
  bucket.push(now);
  quota.set(key, bucket);
});

Making the limit follow the token instead of the network address is the fix. The test proves the (N+1)th intake gets a 429 while reads and other paths never touch the counter.

Property 4 — Atomic governance receipts

A state flip and its audit record must commit together or not at all. The nightly auto-govern sweep writes a single batch governed receipt covering the candidates it processes. Some of those flips — quarantine and duplicate — have no individual receipt; the batch event is the only on-chain record they exist. And they were autocommitting inside the loop, while the batch receipt got inserted afterward in a try/catch that logged to stderr on failure.

Walk that failure through. The loop flips a candidate to quarantine and commits. The batch receipt insert throws. The catch logs a line to stderr. Now the candidate is out of the inbox with no on-chain record of why it left. That is chain-invisible governance — silent drift, discoverable only by noticing something is missing.

The fix defers the receipt-less flips and applies them in one non-swallowing transaction with the receipt insert:

const deferred = [];
for (const c of batch) {
  if (c.decision === "promoted") {
    applyPromotion(c);            // already carries its own transactional receipt + memory
  } else {
    deferred.push(c);             // quarantine / duplicate — no individual receipt
  }
}
db.transaction(() => {
  insertGovernedReceipt(batchEvent);          // the ONLY record of the deferred flips
  for (const c of deferred) applyFlip(c);      // commit them WITH the receipt
})();                                          // throws → everything rolls back, error propagates

promoted stays in-loop deliberately, because a promoted row already writes its own transactional promoted receipt and curated memory — it is never chain-invisible. On a receipt failure the quarantine and duplicate flips roll back, those candidates stay inbox and get retried idempotently on the next run, the promoted rows persist with their receipts, and the error propagates so runGovern surfaces it and the nightly wrapper's fail-loud path alerts. Atomic rollback over best-effort retry: a governance action you can't record is a governance action you don't take.

Property 5 — Consent-gated opt-in

Four properties made the write path safe. The fifth governs the switch, and it is the one thing you cannot automate.

The hook (hooks/session-end-capture.mjs) is the automatic twin of manual /brain-save. It carries three hard guards, and any unmet guard is a silent no-op: an opt-in marker file ~/.teamkb/autocapture.enabled, team mode configured, and not a recursive child. It runs the distiller detached — it never blocks or fails the session — and the distiller holds a member token, so it can only propose to the inbox, never write durable state.

if (!fs.existsSync(markerPath())) return;        // (a) opt-in marker absent → no-op
if (!teamModeConfigured()) return;               // (b) team mode off → no-op
if (process.env.TEAMKB_RECURSIVE) return;        // (c) recursive child → no-op
spawnDetached(distiller, { env: memberTokenEnv() }); // detached, member-scoped, never blocks

Why isn't this just a plugin.json hooks entry? Because a plugin.json-declared hook auto-installs the instant the plugin installs. That is exactly the silent push that was forbidden — a teammate updates the plugin and their sessions start shipping distilled learnings to a shared brain they never agreed to feed. The constraint that shaped the design was blunt: "I don't want them to ship it and be like wtf." So the hook is deliberately not plugin-declared. Installing the plugin enables nothing.

The only way to turn it on is a consent-gated enable script (hooks/enable-autocapture.mjs) that prints the full disclosure — what it does, what it does not do, how to pause, how to purge, and why — then requires typing I CONSENT. Only then does it register the hook in the teammate's own ~/.claude/settings.json (backing the file up first) and write the marker. --off pauses, --off --purge deletes local logs, --status shows state.

Consent is the property you cannot make idempotent or durable. It has to be a deliberate human act, performed once, by the person whose sessions will be read. Even then the hook stays a silent no-op unless all three guards pass — and the distiller it spawns only does real work on a real transcript, so a hook that fires on an empty session burns nothing. A 5/5 hook-guard smoke test pins that whole no-op envelope against a stub claude on PATH, with zero model invocation. Verification across the change: node --check on both scripts, typecheck + lint + build exit 0, and remote-server.test.ts's 69 tests green — with store 224 / api 323 still clean.

The boring bugs that would have eaten the captures anyway

Two footguns fixed the same day, both the kind that lose data quietly:

Default DB path drift. The API defaulted to ~/.teamkb/data/teamkb.db while the plugin and brain used ~/.teamkb/teamkb.db. A no-override deploy would write every teammate capture to a database the govern sweep never reads. Aligned the default via resolveTeamKbPath('teamkb.db') — chosen over assert-and-warn, because a warning still loses the capture. The live deploy already overrides TEAMKB_DB_PATH, so it was never hit; the next fresh deploy would have been.
No-op anchor skip. appendAnchor now returns the last record unchanged when both the chain head and the row count match the last anchor. A nightly run that changed nothing was still appending to the tamper-evidence log and to git history for zero evidentiary gain. A real new head still anchors — the never-rehash-the-chain rule holds.

Also shipped

The same "make an unattended watcher safe to cron" theme ran through the secondary repos. agent-governance-plane got a release-watcher meaningfulness filter (draft and prerelease releases dropped by default, includePrereleases to opt in — the skip runs before tag validation, so a tagless GitHub draft is skipped rather than crashing the watcher), a notify-mode one-way webhook that needs no human in the loop, and a one-command installer plus an extraction-readiness ADR, with three irreversible doors — public flip, npm publish, external reply — kept explicitly human-gated. intentsolutions-vps-runbook and intent-os got a torn-datadir backup fix: consistent DB dumps plus a restore-verification drill, correcting an over-optimistic single-host restore claim.

The pattern behind safe unattended writes

A best-effort, fire-and-forget writer is a distributed system wearing a convenience costume. The moment the human leaves the loop, every property the human was silently providing has to be built into the write path: dedup so retries collapse, a spool so blips don't vanish, a per-actor cap so one identity can't flood, atomic receipts so governance can't drift invisibly, and a consent gate so the on-switch stays a human decision. Four of those you can engineer. The fifth you have to refuse to engineer.

Adversarial Review: The Six Lenses That Halted a Rollout — the governance review that gated this same team-brain rollout.
Every Safety Gate Has a Failure Direction — why drop-on-4xx / keep-on-5xx is a direction choice, not a detail.
Liveness Without Health Is Theater — the sibling lesson for unattended systems: running is not the same as working.

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Making a Fire-and-Forget Writer Safe Under Failure",
"description": "Five properties an unattended capture hook needs before you turn it on: idempotent intake, durable outbox, backpressure, atomic receipts, consent gate.",
"author": {
"@type": "Person",
"name": "Jeremy Longshore"
},
"publisher": {
"@type": "Organization",
"name": "Start AI Tools"
},
"datePublished": "2026-07-11T10:00:00-05:00",
"dateModified": "2026-07-11T10:00:00-05:00",
"url": "https://startaitools.com/posts/making-fire-and-forget-capture-safe-under-failure/",
"keywords": ["architecture", "idempotency", "ai-agents", "claude-code", "distributed-systems"],
"articleSection": "Technical Deep-Dive"
}

Liveness Without Health Is Theater

Jeremy Longshore — Mon, 13 Jul 2026 10:21:03 +0000

Every team that runs scheduled work eventually ships the same bug, and it does
not look like a bug. A cron job, a nightly workflow, a backup timer — something
that is supposed to run on a clock and quietly do useful work. To keep an eye on
it, someone adds a heartbeat: each time the job runs, it touches a file, pings a
status endpoint, or emits a "still alive" event. A sweep watches those beats and
pages when one goes stale. The dashboard is green. Everyone moves on.

Then one morning the useful work has not happened for three weeks, and the
dashboard is still green.

The monitor that lies

Liveness proves a scheduled job ran. Health proves it succeeded. A
liveness-only heartbeat fires on every invocation regardless of outcome, so it
hides the one failure that matters most: a job that runs on schedule and
silently does nothing useful. Two independent markers restore the missing
distinction — is the system stopped, failing, or actually working?

The failure is structural, not incidental. A heartbeat that fires on every
invocation — before the work runs, or regardless of how the work exits —
collapses two very different facts into one signal:

The job ran (the scheduler fired, the process started).
The job succeeded (the work it exists to do actually happened).

A liveness-only heartbeat attests only to the first. So the single worst failure
mode of any scheduled system — a job that runs forever while silently doing
nothing useful — is precisely the mode it is blind to: the process starts on
schedule, the beat fires like clockwork, and inside it the real work is throwing,
short-circuiting, or writing to the wrong place. Liveness is perfect, health is
zero, and the monitor reports the metric it can measure while staying silent on
the one that matters.

This is not a monitoring gap. A gap implies something is unwatched. This is
worse: the thing is watched, the watch is green, and the green is a lie. That is
theater — the appearance of oversight standing in for the substance of it.

Why heartbeats lie

The flaw is in where the beat is written. Touch the heartbeat first and
unconditionally — touch job.heartbeat; do_the_work, with nobody checking $? —
and the beat is decoupled from the outcome. The exit code, the one piece of
information that distinguishes "worked" from "ran," is thrown away. To the sweep,
a job that succeeds and a job that fails emit the identical signal. Writing the
beat in a finally block, or last thing before exit, has the same effect.

The fix is not a better heartbeat. It is a second marker.

Two markers, not one

Separate the two facts into two files. Emit a liveness beat on every
invocation. Write a health mark only when the work exits clean.

#!/bin/sh
# run-job.sh — wrap any scheduled job with a two-marker protocol.
STATE_DIR="${STATE_DIR:-/var/lib/job-markers}"
job="$1"; shift

beat="$STATE_DIR/$job.beat"
ok="$STATE_DIR/$job.ok"

# Liveness: fires on EVERY run, before we know the outcome.
date -u +%s > "$beat"

# Do the actual work and capture its verdict.
"$@"; rc=$?

# Health: refreshed ONLY when the work truly succeeded.
if [ "$rc" -eq 0 ]; then
  date -u +%s > "$ok"
fi

exit "$rc"

The sweep no longer reads a single file — it reads the pair, and the
relationship between the two ages is the diagnosis:

# For each job, the two ages TOGETHER are the verdict (stale = older than one hour).
beat_age=$(( now - $(cat "$job.beat") ))
ok_age=$(( now - $(cat "$job.ok" 2>/dev/null || echo 0) ))

if   [ "$beat_age" -gt "$stale" ]; then verdict=STOPPED   # not running at all
elif [ "$ok_age"   -gt "$stale" ]; then verdict=FAILING   # running, not succeeding
else                                    verdict=HEALTHY    # ran AND worked
fi

The whole point lives in that state table:

Beat marker	Health marker	Verdict	Meaning
stale	(any)	STOPPED	The scheduler is not firing the job at all.
fresh	stale	FAILING	The job runs on schedule but keeps exiting non-zero.
fresh	fresh	HEALTHY	It ran and it worked.

The middle row is the whole point: fresh beat + stale ok is the silent failure
made loud — a job running forever doing nothing, now a first-class alert. The
same signal-pair instinct shows up wherever automation has to police itself: a
circuit breaker for a runaway bot loop
trips on the gap between "still active" and "still making progress."

The seeding gotcha

One rollout trap: the day the protocol ships, every job has a beat file but no
.ok yet, so the first sweep reads every health age as infinite and flags
everything FAILING — a wall of false alarms that trains everyone to ignore the
new alert on day one. Seed each health marker from the beat that already exists
([ -f "$ok" ] || cp "$beat" "$ok"), on the assumption that currently-live jobs
are healthy until proven otherwise.

Watch the watchers

Two markers fix the job. But the sweep, the alert path, and the dashboard are
themselves software — usually written once and never touched again. An unwatched
watcher is not oversight; it is a more expensive blind spot. Meta-monitoring —
observability turned on the watchers themselves — means treating every layer of
the stack as something that can also fail silently.

Canary the alert path

The most dangerous single point of failure is the notification channel, because
its failure is invisible by construction: a broken alert path produces no
alerts, which is indistinguishable from nothing wrong. The only way to know a
page can get out is to send one and require proof it arrived. Probe the
credentials, POST to the chat webhook, and demand a real 200:

#!/bin/sh
# alert-canary.sh — prove a page can actually get out.
# Run this UNDER run-job.sh, so its own success/failure is two-marked.
: "${WEBHOOK_URL:?alert webhook not configured}"

code=$(curl -s -o /dev/null -w '%{http_code}' \
  -X POST "$WEBHOOK_URL" \
  -H 'Content-Type: application/json' \
  --data '{"text":"alert-path canary — ignore"}')

if [ "$code" != "200" ]; then
  echo "alert path DOWN (HTTP $code)" >&2
  exit 1   # non-zero → run-job.sh leaves .ok stale → the sweep sees a dead path
fi

The recursion is deliberate. The canary is wrapped by the same two-marker
runner, so a failed canary refuses to refresh its own health mark, and the sweep
that watches everything else now watches the watcher. On platforms with a service
manager, back it with a native failure hook so a crash that never reaches the
canary still escalates:

# backup.service.d/onfailure.conf — escalate straight from the service manager.
[Unit]
# alert@%n.service is a templated oneshot unit that runs the notifier.
# NOTE: systemd has no inline comments — everything after '=' is the value,
# so this explanation must live on its own line, never trailing the directive.
OnFailure=alert@%n.service

Reconcile the registry

A sweep can only watch jobs it knows about, and the list of "everything that
should be running" is itself state that drifts as jobs get added, renamed, and
retired. A reconcile job closes the gap: diff the live inventory against the
declared registry and report orphans both ways — things running that nobody
registered, and things registered that no longer run.

A healthy first run is ugly. One such diff, run against a live fleet, surfaced
61 orphaned jobs among 138 live automations — things executing for months with
no registry entry, watched by nothing. That is the normal starting state, not an anomaly;
driving it to zero is what makes "the sweep watches everything" true instead of
assumed. The same instinct grows the sweep over time — one system's check count
climbed from 19 to 32 as each new class of silent failure earned its own
assertion.

Assert log freshness

A status dashboard that cats a log and renders it is another liveness-only lie:
it proves the log exists, not that it is current. A pipeline can wedge at
2 a.m. and the dashboard will display yesterday's final line forever. Assert the
log's freshness, not merely its contents:

# A dashboard must prove the log is CURRENT, not just that it has bytes.
log=/var/log/pipeline.log
max_age=900   # 15 minutes

age=$(( $(date -u +%s) - $(stat -c %Y "$log") ))
if [ "$age" -gt "$max_age" ]; then
  echo "STALE: newest log line is ${age}s old — the dashboard is quoting a corpse"
  exit 1
fi

Two root causes worth naming

Two independent production incidents made the abstract argument concrete, and
both traced to causes that generalize well beyond any one system.

Deploy-by-reference into mutable working trees

Both incidents shared one mechanism: a cron job executed whatever branch a
shared git working tree happened to be parked on — a bare symlink into a
directory that humans and other automation also checked out and reset. In one, a
customer-facing uptime monitor ran an old, unarmed version of itself and posted
false positives every day, because the tree sat on a stale branch — and it
exited zero the whole time, so both a liveness beat and an exit-code health
mark would have called it perfectly healthy while it lied to users. In another, a
static-site publishing pipeline only appeared to run nightly — its lone
"heartbeat" was the tree being parked on a feature branch overnight; the run it
seemed to complete never happened. A missed run that was never missed, because
the signal proving it ran was an artifact of the deploy mechanism, not the work.

The rule that falls out is blunt: crons run deployed copies or
verified-main paths — never a bare symlink into a working tree. A working tree
is mutable by definition; anything that reads code from one has no idea what code
it is about to run.

Inherited file descriptors are state

The second cause is subtler and produced weeks of silent underperformance with
zero errors. A push job serialized itself with flock on a lock file held open
as a file descriptor, then spawned a long-lived daemon that inherited that
descriptor — and with it the lock, held for the daemon's entire multi-hour
lifetime. A job scheduled every 20 minutes was throttled to roughly once every
two hours — a sixfold collapse in cadence — silently, for weeks. Nothing failed:
every blocked run exited cleanly on "another run holds the lock." The fix is one
character of intent — close the fd on the child:

#!/bin/sh
# push-cron.sh — serialize with flock, but DON'T leak the lock to children.
exec 9>/var/lock/push.lock
flock -n 9 || { echo "another run holds the lock; exiting" >&2; exit 0; }

# The daemon must NOT inherit fd 9, or it holds the lock for its whole
# lifetime and throttles every future run of this cron.
long_lived_daemon 9>&-

The lesson is the thesis restated from a different angle: silence is not
success. A job that exits zero on a lock it should never have been blocked by
looks identical, to a liveness monitor, to a job doing perfect work — and a
health mark keyed only to the exit code has the very same blind spot here, since
every throttled run's rc was zero. Catching this one needs the third leg from
the last section: assert that the work product is fresh — that the push
actually landed — not merely that the process exited clean. It is the same
discipline behind durable replies that survive a crash:
prove the work persisted; never infer it from a clean return code.

The discipline

None of this is exotic — it is one idea applied consistently across every layer:

Monitoring is software. It has bugs, it rots, and it fails silently like any other code. Test it, canary it, and give it the same adversarial skepticism you would aim at the systems it watches.
Every layer must distinguish ran from succeeded. One marker answers "did it run?" A second, written only on a clean exit, answers "did it work?" The gap between them is where the expensive failures hide.
Monitor the monitors. Canary the alert path so a dead pager pages itself. Reconcile the registry so the sweep watches everything, not just what someone remembered to register. Assert log freshness so a dashboard cannot quote a corpse.

Do this and the word green changes meaning. It stops saying "the cron fired"
and starts saying "the work happened." That is the only definition of green
worth trusting — and the only one a heartbeat, alone, can never give you.

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Liveness Without Health Is Theater",
"description": "A heartbeat that fires on every run proves a job ran, never that it succeeded — the fix is two markers plus meta-monitoring the watchers.",
"url": "https://startaitools.com/posts/liveness-without-health-is-theater/",
"datePublished": "2026-07-10T08:00:00-05:00",
"author": { "@type": "Person", "name": "Jeremy Longshore" },
"publisher": { "@type": "Organization", "name": "Start AI Tools" },
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://startaitools.com/posts/liveness-without-health-is-theater/"
},
"articleSection": "DevOps",
"keywords": "monitoring, reliability, devops, observability, cron"
}

Adversarial Review: The Six Lenses That Halted a Rollout

Jeremy Longshore — Sun, 12 Jul 2026 10:21:51 +0000

"We shipped the safety work" is a feeling, not a fact. Before you hand a shared, governed system to a team, the only thing that converts that feeling into the truth is a structured adversarial review that verifies claims against live state — not against the design doc, and not against the diff.

On 2026-07-09 that review ran against an internal team knowledge system being prepared to open to six people all at once. It validated two judgment calls, broke three, and produced a list of eighteen real risks the plan had not named. The verdict: do not go all-at-once, and do not email anyone a token, until the first gate clears.

That halt was the correct output. This is the story of the method that produced it, the three assumptions it demolished, and the fixes that shipped the same day — every one of which enforces a single boundary: the model proposes; the deterministic system owns identity, trust, and durable state.

The system, in one paragraph

The subject is a "second brain": a governed team knowledge base. Content is captured, run through a deterministic govern pipeline, and — if it survives the gates — promoted into durable memory. Every state change writes a receipt into an append-only, hash-chained ledger, so the store is tamper-evident: you can recompute the chain and detect any row that was altered out of band. Two surfaces front it. A private, tailnet-bound HTTP API is the control plane — it holds the write-gate, the tenant guard, and the audit-actor stamp. A Claude Code plugin (an MCP server) is how people actually talk to it from their editor. Six teammates were about to get access: two admins, four members.

What shipped, and why it wasn't enough

A "safety before people" pass had already landed. It was real work, not theater: bearer tokens were hashed at rest instead of stored as plaintext; per-user tokens were minted so each teammate carried a distinct identity; an onboarding runbook was written; several hardening items were verified against the running service. On paper, the box marked "make it safe for a team" was checked.

The instinct to stop there is the trap. A safety pass tells you what you did. It does not tell you what you have — the residual risk, the assumptions you smuggled in, the interactions between a fix and the rest of the system. Those only surface when someone hostile to the plan goes and looks at the live thing. So before flipping the switch, that is exactly what happened.

The review that verified against live state

An adversarial review checks claims against the running system, not the design doc and not the diff. Independent reviewers each inspect the live service, its databases, and its deployment through a different lens — security, operations, reliability, integrity, rollout, threat model — so an assumption that is locally true but globally false has nowhere to hide.

Six independent engineer agents ran in parallel on the Fable model. Each got the same design brief and one distinct lens. The instruction that made the difference was not "review the plan." It was verify against real code and live state: read the repos, query the live databases, recompute the hash chain yourself, check org membership yourself, inspect the running service unit yourself. A review that only reads a diff can confirm the diff does what it says. It cannot catch a claim that is locally true and globally false — and all three of the broken assumptions were exactly that shape.

Lens	Question it was forced to answer
Data integrity & correctness	Recompute every hash. Does the store actually verify? Can any path write a durable row without its receipt?
Security & secrets	Where do the still-live secrets exist, in every location, including backups?
Deployment & operations	What does the service actually run from? What happens on crash-restart or rollback?
Reliability & concurrency	Who can write concurrently, and what lock — if any — serializes them?
Rollout completeness & teammate UX	What does a real member experience on a bad token, a dead API, or off-network?
Adversarial / threat model	Assume a leaked member token. What can it forge, escalate, or clear?

Independence is the other half of the method, and it is not optional. One reviewer with a six-item checklist shares one set of blind spots across all six items — if their mental model of the system is wrong in a given place, it is wrong for every lens they apply. Six agents that never see each other's work cannot collude into a shared assumption. The security lens does not know the reliability lens exists, so it has no reason to defer to it, and that is exactly why one of them looked at the backups while another looked at the write lock. Deduplication happens after independent discovery, never before.

The integrity lens recomputed all 2,186 candidate content hashes rather than trusting the "verified" label. The security lens enumerated every place a live secret's plaintext could exist rather than trusting "we hashed the file."

That is the entire discipline: independent lenses, each grounded in the running system. It cost six agent-runs and the humility to let them contradict the plan. It was worth it, because it broke three things everyone believed were done.

What the review left standing

Before the breakage, credit where the design earned it — because a review that only ever confirms your fears is as miscalibrated as one that only ever flatters you. The review's two headline validations were calls of restraint: it endorsed holding the riskiest feature — the auto-govern path that drives broken call #3 below — as design-only rather than rushing it into the rollout, and it blessed the sequencing that checked migration safety before restarting the live service. In both cases, not shipping yet was the correct instinct.

Two foundational design bets also held up under every lens, and they are worth naming because the fixes below lean on them.

The first is per-user token identities. Minting a distinct token per teammate, rather than a shared team secret, looked like extra onboarding friction at the time. It is what makes the entire server-side intake override (below) possible: every request carries a resolvable actor, so the server has something to re-derive trust and authorship from. A shared secret would have left it nothing to distinguish — you cannot own identity you cannot tell apart. The friction bought the security model.

The second is the append-only, hash-chained receipt ledger itself. The integrity lens recomputed the whole chain and confirmed it does what it claims: alter any promoted row out of band and the recomputation detects it. The atomicity fix that follows does not replace that design — it protects it, by closing the one window where a durable row could exist without its receipt. A weaker audit design would have had nothing worth protecting.

Naming what held is not politeness. It is calibration: it tells you the review's "broken" verdicts are signal, not a reviewer reflexively torching everything to look thorough.

Three calls the review broke

This is the spine of the story. Two shipped decisions survived scrutiny; three did not — and the three that fell are the transferable lessons, because each was a locally reasonable call that the live state proved wrong.

Broken call #1: hashing in place is not rotation

The safety pass hashed the existing tokens in place. It kept the same secret values and just stored their hashes instead of their plaintext, specifically to avoid the churn of re-issuing tokens to everyone. Locally, that reasoning is sound: the credential file no longer holds plaintext, so a read of that file at rest yields nothing usable.

The security lens went and looked at every place those secret values lived. Retained encrypted backups — one local, one off-host, and at least one target with no retention limit — already held the plaintext of those same still-live secrets, captured before the hashing change. Hashing the file protected it going forward, at rest, in one location. It did nothing for a secret whose plaintext sits in a backup you can restore.

Re-hashing a value you have already exposed is not protection. Retiring an exposed secret requires credential rotation — a genuinely new value — plus purging the exposure. If the plaintext still exists anywhere you can restore from, the old secret is live, and hashing its current home is a false sense of safety.

The fix was not "hash harder." It was rotate the tokens to new values and treat the pre-hash backups as compromised. Hash-at-rest was necessary; it was not sufficient, and believing it was sufficient is precisely the failure a live-state review exists to catch.

Broken call #2: local mode is a single-writer design

To let the system's founder query the brain from any editor session, the plugin was enabled in user-scope local mode. Convenient, and locally reasonable: one person, their own machine, direct access. The reliability lens asked the question the plan never did — what does local mode actually do?

Local mode runs in-process as owner/admin. It does not go through the HTTP API. That means it bypasses the entire control plane: the write-gate, the tenant guard, and the audit-actor stamp all live on the API surface, and local mode is a side door around all three. Worse, "any editor session" is not one writer. Up to roughly eleven concurrent interactive sessions became unlocked writers — none of them taking the single-writer file lock that the nightly cron jobs depend on to serialize their writes.

The concrete failure: a backup taken mid-write, with no lock held, can capture a torn state. Restore it and the brain's own tamper-evidence machinery reports TAMPER DETECTED — not because anyone tampered with anything, but because the hash chain was snapshotted between two writes. The safety mechanism fires on its own operator.

A design built for one writer and one user silently becomes a concurrency-and-authorization surface the instant you make it multi-session. The reach goal — "query from anywhere" — was right. Routing it around the single writer was wrong. It should have gone through the API's single writer, inheriting the write-gate, the tenant guard, the audit stamp, and the lock.

Broken call #3: deleting durable rows needs a data-classification check and an all-consumers review

A planned feature — the next thing on the roadmap — would DELETE candidate proposals after they were governed. The reasoning read cleanly if you looked only at the write path: the candidate has been governed, its outcome is recorded, so the row is spent; delete it to keep the table lean.

The integrity lens read the data-classification doc and every consumer of that table, not just the code that does the delete. The candidates table is documented as insert-only, immutable, and a non-reproducible source of truth. Three consumers depend on that:

Provenance. Remote captures write nowhere else. The candidate row is the only copy of what was seen. Delete it and the provenance back-link points at nothing.
The human-review queue depends on flagged and rejected candidates staying in place so a person can adjudicate them later. Delete-on-govern empties the queue out from under the reviewer.
Re-ingest idempotency. The nightly re-ingest path dedupes against that table. Delete a governed row and the next night re-ingests it as new, re-rejects it, and deletes it again — a permanent nightly re-ingest/re-reject loop.

The plan proposed a "second run is a no-op" test to prove safety. That test would have passed while missing the loop entirely, because it seeded the table rather than the upstream spool files — so the dedupe it exercised was not the dedupe the loop breaks. A green test against the wrong fixture is worse than no test; it launders the bug.

A design that deletes durable rows must be checked against the data-classification doc and every consumer of that data — review queue, re-ingest idempotency, provenance back-links — not just the code path that performs the write. "Is this row still needed by the thing that wrote it?" is the wrong question. "Who else reads this row, and what is the only copy of it?" is the right one.

The 18-risk register, gated

Deduplicated across the six engineers, the findings collapsed to eighteen distinct risks. Where two lenses hit the same issue, severity was taken as the max across engineers — you do not average a "critical" with a "medium" and call it "high." A flat wall of eighteen findings is not actionable; it is a demoralizing to-do list with no critical path. So the register was gated into three tiers, and the gating is what made it usable.

Gate	Meaning	Representative risks
Gate 0	Must clear before any teammate onboards or any token email is sent	Rotate exposed secrets + purge backups; hash tokens at rest; immutable deploy with lockout guard; durable revoke-by-actor
Gate 1	Must clear before the auto-govern feature ships	Server-side candidate-intake override; atomic promotion; the delete-on-govern redesign
Gate 2	Hardening — soon, but non-blocking	Additional plugin observability; expiry policy; ancillary rate limits

Gating converts "here are eighteen problems" into "here is the one blocking set standing between you and a safe rollout." It also makes the halt legible: the verdict "do not go all-at-once and do not email tokens until Gate 0 clears" is a statement about a specific tier, not a vague unease. No emails were sent. No teammate was onboarded. That was the correct state to be in, and the gate model is what let everyone agree on it in one sentence.

The fixes that shipped

Gate 0 and the first Gate 1 items landed the same day. Each is a small, well-reasoned change. Shown below are the ones where the design decision is the point.

Tokens hashed at rest, with a safe fallback

The token registry now accepts an already-salted scrypt$salt$hash value in a record's token field and uses it verbatim, so the credential file can store hashes instead of plaintext bearer secrets. scrypt is a deliberately expensive, memory-hard key-derivation function — a stolen hash is costly to brute-force, unlike a bare SHA-256 digest. Plaintext still works, for operator convenience — and a value that merely looks hashed but has non-hex segments falls back to being hashed as plaintext rather than being trusted as a hash.

// A registry record's token field may already be a hashed secret.
// Accept a well-formed scrypt$salt$hash verbatim; otherwise hash it as plaintext.
function resolveToken(field) {
  const parts = field.split('$');
  const isHashed =
    parts.length === 3 &&
    parts[0] === 'scrypt' &&
    /^[0-9a-f]+$/.test(parts[1]) &&   // salt segment must be hex
    /^[0-9a-f]+$/.test(parts[2]);     // hash segment must be hex
  return isHashed ? field : hashToken(field);
}

Necessary — but, per broken call #1, not sufficient on its own. This fix protects the file at rest. It does nothing about the plaintext already sitting in backups. Rotation plus purge was the other half, and shipping the hash without the rotation would have been the exact false-safety the review flagged. The tension is the lesson: a real fix and an incomplete fix can look identical in a diff.

Immutable, tag-pinned deploy with a lockout floor-guard and auto-rollback

The service had been running from a mutable working checkout. That is three latent failures at once: any rebuild in that repo mutates the live service, a crash-restart can relaunch from a torn or feature-branch build, and there is no immutable artifact to roll back to. And there was a sharper trap hiding inside it. Rolling the checkout back past the token-hashing change would rebuild the pre-hash registry, which would then double-hash the now-hashed credential file — and lock out all six users at once.

The fix builds immutable, self-contained release directories behind an atomic current symlink, and refuses to deploy anything older than the migration that would cause the lockout.

# Floor guard: never deploy a commit that predates the lockout-inducing token-hash migration.
FLOOR_TAG="token-hash-floor"
if ! git merge-base --is-ancestor "$FLOOR_TAG" "$TARGET_REF"; then
  echo "REFUSE: $TARGET_REF predates $FLOOR_TAG — deploying it would rebuild the" \
       "pre-hash registry, double-hash the credential file, and lock out every user."
  exit 1
fi

# Build an immutable, self-contained release dir: no .git, frozen deps, built once.
REL="/opt/brain/releases/$(date -u +%Y%m%dT%H%M%SZ)-$(git rev-parse --short "$TARGET_REF")"
mkdir -p "$REL"
git archive "$TARGET_REF" | tar -x -C "$REL"
( cd "$REL" && install_frozen_deps && build )

# Lockout preflight: the built artifact MUST contain the hash parser, or it will double-hash.
grep -q 'scrypt' "$REL/dist/token-registry.js" || { echo "REFUSE: hash parser missing from build"; exit 1; }

PREV_REL="$(readlink -f /opt/brain/current || true)"   # capture the current target FIRST, for rollback
ln -sfn "$REL" /opt/brain/current                       # atomic promotion via symlink swap
systemctl restart brain-api

Then a health gate with auto-rollback, so a bad release un-ships itself:

if ! curl -fsS --max-time 5 "$HEALTH_ENDPOINT" >/dev/null; then
  echo "post-restart health gate failed — rolling back to previous release"
  ln -sfn "$PREV_REL" /opt/brain/current
  systemctl restart brain-api
  exit 1
fi

The git merge-base --is-ancestor <floor-tag> <target> check is a named, transferable ops pattern: a floor guard against a lockout-inducing rollback. Any time a migration makes older code actively dangerous to redeploy — not just wrong, but destructive — pin a floor tag at the migration and refuse to deploy beneath it. Immutability gives you a rollback target; the floor guard makes sure the rollback target can't itself be the disaster.

Durable revoke-by-actor with a persisted revocation list

Tokens had no expiry, and the only revoke path was in-memory — lost on restart. And now that tokens are hashed at rest, an admin no longer holds any teammate's plaintext bearer secret, so "revoke by value" is impossible for the realistic incident: a teammate's laptop was stolen. You cannot revoke a secret you deliberately no longer possess.

So revocation keys off the audit identity the token already carries, and persists to an append-only file read at boot.

// Tokens are hashed at rest, so no admin holds a plaintext bearer secret to revoke by value.
// Revoke by the audit identity the token carries; persist it so it survives a restart.
function revokeByActor(actor, reason) {
  revoked.add(actor);                                   // in-memory guard, effective immediately
  appendFileSync(REVOCATION_LIST,                       // append-only ban-list: durable + audit trail
    JSON.stringify({ actor, reason, at: new Date().toISOString() }) + '\n');
}

// At boot, replay the ban-list so a revoked actor stays revoked across restarts.
function loadRevocations() {
  for (const line of readLines(REVOCATION_LIST)) revoked.add(JSON.parse(line).actor);
}

The design choice worth naming: a separate append-only ban-list file was chosen over mutating the source token file on every incident. Rewriting the credential store during an active incident — the highest-stress moment — is how you fat-finger a lockout. Append-only is safer than rewrite, and it doubles as a revocation audit trail: who was revoked, when, and why, in order, forever.

Server-side candidate-intake override with a provenance receipt

This is the clearest instance of the whole thesis. In team mode the client built the entire candidate object, and the server trusted it verbatim. That means a member — or a leaked member token — could self-assert trustLevel: 'high' to clear a minimum-trust gate, forge the author, set an arbitrary tenant, and clear the "potential secret" flag on their own content. And intake wrote no audit event, so none of it left a trace.

The fix: the server re-derives the fields that decide trust, authorship, and tenancy from the bearer-token identity, ignoring the body for exactly those fields, and writes a provenance receipt on every proposal.

// TEAM MODE: the client builds the whole candidate, but the server trusts NONE of it
// for the fields that decide trust, authorship, and tenancy. Re-derive from the token.
function intake(reqBody, auth) {
  const candidate = {
    ...reqBody,
    trustLevel:      auth.trustLevel,          // NOT reqBody — client cannot self-assert 'high'
    author:          auth.actor,               // NOT reqBody — no forging provenance
    tenant:          auth.tenant,              // NOT reqBody — no cross-tenant writes
    potentialSecret: scan(reqBody.content),    // server re-scans; client cannot clear the flag
  };
  writeReceipt('candidate.intake', candidate, auth);   // every proposal leaves a receipt
  return store(candidate);
}

Two alternatives were considered and rejected:

Reject any candidate that asserts non-default fields. Brittle. The client always sends a full body, including those fields, on every legitimate proposal — so this would reject every real submission, not just malicious ones.
Trim the client body to a minimal DTO at the boundary. A bigger cross-surface contract change — new schema, new client code, new failure modes — for no additional safety, because the override already neutralizes every one of those fields. More work, more blast radius, same result.

The override wins because it is the smallest change that makes the guarantee true: the client proposes a candidate; the server decides what that candidate is. Identity, trust, and tenant are server-owned. That sentence is the whole security model in miniature.

Atomic promotion

The promotion path — moving a governed candidate into durable memory — did roughly five separate autocommits: a supersession update and its event, the memory insert, the graph-edge links, and the "promoted" receipt. A kill mid-promote — the compile cron hitting its timeout, or a plain SIGTERM — could leave a promoted memory with no "promoted" receipt. A durable row without its audit receipt violates the product's core promise — every state change has a receipt — and it never self-heals, because nothing re-derives a receipt for a row that already exists.

The fix wraps the whole write block in one BEGIN IMMEDIATE transaction — SQLite's mode for taking the write lock up front instead of deferring it to the first write statement — so the memory and its receipt commit together or not at all.

// Memory row + its "promoted" receipt must commit atomically, or the
// append-only-receipts promise breaks and never self-heals.
function promote(candidateId, auth) {
  const db = this.getDb();               // read-only getter over the shared connection
  db.exec('BEGIN IMMEDIATE');
  try {
    supersedePrior(candidateId);         // supersession update + its event
    const memId = insertMemory(candidateId);
    linkGraphEdges(memId);
    writeReceipt('promoted', memId, auth);   // nested audit inserts degrade to SAVEPOINTs here
    db.exec('COMMIT');
  } catch (e) {
    db.exec('ROLLBACK');
    throw e;
  }
}

Two design details carry their own rationale. First, the shared DB connection was exposed via a read-only getter rather than threading a new db handle through the constructor — so every existing caller's signature stays unchanged, keeping the blast radius small. Second, the nested audit inserts each open their own BEGIN IMMEDIATE; inside the outer transaction those degrade to savepoints, which preserves the audit chain's anti-fork guarantee rather than fighting it. And the negative control matters: with the outer transaction bypassed, the atomicity tests fail — the memory is orphaned — and with it in place they pass. The test catches the real regression, not a tautology that would pass either way.

Plugin hardening, shipped alongside

The plugin got its own Gate 0 work, because it is the surface six people will actually touch:

Local-mode writers now take the same single-writer flock(2) advisory lock the cron jobs use — a real kernel flock(2) on the same file, so it interoperates with the cron's flock(1). A PID-lockfile library was rejected precisely because it shares no kernel lock with flock(1); two "locks" that don't see each other are not a lock.

// Local-mode writers take the SAME kernel lock the cron holds (flock(2) <-> flock(1)).
const fd = openSync(LOCK_PATH, 'r');
flockSync(fd, 'ex');
try {
  await writeToBrain(payload);
} finally {
  flockSync(fd, 'un');
}

Team-mode search now surfaces errors instead of swallowing them into an empty result. Previously a bad token, a dead API, and being off-network all rendered identically as "the brain found nothing." For a tool whose entire value is trust by receipts, silently returning empty on failure is the worst possible failure mode — it teaches the user the brain has no answer when the truth is it never got the question.
A new read-only brain_status probe answers "am I connected, in which mode, and do I have a token?" — the three things a confused teammate needs before they can even ask for help.
The plugin got its first unit test, lint, and CI gate — before it received the review's riskiest changes. You do not hand new safety-critical code to a surface with no test wall.

The tradeoffs

None of this was free, and pretending otherwise would undercut the point.

The rollout got slower. All-at-once for six people became "one person proves the path end-to-end, Gate 0 clears, then six." That is the right call, but it is a real delay against a plan that was a switch-flip away from done.
There is more operational machinery to maintain now. Immutable release directories, a floor tag to keep current, a revocation-list file to back up, a health gate and rollback path. Every one of those is a thing that can itself break or drift. Determinism and durability cost surface area; the bet is that the surface is cheaper than the incidents it prevents.
Hash-at-rest was a false sense of security until rotation happened. This is the honest one. For a window, the credential file looked safe while the live secrets sat in plaintext in backups. Anyone reading only the "we hashed the tokens" line would have believed the job was done. The review is the only reason that gap was named before it was a breach instead of after.
The review itself has a cost, and it is not just compute. Six independent agent-runs is the cheap part. The expensive part is the discipline to accept a halt the day before launch — to let a review overrule a plan that felt finished, hold the line on "no tokens emailed until Gate 0 clears," and eat the schedule slip. A review you are unwilling to act on is theater with a receipt. The value was realized only because the verdict was allowed to stop the rollout.

The boundary underneath all of it

Read the Gate 0 and Gate 1 fixes together and they are not five unrelated patches. They are five expressions of one boundary:

The server derives identity, trust, and tenant from the token, not the request body. (intake override)
Receipts commit atomically with the data they describe. (atomic promotion)
Deploys are immutable artifacts, not a mutable checkout. (tag-pinned release + floor guard)
Revocation is durable and keyed to identity, not to a plaintext value you no longer hold. (revoke-by-actor)
Credentials live at rest as hashes the system re-derives, not as plaintext a disk or a backup can hand back. (hash-at-rest, paired with rotation)

The model — or the client, or the convenient side door — proposes. The deterministic system owns identity, trust, and durable state. Every place the old design let the proposer also decide what its proposal was, the review found a hole, and every fix closed it by moving that decision back to the deterministic side. That boundary is not specific to this system. It is the load-bearing wall of any governed AI system where something upstream is allowed to be creative and something downstream has to be trustworthy.

And there is a standing invariant the review left behind, worth more than any single fix: all-at-once is the right rollout for six people — but only after one person has walked the entire path end-to-end and Gate 0 is clear. Confidence at team scale is earned by one proof at individual scale, not asserted by a completed checklist.

The meta-lesson is the method itself. Six independent lenses, each forced to verify against live state rather than read the design, are what caught "the backup still holds the plaintext" and "local mode runs as admin, in-process, unlocked." A review that only read the diff would have blessed all three broken calls — because in the diff, all three looked done. The difference between shipping the safety work and knowing you shipped it is whether someone went and looked at the running thing with hostile intent, before the people arrived.

The Moat Is the Trust Layer: Turning a Local-RAG App into a BYOK Document-Intelligence Platform — where adversarially reviewing the graders caught the eval suite lying green.
Every Safety Gate Has a Failure Direction — why one gate crashes fail-closed on bad data while a swallowed error lets another pass fail-open.
Noise-Robust LLM-Judge Evals: Don't Sign a Coin Flip — on not trusting a measurement you haven't proven can go red.

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Adversarial Review: The Six Lenses That Halted a Rollout",
"description": "A six-lens adversarial review checked a team knowledge system against live state, broke three shipped assumptions, and gated 18 risks to halt the rollout.",
"author": { "@type": "Person", "name": "Jeremy Longshore" },
"publisher": {
"@type": "Organization",
"name": "Start AI Tools",
"logo": { "@type": "ImageObject", "url": "https://startaitools.com/favicon.ico" }
},
"url": "https://startaitools.com/posts/adversarial-review-before-team-rollout/",
"datePublished": "2026-07-09",
"keywords": "adversarial review, security review, governed AI system, team rollout, authentication, claude-code",
"articleSection": "Architecture"
}

DEV Community: Jeremy Longshore

Do Not Blindly Restart: Designing a Self-Healing Watchdog That Stays Honest

The problem: a phantom latch

Why not just raise the threshold

The redesign: classify, then bound, then latch

C1: Fail-closed classifier

C2: Slow-flap guard

C3: Health poll, not snapshot

C4: Reset hygiene and honest outcome

Verification is the product

The parallel honesty story

Also shipped

Related posts

Passing Is Not Validating: A Green Check With No Teeth

The Contract

The Symptom

The Bug

The Fix

The Lesson

Also Shipped

Related Posts

A Green Recovery Drill Can Still Be Lying

The shape of the work

Trap 1: the macros trap

Trap 2: the vacuous canary

Trap 3: the poisoned run

Trap 4: the false-pass by default

The collaboration beat

Result

Takeaway

Let the Model Judge. Make the Code Decide.

The pipeline had let the model creep

Produce then land: the model can write junk, but it can never ship it

Single source of truth, applied twice

The voice deny-list

The persona star count

Deterministic enforcement of learned judgment

Measure how the work actually happened

The honest calibration call

The silence bug

Why deterministic-owns-the-gates beats letting the model do it all

The lesson

Copying Files Is Not Installing

Two modes, one of them has native code

The exact failure

The bootstrap launcher

The concurrency protocol

Proving it stays fixed

The general lesson, and two smaller ones

Producer Fallback: When Claude Hits the Weekly Limit, the Pipeline Still Ships

What broke at 04:00

Fix one: do not brick the cron on beads interaction noise

Fix two: hard voice lint (already on master)

Fix three: Grok as producer fallback

What actually shipped for 2026-07-14

Also on the board today

Takeaway

Related Posts

Exit 0 Is Not Success: Automation Assurance That Verifies Outcomes

The lie exit codes tell

What we shipped: independent outcome verification

Why not trust the job?

Fail-closed neighbors on the same day

Also shipped

The dirty-tree lesson (same failure shape)

What is still honest ABSENT

Takeaway

Related Posts

Empty Is Not Clean: Five Fail-Open Bugs in an AI Agent

The pattern: absence read as success

Disguise 1 — empty args made the deny rule vanish

Disguise 2 — a wiped audit log verified clean

Disguise 3 — a prototype key past a truthiness gate, and the near-miss

Why one chokepoint instead of hardening seven sites?

Disguise 4 — a sheared, renumbered chain recomputes clean

Why not a self-describing anchor?

Disguise 5 — a thread-scoped rule silently covered every thread

The shape of the fail-open bug

Also shipped

Related posts