DEV Community: Jeremy Longshore

Honor the Gate When the Verdict Is Inconvenient

Jeremy Longshore — Sun, 14 Jun 2026 13:00:28 +0000

On June 10, 2026, two completely unrelated systems hit quality gates that came back the wrong way. Neither system rationalized the verdict. Both honored it. That honesty is what makes the gate worth having.

The Thesis

A quality gate is only worth building if you will honor its verdict when the verdict is inconvenient. If you override a STOP without pre-registering a follow-up test, or fake a green check because a tool can't run, you're not adding rigor—you're adding theater. The gate becomes another notification that landed in your inbox yesterday.

On the same day, two teams in the Intent Solutions portfolio faced exactly that choice. The stories look nothing alike. The discipline they landed on was identical.

Story 1: Semantic-Flux—Honoring a Pre-Registered STOP on a $4 ML Signal Test

semantic-flux is building QCSS, a query-conditioned semantic search architecture. The patented core is a FiLM layer (feature-wise linear modulation) that lets the query modulate the document encoder's activations.

The project history matters because it shows why pre-registration exists. Earlier runs—v1, v2, v3—each produced results that got rationalized:

v1: 4.79% nDCG FAIL. Someone overrode it. Reason: "It was undertrained." But no pre-registered test was written that could have disproven that excuse. The goalposts moved after seeing the number.
v3: Used a pretrained MiniLM backbone, which is itself a trained retriever. The PASS measured the backbone's strength, not QCSS. The FiLM layer was never ablated away.

Three runs. No trustworthy verdict. Because every excuse closed the gate after the number landed.

So before v4 ran, the team locked a commitment into DECISIONS.md:

PROCEED to Phase 1 only if ALL three:
  - film_lift (nDCG@10 with ON minus OFF) ≥ +0.03, 95% CI excludes 0
  - scratch encoder beats BM25 floor (0.268)
  - seed std ≤ 0.02

STOP if:
  - film_lift ≤ +0.01 (conditioning inert)
  - scratch encoder ≤ BM25 (can't beat 1990s baseline)

Timestamped. Before the number. The FiLM-ablation A/B (query-conditioning ON vs OFF on the same encoder) was the only way to isolate the patented mechanism.

On June 10, the v4 result landed (HF job 6a28d1c9, NVIDIA L4, 3 seeds, full NFCorpus = 3,633 docs):

BM25 floor: 0.268
BGE ceiling: 0.371
Scratch encoder with FiLM-on: 0.0096
Scratch encoder with FiLM-off: 0.0086
film_lift = +0.001—95% CI straddles zero

The FiLM weights never moved from 0.0002–0.0014 across every arm. The mechanism sat at identity.

STOP fired on two counts: film_lift inert AND scratch encoder 0.01 ≪ BM25 0.27.

The team honored it.

The honest read from DECISIONS.md: "This recipe does not work" because two things went wrong: (1) the from-scratch encoder learned almost nothing (trained on only 2,202 queries; MS MARCO has millions); (2) FiLM never trained—a zero-init / backbone-competition deadlock the team had flagged in advance. The result did not falsify QCSS—the mechanism never engaged—and did not support it either. The ~$4 signal test correctly said: do not spend the $2–5K Phase 1 yet.

Here's the transferable artifact. The team baked an anti-rationalization rule into the project: overriding a STOP requires a NEW DECISIONS.md entry that (i) names the specific confound believed to explain the result AND (ii) pre-registers a falsifying follow-up test. "It was undertrained" is not an admissible override without a pre-registered test that could disprove it.

That rule exists because v1's FAIL was overridden without (ii).

The result didn't make the business decision. But it made the decision honest and bounded: either retry with frozen backbone / non-zero FiLM init / higher FiLM learning rate + MS MARCO data (same locked gate, pre-registered falsifier), or halt empirics and file the structural claim. Either way, $4 stopped from turning into $5K of motivated spending.

Story 2: agent-governance-plane—Deleting a CI Gate Rather Than Faking It Green

agent-governance-plane is a TypeScript/Bun project—a policy and governance layer for agent sessions. On June 10 it shipped v0.1.46 and v0.1.47.

v0.1.46 delivered real infrastructure: PR #67 added credential injection into a sandbox plus a test that actually proves network isolation, not just assumes it. Shipped clean.

v0.1.47 (PR #68) set out to close two test-infra gaps:

A Stryker mutation-testing gate—instrumentation of the two highest-risk files (src/policy/engine.ts, src/policy/dangerous.ts), a fail-closed mutation-gate.sh script, and the runner pinned to @stryker-mutator/core@9.6.1.
A Gherkin BDD acceptance layer—tests/features/J1-governed-session.feature with 6 scenarios, backed by real assertions against real modules, 24 expect() calls, no tautologies.

Then Stryker hit a wall: v9 cannot instrument this Bun/TS codebase. The babel instrumenter threw TypeError: generator is not a function (CI run 27325670887). The Bun mutation-runner toolchain isn't there yet, and no fix was in reach for this PR.

That left exactly three options for the new "Mutation testing" CI check:

Leave it permanently red—every PR shows a failing check that means nothing. The team learns to ignore red checks. This is exactly alert fatigue, and it kills every gate downstream.
Make it fake-green—continue-on-error: true or a script that always exits 0. A green checkmark that lies. The label says "Mutation testing passed" when mutation testing never ran.
Remove the check—keep the stryker.config.json and mutation-gate.sh as hash-pinned scaffolding, document the toolchain block in tests/TESTING.md, and file a tracked bead to re-wire it when a Bun-compatible runner exists.

They chose removal.

The commit message: "A permanently-red OR fake-green 'Mutation testing' check both violate the repo's honest-gate culture, so the CI job is removed."

The actual deliverable—the BDD acceptance layer—shipped unaffected. All the real hard gates passed: typecheck, biome lint, coverage-gate at 91.43%, claim-scan, doc-drift audit, harness verify, escape-scan (REFUSE=0, CHALLENGE=0).

The transferable point: a green checkmark is a claim. "Mutation testing passed" has to mean mutation testing ran. If a tool can't run against your codebase, a check that always passes is worse than no check. It launders trust. Removing it (with scaffolding retained and a tracked bead filed) is the honest move, not a regression.

The Parallel

Two domains that share nothing technically—a 530K-param IR encoder on an L4 GPU and a Bun/TS governance layer's CI pipeline—converged on the same discipline:

A gate's value is entirely in whether you honor its verdict when it's inconvenient.

The two failure modes are symmetric:

Rationalizing a STOP after the number lands (semantic-flux v1: "it was undertrained")—textbook confirmation bias. Converts a signal into a comforting excuse.
Faking a green when you can't run honestly (the Stryker check: continue-on-error: true). Converts a missing signal into a false confirmation.

Both erode trust. Both invite drift.

The defenses are also symmetric:

Pre-registration: Write the gate before the number lands so the verdict is binding. The anti-rationalization rule is the written form—name the confound, pre-register the falsifier, or the override doesn't count.
Honest-gate culture: A check must mean what its label says. If a tool can't run against your codebase, remove the check openly rather than let it lie. Keep the scaffolding, file a bead, move on.

Both teams kept an audit trail and a path forward. semantic-flux logged the STOP, the two confounds, and the pre-registered retry option. agent-governance-plane kept the stryker.config.json hash-pinned and filed bead agp-7r4 to re-wire it when the Bun runner ships.

Honoring a gate is not giving up. It's refusing to lie about where you are.

Also Shipped

Elsewhere in the portfolio on the same day, the discipline was quieter but the same — ship the real thing, document the state honestly:

intent-solutions-landing: Deep clean—the marketing site went zero-GCP. Firebase Analytics → self-hosted Umami, deleted ~116MB Firebase Cloud Functions stack, npm audit fix took 25 vulns (13 high) down to 5 moderate / 0 high.
intentsolutions-vps-runbook: Slack alerting cutover from one firehose channel to 11 named channels, fully wired end-to-end with smoke-test evidence.
learn-intentsolutions: 5 study-notes pages distilling a Kobiton Phase-A literature pass (OSS community formation, pain-driven adoption, real-device-cloud market positioning, DevRel effectiveness).

The Cost Asymmetry

The semantic-flux gate cost ~$4 and saved a possible $2–5K of motivated spending downstream. The mutation-check decision cost one CI job and saved the credibility of every other green check in the repo. Cheap insurance against expensive self-deception. Honor the gate when the verdict is inconvenient—that's the whole discipline.

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Honor the Gate When the Verdict Is Inconvenient",
"description": "A quality gate only matters if you honor its verdict. How pre-registration and honest-gate culture stopped two teams from faking green or rationalizing a STOP.",
"datePublished": "2026-06-10T10:00:00-05:00",
"dateModified": "2026-06-10T10:00:00-05:00",
"author": {
"@type": "Person",
"name": "Jeremy Longshore"
},
"publisher": {
"@type": "Organization",
"name": "Start AI Tools"
},
"url": "https://startaitools.com/posts/honor-the-gate-when-the-verdict-is-inconvenient/",
"keywords": "quality gates, CI gates, mutation testing, pre-registration, honest metrics, testing, ci-cd, ml-engineering, devops"
}

Human-in-the-Loop Is a Delivery Guarantee, Not a UI Feature

Jeremy Longshore — Sun, 14 Jun 2026 13:00:27 +0000

Two repos. One missing guarantee.

In agent-governance-plane, a human's approval was cryptographically signed with Ed25519 and written to a tamper-evident journal. Solid. Except the only InteractionSource wired into the system was an in-memory test stub. A human could see an Allow/Deny prompt — but the click had no way home. The signed approval was a letter with no mailbox.

In claude-code-slack-channel, an agent's reply to a Slack thread was a synchronous tool call. If the process died between "decide to send" and "send," the reply vanished. No turn-terminal flush, no retry, no record that an obligation ever existed. The user just waited.

Different repos, different directions of travel — one inbound (receive a human's decision), one outbound (deliver an agent's reply). Same hole: the part where a human and an agent actually hand work to each other was the part nobody made durable.

On 2026-06-07 both repos closed that hole. They shipped the same four-move discipline. And one repo's spec was lifted, by name, from the other's pattern. That is the part worth your attention.

The reframe: this is distributed systems, not UI

Human-in-the-loop gets filed under "product" — a button, a modal, a confirmation dialog. That framing is why it breaks in production. The moment a decision has to survive a crash, an ack-loss, or a dropped socket, you are no longer building UI. You are building a durable message system, and the well-known failure modes apply.

The receiver and the reply path are both solving exactly-once delivery, message deduplication, and fail-closed defaults. Strip away Slack and the problem is identical to any outbox or any consumer that must not lose, must not duplicate, and must not silently double-act.

Four moves fall out of that framing:

Record the obligation before the send. Crash-before-send must be safe — the obligation outlives the process.
Stamp an idempotency key into the message so a later scan can recognize "already delivered."
Redeliver idempotently from a poller — a background drain that reconciles from disk, so ack-loss redelivery is a no-op.
Fail closed on no-decision. Timeout, dropped socket, no lease → deny and journal, or queue. Never crash, never silently double-act.

Hold those four moves. They recur on both sides.

The reply outbox: CCSC's durable delivery pattern

CCSC has an inverted architecture, and the inversion is exactly why durability is hard: a reply is a synchronous tool call, not a turn-terminal event. There is no natural "end of turn" to flush at. That is what the outbox pattern is for: record the obligation durably before attempting the send, so a crash can't lose it. The agent calls a reply tool mid-reasoning and expects delivery to just happen. So the durability machinery has to live behind the tool, invisible to the caller.

The reply-delivery contract (ADR-002 addendum, "Option A: a safety-net behind the reply tool") is deliberately narrow: single-message text replies only. One obligation equals one message, so the idempotency is exact. Chunked, file, and streaming replies stay best-effort and do not enqueue — zero double-send risk, durability deferred to later beads. Scope discipline is part of the design, not a shortcut.

The machinery lives in slack-delivery.ts — a side-effect-free sibling module, deliberately not inline in server.ts, so it is testable without dragging in server module-load side effects. The center of it is deliverReplyDurably:

// Record the obligation BEFORE the send (move 1).
// Crash here is safe — the poller will find the pending obligation and drain it.
async function deliverReplyDurably(deps: IdempotentSendDeps, reply: ReplyObligation) {
  const obligation = await deps.recordPending(reply); // UUID id == idempotency key
  try {
    const ts = await deps.send(obligation);           // one inline attempt
    await deps.markDelivered(obligation.id, ts);
    return { status: "delivered", ts };
  } catch (err) {
    if (isTransient(err)) {
      // Queued — the poller redelivers idempotently. Tell the agent "queued"
      // so it does NOT retry and double-post (move 4: never double-act).
      return { status: "queued" };
    }
    await deps.markDead(obligation.id, err);           // non-retryable → recorded dead
    throw err;
  }
}

The obligation id is a fresh UUID per reply call, and that id is the idempotency key. So when the poller later redelivers the same obligation, it dedups against itself. That is move 2, and it lives in the message metadata:

// The single site that stamps delivery metadata onto the outbound message.
async function postReply(client: WebClient, obligation: ReplyObligation, key: string) {
  return client.chat.postMessage({
    channel: obligation.channel,
    thread_ts: obligation.thread,
    text: obligation.text,
    metadata: {
      event_type: DELIVERY_METADATA_EVENT_TYPE,
      event_payload: { idempotency_key: key },
    },
  });
}

Move 3 is the scan. Before sending, or when redelivering, findDelivered walks the destination thread via conversations.replies with include_all_metadata: true, looking for a message we posted carrying our delivery event_type and a matching idempotency_key. A hit returns the existing ts — so an ack-loss redelivery becomes a no-op instead of a duplicate.

// A redelivery after ack-loss finds the prior post and returns its ts. No second message.
async function findDelivered(client: WebClient, channel: string, thread: string, key: string) {
  const res = await client.conversations.replies({
    channel, ts: thread, include_all_metadata: true,
  });
  const hit = res.messages?.find(
    (m) => m.metadata?.event_type === DELIVERY_METADATA_EVENT_TYPE
        && m.metadata?.event_payload?.idempotency_key === key,
  );
  return hit?.ts ?? null;
}

The poller itself (PR #228) is a deliveryTimer tick calling supervisor.drainOutbox() on an interval — SLACK_DELIVERY_POLL_MS, default 15s, the timer unref'd so it never holds the process open. A boot-time drain recovers crash-pending obligations on startup, and the timer is cleared on shutdown before the supervisor drain, mirroring the existing idle-reaper exactly.

Fail-closed also means degrading gracefully when the outbox itself is unavailable. DurableUnavailableError is thrown before any obligation is recorded — the outbox isn't activated, or there's no lease. The caller catches it and falls back to the prior direct send. Nothing is persisted, nothing needs redelivery, and there is zero regression versus the old path. Durability is additive; its absence degrades gracefully to what shipped before.

Why not the obvious approach?

Why not just retry inline? Because inline retry only survives failures the process is alive to handle. The crash-before-send window — record nothing, send nothing, die — is exactly the window inline retry can't cover. The obligation has to exist on disk before the attempt, or there is nothing to retry from.

Why not best-effort fire-and-forget? Because "the reply usually arrives" is not a contract a human can build on. In a HITL loop the reply is the work product. Best-effort means the agent thinks it answered and the human is still waiting — the worst failure, because nobody knows it failed.

Why is fail-open the dangerous default for an approval gate? This is the load-bearing one. If a receiver times out and the system fails open, the gated action proceeds without a human decision. An approval gate has to act exactly once on a real human decision; failing open makes it act on no decision at all. The entire reason the gate exists is to stop unapproved actions; failing open deletes the gate precisely when it matters. For anything guarding an action, no-decision must mean deny, not proceed.

The approval receiver: AGP's Socket Mode pattern

AGP comes at the same four moves from the inbound side. PR #66 builds the production receiver per spec 033-AT-SPEC — which is explicitly "lifted from the CCSC pattern, completes the HITL round-trip." This is the keystone. What transferred wasn't code: CCSC delivers outbound and AGP receives inbound, so the two share not a single line. What transferred was the discipline — record the obligation, key it, reconcile it, fail closed — restated as a spec for an inbound approval channel. The convergence is not two teams independently reinventing a wheel; one read the other's pattern and applied it to the mirror-image problem.

The transport is Socket Mode: an outbound WebSocket from the control plane to Slack. No public ingress — which honors AGP's "no public surface until defensible" P0 decision. You get durability and no inbound attack surface. That combination is the design point.

Parsing is a pure function, so it is trivially testable and impossible to make stateful by accident:

// parseBlockAction — pure. block_actions payload → SlackInteraction, or null for noise.
function parseBlockAction(payload: SlackPayload): SlackInteraction | null {
  const action = payload.actions?.[0];
  if (!action || payload.type !== "block_actions") return null; // ack-and-ignore
  return {
    nonce: action.value,
    approved: action.action_id === "approve",
    userId: payload.user.id,
    isBot: Boolean(payload.user.is_bot),
  };
}

The receiver holds a pending-by-nonce promise Map, acks every envelope first (Slack drops you if you're slow), then resolves the awaiting awaitInteraction(nonce) on a matching click. A resolved Set makes replay detection explicit — a click that arrives for an already-settled nonce is reported as a replay, never acted on a second time:

class SocketModeInteractionSource {
  private pending = new Map<string, Deferred<Decision>>();
  private resolved = new Set<string>();

  // Surface a stray/replayed click via onRejected — report it, never act on it.
  private reject(nonce: string, reason: string) { /* onRejected({ nonce, reason }) */ }

  onInteraction(i: SlackInteraction) {
    if (this.resolved.has(i.nonce)) {
      // move 4: a replayed click is a no-op, surfaced as a reason — not a second approval.
      this.reject(i.nonce, "nonce already used (replay)");
      return;
    }
    const d = this.pending.get(i.nonce);
    if (!d) { this.reject(i.nonce, "unknown nonce"); return; }
    this.resolved.add(i.nonce);
    d.resolve({ approved: i.approved, userId: i.userId });
  }

  // Timeout defaults to the nonce TTL (5 min) — the receiver never out-waits the nonce.
  awaitInteraction(nonce: string) { /* register + arm TTL timer */ }
}

That replay-detection guard is move 4 again, mirrored: the outbox's never-double-send rule becomes the receiver's never-double-act rule. Same principle, opposite direction of travel.

Fail-closed shows up in three places, and all three matter:

stop() closes the socket and fails closed on every still-pending approval — a shutdown mid-decision denies, it does not hang.
In run.ts, AGP_CHANNEL=slack plus AGP_SLACK_LIVE=1 constructs the live receiver. Unset AGP_SLACK_LIVE fails closed — the system refuses to post a prompt that nothing can answer. A prompt with no receiver is worse than no prompt; it implies a decision is being collected when none can be.
In daemon.ts, a no-decision — receiver timeout or socket drop — now fails closed (deny + journaled reason) instead of crashing the loop, in both mediate() and gate(). It reuses the existing approval.denied journal kind, so there is no schema change: a fail-closed deny is indistinguishable, downstream, from an explicit human deny. The action does not happen, and there is a signed record of why.

The FetchWebSocketDialer (apps.connections.open → wss) injects both fetch and the socket constructor, so the response and adapter logic is CI-tested; only the real new WebSocket seam runs off-CI under AGP_SLACK_LIVE. The same testability discipline as CCSC's side-effect-free module — extract the seam, test everything up to it.

The transferable rule

If you are building any human-in-the-loop agent — Slack, email, a web approval, a CLI prompt — these four moves are your checklist:

Record the obligation before the send. Crash-before-send must be safe.
Stamp an idempotency key into the message. A later scan must be able to recognize "already done."
Redeliver idempotently from a poller. Reconcile from durable state; make redelivery a no-op.
Fail closed on no-decision. Timeout, dropped socket, no lease → deny and journal, or queue. Never crash, never silently double-act.

Move 4 is the non-negotiable one. Anything that gates an action must treat the absence of a human decision as a no. Fail-open turns a safety gate into a rubber stamp the instant the network hiccups.

The convergence is the evidence. When two systems built for mirror-image roles — one delivering replies out, one receiving approvals in — arrive at the same discipline, and one explicitly cites the other, that is not a local trick. It is the shape of the problem. And the discipline it demands is non-negotiable: an agent you can't trust to fail safely is an agent you can't deploy.

Both shipped clean. CCSC's slack-delivery.ts hit 100% line coverage; the test suite grew from 1127 to 1133 tests across the three PRs with all nine gates green each time, and the durable path was extracted to executeReplyDurablePath to keep executeReply under CRAP 30. AGP landed at 88.89% function / 91.43% line coverage — over the repo's configured floor — with typecheck, Biome, claim-scan, harness verify, and escape-scan all green. The PR sequencing in CCSC is itself the lesson: #228 wired the poller into the runtime without touching the reply tool path (machinery live but dormant), #229 added the tested building block plus the ADR (design-first), #230 flipped executeReply to route through it. The security-sensitive change got its own isolated PR. Ship dormant, wire later.

Also shipped

The same day, claude-code-plugins landed a deterministic-CI grading track: a pr-classifier doing file-level PR component detection, feeding per-domain lint workflows plus actionlint and a path-routing test, then a PR-level grade coordinator with golden fixtures — alongside the public 100-point grading rubric at /grading, a 176-test pytest harness for the penetration-tester pack, and a switch of the PR pre-screen LLM from Groq to DeepSeek. And contributing-clanker added two gates: C24 (engagement-frame) and C25 (maintainer-URL leakage).

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "Human-in-the-Loop Is a Delivery Guarantee, Not a UI Feature",
"description": "Human-in-the-loop agent delivery is exactly-once, fail-closed. Two repos shipped the same four-move discipline the same day — convergence, not coincidence.",
"datePublished": "2026-06-07T08:00:00-05:00",
"author": {
"@type": "Person",
"name": "Jeremy Longshore",
"url": "https://startaitools.com/about/"
},
"publisher": {
"@type": "Organization",
"name": "StartAITools",
"url": "https://startaitools.com"
},
"articleSection": "Technical Deep-Dive",
"keywords": "ai-agents, typescript, architecture, slack, distributed-systems",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://startaitools.com/posts/hitl-delivery-is-a-fail-closed-exactly-once-problem/"
}
}

The Wrong Product, Built Perfectly

Jeremy Longshore — Wed, 10 Jun 2026 13:27:20 +0000

A clean pipeline faithfully amplifies a requirements-level error all the way to production. Nothing in a good build process catches a misread spec. The process makes the misread arrive faster, signed, and TLS-valid.

On June 5 we stood up a new site, deployed it through the canonical VPS-as-the-home pattern, and watched it go green — valid Let's Encrypt cert, healthz returning 200, the whole chain — in under an hour. Then it was declared the wrong product. The entire premise was inverted in one sentence.

Here is the part worth keeping: the reversal was cheap. Not because the process was good — it was, but that's not the reason. It was cheap because the expensive, durable infrastructure had been decoupled from the product frame we'd read wrong. A 100% inversion of the product cost us content, not a rebuild. That's the thesis, and it's the only insurance that paid off.

The build that worked

The request arrived around 09:35 local: stand up learn.intentsolutions.io as a "learning hub, open to public," with links out to the owner's other properties so it's "easy to know where to click for what." Deploy it the usual way.

We read "learning hub, open to public" as an outward-facing marketing property — a hub where the public comes to learn. That reading drove everything downstream.

The plan was complete and specific: an information architecture of hero → four-role audience triage → property cards → featured content → FAQ → footer. Charcoal Slate and Zinc color tokens with brutalist CTA accents. Full SEO: <title>, OpenGraph and Twitter meta, JSON-LD Organization + WebSite + FAQPage schema. The option to fan out five design subagents was on the table and declined as process theater — the IA was already specified, so the build went direct. Hold that detail; it matters later, and not in the direction you'd guess.

The execution was clean and fast. The error was entirely upstream of it.

The deploy followed the VPS-as-the-home pattern — eleven steps, every one of them touching real, durable external state. A public GitHub repo, jeremylongshore/learn-intentsolutions, with the Hugo scaffold pushed. An ed25519 deploy key whose public half lands on the VPS behind a force-command lock, so that key can do exactly one thing and nothing else:

# /home/deploy/.ssh/authorized_keys on the VPS
command="/usr/local/sbin/deploy-learn-intentsolutions",no-port-forwarding,no-pty ssh-ed25519 AAAA... learn-deploy

The key cannot open a shell. It cannot forward a port. It triggers one script and exits. The script is the entire deploy surface:

#!/usr/bin/env bash
# /usr/local/sbin/deploy-learn-intentsolutions
set -euo pipefail
cd /srv/learn-intentsolutions/checkout
git fetch --quiet origin main
git reset --hard --quiet origin/main
hugo --minify --gc
rsync -a --delete public/ /srv/learn-intentsolutions/dist/
test -f /srv/learn-intentsolutions/dist/healthz   # fail the deploy if the build is empty

Then the trust plumbing — Tailscale OIDC scoped to exactly this repo, deliberately not the org wildcard:

# Tailscale ACL — OIDC subject scoped to ONE repo
"subject": "repo:jeremylongshore/learn-intentsolutions:*"
# NOT "repo:jeremylongshore/*:*" — a prior AAR's root-cause forbids the wildcard.
# A wildcard subject means any repo in the org can assume the deploy identity.

Four GitHub Actions secrets set with the direct-argument form, because a prior post-mortem found zsh corrupts secrets piped through stdin:

gh secret set TS_OAUTH_CLIENT_ID    --body "$CLIENT_ID"
gh secret set TS_OAUTH_SECRET       --body "$CLIENT_SECRET"
# --body "$VALUE", never `echo "$VALUE" | gh secret set` — stdin gets mangled.

A Caddy block for the subdomain, a Porkbun A-record at 60-second TTL for a fast cutover, and a .github/workflows/deploy.yml calling the reusable vps-deploy.yml with variant: static. Eleven steps, all of them the kind of state that's annoying to create and annoying to recreate.

By every operational metric, this was a flawless ship.

The detour

The deploy didn't go clean on the first pass, and the failure is worth a paragraph because it relocated the blame correctly.

The Caddy reload hung. Two things were wrong at once, which is the worst kind. First, systemctl reload caddy was timing out mid-apply — the systemd unit wrapper, not Caddy itself, was the bottleneck. Second, Let's Encrypt's HTTP-01 ACME challenge couldn't complete because the DNS A-record didn't resolve yet; there was no resolvable name for the challenge to hit.

The fix was two moves. Reorder, so the DNS record exists before the reload and the ACME challenge has something to resolve:

# WRONG order: reload first, then DNS — ACME challenge has no name to hit.
# RIGHT order: DNS first, let it propagate, then reload.
porkbun-cli dns create intentsolutions.io --type A --name learn --content "$VPS_IP" --ttl 60
# ...wait for resolution...

And when the systemd wrapper still timed out mid-apply — leaving Caddy half-configured, HTTP routes loaded but the TLS app missing the new subject — bypass the wrapper and talk to Caddy directly:

# systemctl reload caddy → hangs, half-applies, exit 1
# caddy reload directly → returns immediately
sudo -u caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
# EXIT=0
# "certificate obtained successfully for learn.intentsolutions.io"

That returned 0 instantly and the cert came through. The lesson inside the lesson: the systemd unit was the slow part the whole time. Caddy reloads in well under a second; the wrapper was adding the timeout. End-to-end deploy SLA is around 22 seconds once the infrastructure exists.

Site live. Valid TLS. healthz green. Under an hour, start to finish.

The reversal

The clarification, when it came, was one sentence. The site was meant to be a place for the owner to study — a personal reference and notebook for AWS, Claude, Bedrock, and enterprise AI — not a public destination for an audience to learn alongside them.

Wrong product. Not wrong execution — wrong product.

Read the original phrase again: "learning hub, open to public." It has two opposite readings. One is a hub where the public learns — outward-facing, audience-driven, the thing we built. The other is a hub where the owner learns — a private notebook that happens to live on the open web. The first reading made the public the protagonist. The second makes the owner the only reader who matters.

Everything we'd built assumed the first. The four-role audience triage sorted visitors who weren't coming. The property cards pitched to a reader who didn't exist for this site. The featured-content section curated for an audience of one who didn't want to be an audience. The FAQPage schema answered questions nobody was going to ask. The brutalist CTA called an action that had no taker. The product layer was internally coherent and pointed entirely the wrong way.

What the reversal cost

The frame inversion was a single commit: +436 / −687 across 16 files. It deleted more than it added, which is the signature of a frame inversion rather than a feature change. You don't tweak a misread product; you demolish the part that encoded the misread.

Five partials came out — role-triage.html, property-cards.html, featured.html, faq.html, hero.html — along with the brutalist CTA that was the old index.html. In came reading-optimized layouts — a list.html topic landing and a single.html note page with prev/next navigation. The CSS was rebuilt for long-form reading: a real typography scale, code blocks, tables, blockquotes, breadcrumbs. The dark theme and Inter typeface stayed because they were never the problem. We seeded a /aws/ section with canonical reference links and the first study note, "The AWS mental model," and rewrote the project's CLAUDE.md so the operating frame became "the owner studying," not "the owner teaching."

A second commit — +212 / −3 — fleshed the home page into a curated link directory: 122 links across Anthropic & Claude (docs, cookbook, courses, MCP, prompt caching, vision, extended thinking, Trust Center, Privacy, Commercial Terms, AUP), Amazon Bedrock (Claude-on-Bedrock, Knowledge Bases, Agents, Guardrails, Prompt Management), AWS general, Enterprise AI (PrivateLink, HIPAA, Organizations, SCPs, Control Tower, Amazon Q), and a GDPR / EU data-protection section spanning both vendors — including the sharp note that Bedrock acting as an intermediate processor is not in Anthropic's sub-processor list, because that relationship lives in the AWS contract, not Anthropic's.

Now the ledger that explains why this didn't hurt:

Survived the reversal at zero cost	Discarded
GitHub repo `learn-intentsolutions`	`role-triage.html` partial
VPS checkout + dist directory	`property-cards.html` partial
Force-command deploy key	`featured.html` partial
Tailscale OIDC trust (repo-scoped)	`faq.html` partial + `FAQPage` schema
Caddy subdomain block	`hero.html` partial
Porkbun DNS A-record	Brutalist CTA `index.html`
Let's Encrypt TLS cert	~687 lines of layouts + CSS
`deploy.yml` CI workflow	The entire "audience" frame

Every expensive, durable thing survived untouched. The deploy key didn't care that the product flipped. The TLS cert is for a domain, not a design. The CI workflow ships whatever is in public/. What got thrown away was the cheap, reversible layer — markup, partials, copy. The whole reversal was about an hour of content rework. Not a rebuild from zero.

Why a clean pipeline can't save you

This is the structural point, and it's the reason the post exists.

The error lived at the requirements layer. The pipeline below it was faithful — and faithful is exactly the problem. A high-quality pipeline does not catch a misread spec. It executes the misread perfectly and hands you the wrong thing at production grade. We've watched the same shape from a different angle — a React app whose container shipped the Vite dev server to every visitor, green health checks and all. Every downstream step compounded the original reading: the plan assumed an audience, the IA sorted that audience, the schema described that audience, the deploy shipped it to that audience. Each step was correct relative to the one above it, and the one at the very top was wrong.

The five declined design subagents are the tell. People reach for "more process at the execution layer" as the fix for shipping the wrong thing. It isn't. Those five subagents would have made the audience version more beautiful — better triage copy, tighter property cards, a more polished FAQ. They would have polished the wrong product. Spending more at the execution layer when the defect is at the requirements layer just buys you a higher-fidelity mistake.

Declining them was the right call. It made the build fast, and the speed didn't cause the miss — the miss was already baked in before the first subagent could have run. Execution-layer process spends its budget improving an artifact you've already committed to. It pays off only when the artifact is the right one. The cheap save here was never available at the execution layer; it was one clarifying question at the spec layer, which costs a sentence.

The principle: couple to what you trust

When you build something speculative — a new property, a first cut, anything where the spec might be a misread — the risk question is not "will I get the spec right?" You might not. The honest question is: what did I couple to the part I might get wrong?

Couple the expensive-and-durable to the cheap-and-reversible and a misread spec becomes a demolition — the blast radius is the whole system. Decouple them and the same misread is contained to the product layer: a content edit. That's the entire move:

Expensive and durable — repo, deploy key, OIDC trust, domain, DNS, TLS, CI. Annoying to build, annoying to rebuild. Make it product-agnostic.
Cheap and reversible — IA, layouts, partials, copy, schema, color. Easy to throw away and redo.

The VPS-as-the-home pattern is product-agnostic by design — it ships whatever is in public/ to whatever domain you point it at. It never knew or cared whether the site was a marketing hub or a private notebook. That indifference is the feature. Because the durable layer didn't encode the product frame, flipping the frame couldn't damage it.

So the takeaway, stated flat: decouple the parts you can't cheaply rebuild from the parts you might have read wrong. Then being wrong costs you content, not infrastructure.

Tradeoffs, honestly

Decoupling is not free. The generic VPS-as-the-home pattern is more ceremony than scp -r public/ server:/var/www. Force-command keys, repo-scoped OIDC, reusable workflows, DNS-before-reload ordering — that's real setup overhead, and most of it is invisible until the day you're wrong. If you're never wrong about a spec, you paid for insurance you didn't use. The payoff is asymmetric: small, constant cost; large, occasional save. On a speculative build, where "wrong spec" is a live possibility, the trade is worth it.

And don't draw the wrong conclusion about the five design subagents we declined in the planning stage. Building direct was correct. The fix for this miss is not "use more agents" or "add a design review stage." Both spend at the execution layer, which is the wrong layer. The fix is a cheap clarifying question at the spec layer — "open to public, meaning the public reads it, or meaning it's your notebook that's publicly visible?" — which was one sentence away the entire time. Heavier execution is the expensive cure for a cheap disease.

Also shipped

agent-governance-plane released v0.1.44, anchored by a 546-line competitive-landscape analysis that did something most competitive docs don't: it corrected the project's own prior claims. A competitor's audit log turned out to be Merkle + HMAC — symmetric — not "merely tamper-evident" as we'd characterized it, which means the real differentiator isn't "they don't sign," it's that our signing is asymmetric and publicly verifiable via Ed25519. The doc also pinned the EU AI Act high-risk obligations to the date set by the May 2026 Digital Omnibus agreement — 2027-12-02, pending Official Journal publication — rather than the often-cited 2026 deadline. The discipline worth naming: a competitive document is more useful when it corrects your own marketing than when it flatters it.

intentsolutions-vps-runbook got its production alerting rebuilt ntfy-first. Six topics — health, uptime, backups, deploys, security, incidents — where routine alerts go to ntfy only and high/urgent plus every security event also hit Slack. The root cause of the old single-channel firehose was unglamorous: a SLACK_WEBHOOK_FIREHOSE variable was never set, so every alert fell back to one webhook. The rebuild added an llm_normalize() step (Groq → NVIDIA → raw passthrough fallback chain, hard 6-second timeout so it never blocks an alert) to render alerts in plain English, plus a 398-line operator handover packet for a new DevOps owner.

The Vite Dev Server in Production: The 871-Byte Tell — another story of shipping the wrong thing to production cleanly, and the small artifact that finally gave it away.
Server-Ops MCP: Safety Before Tools — the same instinct as the force-command deploy key: constrain the blast radius before you hand anything the keys.
Self-Expiring, Report-Only CI Gates — on putting process spend where it actually pays instead of where it feels productive.

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "The Wrong Product, Built Perfectly",
"description": "Why a misread specification arrives faithfully at production — and how decoupling durable infrastructure from the product frame makes a total reversal cost content, not a rebuild.",
"author": { "@type": "Person", "name": "Jeremy Longshore" },
"publisher": { "@type": "Organization", "name": "Start AI Tools" },
"datePublished": "2026-06-05T08:00:00-05:00",
"dateModified": "2026-06-05T08:00:00-05:00",
"url": "https://startaitools.com/posts/the-wrong-product-built-perfectly/",
"keywords": "architecture, devops, claude-code, hugo, deployment, requirements, infrastructure decoupling, blast radius"
}

From one adopter to two: the discovery-affordance spec just got named

Jeremy Longshore — Sat, 06 Jun 2026 13:00:21 +0000

A few weeks ago I shipped schema 3.4.0 on claude-code-plugins-plus-skills. The headline feature: a single machine-crawlable manifest indexing 2,783 skills, ~97 KB gzipped. Any client — a marketplace UI, a CLI installer, a federated search index — can browse or search the catalog without scraping the repo directory by directory.

The point of the manifest wasn't "here is a better README." It was a bet that skill ecosystems past a few hundred entries need an affordance for clients, not just humans. Without one, every new tool has to write a custom crawler against each repo's layout. With one, the work happens once at the spec layer and any compliant index is queryable.

I shipped it. One launch consumer (mine). Whether the bet held depended entirely on whether anyone else built against the same shape.

The validation moment

Two weeks ago latentloop07 — building skillnote at luna-prompts — opened issue #596 on sickn33/antigravity-awesome-skills. The repo is a 39,715-star skills library, currently around 1,500 skills across Claude Code, Cursor, Codex CLI, Gemini CLI, Antigravity, and others. The issue title: "at 1,400 skills, how are users actually picking which to install?"

The body lays out the discovery problem at scale, then names my work as the existing model:

jeremylongshore just shipped one for his 2,783-skill claude-code-plugins-plus-skills collection (schema 3.4.0, 97 KB gzipped) and it's the cleanest path we've seen for federated discovery so far.

I didn't file the issue. I didn't post in the thread. The pattern got picked up because the artifact was there to point at.

sickn33 — the repo owner — responded by shipping it. The closeout comment is the part worth reading verbatim:

We addressed it on main with a stable discovery-manifest contract rather than asking clients or users to scrape the full repository: kept the root skills_index.json as the canonical public manifest in the existing array format; mirrored that manifest exactly to data/skills_index.json for compatibility with integrations that already look under data/; added schemas/skills-index.v1.schema.json so downstream consumers can validate the manifest shape; added docs/users/discovery-manifest.md documenting the canonical root manifest, compatibility mirror, array compatibility, lazy-loading pattern, and the expectation that clients should not load every skill up front.

That gives us the Jeremy Longshore-style discovery affordance without replacing the existing canonical public manifest or forcing a breaking format change.

That's the line that made me pause. Not the adoption — the naming. A 39.7k-star repo's maintainer wrote "Jeremy Longshore-style discovery affordance" into a public closeout comment, into a file format that lives in the repo on main, where future contributors will read it as the canonical reference for why the schema exists.

Why "spec, not opinion"

Single-repo specs are easy to mistake for what they're not. Anyone can write a JSON schema, slap a version on it, and call it a standard. Until someone else builds against it, the schema is a strongly-held opinion with a .json extension.

MarketplaceJsonProvider — the consumer-side abstraction skillnote is building — needed at least two real upstream implementations before its shape could be trusted. With one source, the abstraction quietly inherits everything that source happens to do. With two, the abstraction has to actually hold up against both.

As of the closeout, that's the state:

claude-code-plugins-plus-skills schema 3.4.0 — 2,783 skills, my repo
sickn33/antigravity-awesome-skills skills-index.v1.schema.json — ~1,500 skills, independent author

latentloop07's skillnote is wiring its MarketplaceJsonProvider adapter to ingest both from day one. That's the consumer side of the same spec story: two independent upstreams, one consumer abstraction, and the abstraction has to survive being implemented against both without leaking either schema's quirks.

Two adopters is the floor, not the ceiling. But it's the line below which "spec" doesn't mean anything.

What this means for tonsofskills.com users

The practical effect for anyone using the tonsofskills.com marketplace: the catalog isn't trapped in a single-vendor index anymore.

When a third-party tool wants to surface a skill from claude-code-plugins-plus-skills or from antigravity-awesome-skills, it doesn't need a custom integration for either. It reads the manifest, filters by category or risk profile, and lazy-loads only the specific SKILL.md bodies it needs to render. The marketplace UX gets to focus on browse-and-install instead of crawl-and-cache.

For end users that translates to one thing: discovery scales past the point where reading a README is feasible. If you've ever tried to find the right skill in a 1,400-entry directory tree without a search box, you've felt the gap the manifest closes.

What comes next: `install_source_url`

The current schema lets a consumer find a skill. It doesn't yet tell the consumer where to install it from in a fully machine-readable way. The convention right now is heuristic — strip the path, prepend the GitHub raw URL, assume the layout. Works most of the time. Fails silently the rest.

The next schema-feature is a per-entry install_source_url field: an explicit, validated URL pointing to the raw SKILL.md (or skill bundle) that a client can fetch and install verbatim. No path heuristics. No "assume the repo's layout follows convention X." The skill says where it lives, and the installer respects that.

Why it matters past the immediate convenience:

Vendored skills become first-class. A skill that lives outside the manifesting repo (mirrored, forked, hosted on a CDN) can declare its real install URL without the manifest having to lie about location.
Cross-repo installs become safe. A consumer that pulls from multiple upstreams doesn't have to maintain a per-upstream URL-construction rule. Each entry self-describes.
Audit trails get cleaner. "Where did this skill come from" becomes a single field lookup, not an inference.

latentloop07 mentioned filing the install_source_url schema-feature request against my repo this week, sketched against the real sickn33 dataset. That's the next milestone for the spec — the first field added under multi-adopter pressure rather than single-vendor opinion. Different bar.

The take-home

If you write a spec, you find out fast whether anyone wanted it. The signal isn't stars or downloads — it's whether somebody else, working independently, builds against the same shape and names it the same thing. That's the moment an opinion stops being yours.

In this case, the moment came with a verbatim closeout comment on a 39.7k-star repo and a third-party consumer wiring both upstreams into one abstraction. Two adopters, one spec, one consumer in progress. The next test is whether install_source_url survives the same scrutiny — added because both implementers want the same field, not because one repo's maintainer thought it'd be nice.

I'll write up the install_source_url design when the schema PR lands. In the meantime:

The spec lives at claude-code-plugins-plus-skills.
The marketplace surfacing the catalog is at tonsofskills.com.
The thread that named the pattern: sickn33/antigravity-awesome-skills#596.
The skillnote adapter (consumer side): luna-prompts/skillnote.

PRs and feedback welcome.

Vite Dev Server in Production: The 871-Byte Tell

Jeremy Longshore — Sat, 06 Jun 2026 03:36:41 +0000

The 871-byte tell

A user reported scorecardecho.com was slow and "had to be refreshed" to render. The site fronts a React SPA behind Caddy, behind Docker Compose, behind a single VPS — a stack that had been "fine" for months. One curl -s https://scorecardecho.com/ | head -40 answered the ticket in under a minute. The HTML shell was 871 bytes. It contained <script type="module" src="/@vite/client">, /@react-refresh, and a literal src="/src/main.tsx" reference. That output is not a production response. That is the Vite dev server, exposed straight to the public internet, transpiling raw TypeScript per-request for every visitor.

Three smoking guns (the 60-second checklist)

Each signal below is independent. Any one of them is enough to know. All three together is diagnosis-complete.

Signal 1 — the container's own CMD. Inspect what the running image was actually told to do:

docker inspect frontend --format '{{.Config.Cmd}}'
# [npm run dev -- --host]

A production container has no business running npm run dev. That string is the smoking gun in the container definition itself, surviving every restart.

Signal 2 — the port mapping. Compose binds the framework's default development port:

docker compose ps
# frontend ... 127.0.0.1:5173->5173/tcp

Vite's dev port is 5173. Next.js dev is 3000. CRA is 3000. webpack-dev-server is 8080. If the port mapping matches a framework default-dev port (not a chosen runtime port like 80 or 8080-for-nginx), the container was built from a dev recipe.

Signal 3 — the HTML payload. The framework injects dev-only client scripts into the page shell. One grep tells the whole story:

curl -s https://scorecardecho.com/ | \
  grep -E '@vite/client|@react-refresh|webpack-hmr|src/main'
# <script type="module" src="/@vite/client"></script>
# <script type="module">import RefreshRuntime from "/@react-refresh" ...
# <script type="module" src="/src/main.tsx?t=1748394610234"></script>

A built bundle has none of these. A dev server cannot avoid them — they are the runtime hooks that HMR depends on.

Why this happens

The Dockerfile was a single-stage node:22-slim image with CMD ["npm", "run", "dev", "--", "--host"]. It worked on a laptop. Someone (the project's own past self) copied it into production months ago because the site rendered, the port forwarded, and Caddy in front returned 200 OK on every probe. Compose layered on two volume mounts — ./frontend/src:/app/src and ./frontend/index.html:/app/index.html — which made HMR-driven local development feel instant. In production, those same mounts shadowed any dist/ directory the container might have built, guaranteeing the dev server was the only thing that could ever serve a request. Nothing alarmed because nothing was failing — the site was just doing transpile-on-demand for every visitor, every page load, forever.

The fix: a two-stage Dockerfile

The new frontend/Dockerfile builds the SPA in one stage and serves the static dist/ in a second:

-FROM node:22-slim AS base
+FROM node:22-slim AS build
 WORKDIR /app

 COPY package.json package-lock.json ./
 RUN npm ci

 COPY . .
+RUN npm run build
+
+FROM node:22-slim AS runtime
+WORKDIR /app
+
+# `serve` is a tiny static file server with built-in SPA fallback
+# (rewrites every unknown path to /index.html via `-s`).
+RUN npm install -g serve@14
+
+COPY --from=build /app/dist ./dist

 EXPOSE 5173
-CMD ["npm", "run", "dev", "--", "--host"]
+CMD ["serve", "-s", "dist", "-l", "5173", "-L"]

Three flags carry the load. -s is "single-page application" mode: any path that does not resolve to a real file in dist/ rewrites to /index.html, so the React Router routes survive a hard refresh. -l 5173 keeps the listen port identical to the dev configuration — every upstream piece (Caddy block on the VPS, compose port map, internal service name) keeps working with zero coordinated change. -L is serve's --no-request-logging switch; it suppresses the per-request log line the process would otherwise emit, which keeps the container's stdout channel quiet enough that real anomalies (a sudden burst of 4xxs, a backend exception leaking through) are not buried under access-log noise. The port was kept deliberately. Picking 80 or 3000 would have meant editing the Caddyfile and the compose file in the same deploy — a coordinated change for no benefit. Keeping 5173 made the switch a one-file diff at the perimeter.

The compose change

The two source-mount volumes had to go:

   frontend:
     build: ./frontend
     restart: unless-stopped
+    # Production: built static bundle served by `serve` (no dev server).
     environment:
       - VITE_BACKEND_URL=http://backend:3001
     ports:
       - "127.0.0.1:5173:5173"
-    volumes:
-      - ./frontend/src:/app/src
-      - ./frontend/index.html:/app/index.html

Those mounts are HMR-essential in development — they are how a file save on the host instantly reflects in the running container. In production they actively shadow whatever the multi-stage image built into /app/dist. Without removing them, the new build stage would still produce a real bundle, the runtime stage would still copy it into dist/, and then compose would lay raw source files right back over the top of the runtime filesystem the instant the container started. The container would silently revert to transpile-on-demand from the mounted source — same behavior as before, just with extra build time and a confused operator wondering why the fix did not take. Removing the volumes is half the fix; the Dockerfile is the other half. Neither one in isolation would have worked.

Verification: predicted before / after

Signal	Before (Vite dev server)	After (built bundle)
SPA shell HTML	~871 bytes (dev injects `/@vite/client`, `/@react-refresh`, `src/main.tsx`)	~250 bytes (minified `index.html`)
JS payload	N raw `.tsx` modules, transpiled per-request	one minified bundle
Container memory	~136 MB (Node + Vite process)	~30 MB (just `serve`)
TTFB on `/`	~370 ms	single-digit ms (Caddy → `serve` ≈ sendfile)
HMR WebSocket	attempted; periodically renders blocked	no WebSocket at all

The HMR WebSocket line is the one users were feeling. Vite's client repeatedly tried to attach an HMR socket through Caddy, which had no handle block for the upgrade. The attempt eventually timed out, sometimes mid-render, leaving the page partly hydrated — which is the exact "have to refresh" symptom that surfaced the bug.

What would have caught this earlier

Three of the five rows in the verification table are observable from outside the container, which means every one of them is automatable as a deploy-time smoke test. A post-deploy GitHub Actions step running against the public URL would have failed loudly on day one of the misconfiguration:

# Fail the deploy if the production shell still contains dev-mode hooks.
HTML=$(curl -fsSL "https://scorecardecho.com/")
echo "$HTML" | grep -qE '@vite/client|@react-refresh|webpack-hmr|sockjs-node' && {
  echo "FAIL: production HTML contains dev-server hooks" >&2
  exit 1
}

# Sanity-check the shell size. A built SPA index.html for a small site
# lands in the low hundreds of bytes; a dev-injected shell is consistently
# north of 700. Tune THRESHOLD to your own built index.html size + headroom
# (this site's built shell is ~250 bytes, dev-injected was 871 — 600 is the
# midpoint with room to grow before the threshold itself needs revisiting).
THRESHOLD=600
SIZE=$(printf '%s' "$HTML" | wc -c)
if [ "$SIZE" -gt "$THRESHOLD" ]; then
  echo "FAIL: shell HTML is ${SIZE} bytes (> ${THRESHOLD}) — looks dev-injected" >&2
  exit 1
fi

Six lines of shell, run once after every deploy, would have caught the regression the same hour it landed instead of waiting for a user to file a "feels slow" ticket months later. The smoke test does not require introspection inside the container; it does not need access to the VPS; it does not need credentials. It treats the production URL as the contract and asserts the contract holds. That is the cheapest possible enforcement layer, and it is the layer that should have existed from the first deploy.

The transferable lesson

The three-signal checklist works on every Node-based SPA container, not just Vite ones. Next.js dev injects /_next/static/chunks/webpack.js with an HMR client. CRA dev injects webpack-dev-server/client/index.js?protocol=ws. webpack-dev-server in any flavor injects /sockjs-node. The signature is always the same: framework injects dev-only HMR client into the HTML shell, default dev port is exposed on the host, the CMD line still says "dev." Run curl -s https://<your-site>/ | grep -E '@vite/client|webpack-hmr|_next/static/chunks/webpack|sockjs-node|src/main\.(tsx|jsx)' against your own production URL right now. If any line comes back, the container is shipping its dev server to every visitor, and the fix is a multi-stage Dockerfile that costs nothing but stays structurally invisible until somebody looks.

The Unicode Layer Your Validator Can't See

Jeremy Longshore — Thu, 28 May 2026 02:41:56 +0000

A schema validator reads parsed structure. It never sees the bytes.

That gap is where a whole class of supply-chain attack lives. The claude-code-plugins marketplace already ran a schema validator over every skill, agent, command, and catalog file — confirming required fields, enum values, shapes. All of it operating on the parsed document. None of it looking at the raw codepoints underneath.

An attacker can hide an instruction in characters that are invisible to a human reviewer and invisible to a structural validator, yet fully meaningful to an LLM that parses the file as text — or to a shell that executes a line copied out of it. The reviewer sees npm install left-pad. The model sees something else.

On 2026-05-24 the Socket "TrapDoor" advisory described exactly this: invisible Unicode tag characters smuggling instructions into LLM-parsed content. The same day, we shipped a CI gate for it — and folded in the older Trojan Source class (CVE-2021-42574) while we were at it.

The threat model has three shapes

The detection model is built around three distinct attack surfaces, each with a different blast radius and a different rate of legitimate false positives. That asymmetry is the whole reason the gate is tiered instead of binary.

Tag characters (U+E0000–U+E007F). These render as nothing. A human sees an empty span. An LLM reading the file as a token stream reads them as text — they can carry a complete hidden instruction inside what looks like an innocent line of documentation. This is the TrapDoor vector. There is no legitimate reason for a tag character to appear in a skill file. Unambiguous attack.

Bidirectional (bidi) overrides and isolates (U+202A–U+202E, U+2066–U+2069). Trojan Source. These reorder how text renders without changing how it parses. The classic demonstration: code that displays as a benign comment to a reviewer but executes as an active statement. Renders as one thing, parses as another. Also unambiguous.

Homoglyphs. A Cyrillic а (U+0430) is pixel-identical to a Latin a (U+0061) in most fonts. Drop one into npm install pаckage and the reviewer reads the right name while the resolver fetches a different one. This is a real attack — but mixing scripts is also completely normal in human prose. Cyrillic next to Latin in a sentence is not suspicious. The signal only matters in a narrow context.

The artifact

scripts/validate-unicode-hygiene.py — 317 lines, stdlib only. argparse, pathlib, re, unicodedata, dataclasses. No third-party dependency, nothing to pin, nothing to audit transitively. The detection rules are original, derived from the public Unicode Standard, the CVE-2021-42574 advisory, and the TrapDoor advisory. Not a fork of any existing scanner.

The codepoint classes map straight onto the threat model:

TAG_CHARS = range(0xE0000, 0xE0080)  # exclusive end -> U+E0000-U+E007F inclusive
BIDI_CONTROLS = frozenset({0x202A, 0x202B, 0x202C, 0x202D, 0x202E,
                           0x2066, 0x2067, 0x2068, 0x2069})
ZERO_WIDTH_MAJOR = frozenset({0x200B, 0x200C, 0x200D, 0x2060, 0xFEFF})
OTHER_INVISIBLE = frozenset({0x00AD, 0x034F, 0x115F, 0x1160, 0x17B4, 0x17B5})
HOMOGLYPH_SCRIPTS = frozenset({"Cyrillic", "Greek", "Armenian", "Cherokee"})

Tiered severity, not a single verdict

A binary pass/fail gate forces a bad choice: either it's too loud (every zero-width space halts the build) or too quiet (you skip the ambiguous cases and miss the homoglyph in an install line). Three tiers resolve the tension.

BLOCKER — tag characters and bidi controls. These fail CI by default. There is no benign use, so there is no false-positive cost to refusing them outright.

MAJOR — zero-width and format characters (U+200B, U+200C, U+200D, U+2060, U+FEFF) anywhere they don't belong, plus other invisibles like soft hyphen (U+00AD), combining grapheme joiner (U+034F), Hangul fillers, and Khmer zero-width vowels. These can be attacks, but they also show up in legitimate-if-messy authoring. So they warn by default and only fail under --strict.

MINOR — mixed-script identifiers, but scoped hard. The homoglyph pass fires only inside URLs, package-manager install lines, and code-fence language tags. Never on prose.

That scoping is a deliberate design call. The line patterns the homoglyph pass inspects: https?://, npm/pnpm/yarn/bun install, pip/uv install, cargo install/add, brew/gem/composer/go install, and gh repo clone. Those are the lines a reader copies and runs. Prose is left alone because flagging every Greek letter in a math explanation would bury the one finding that matters.

Severity tiers at a glance

Tier	Character class	Examples	Action	False-positive cost
BLOCKER	Tag chars, bidi controls	U+E0000–U+E007F, U+202A–U+202E	Fail CI immediately	None — no legitimate use
MAJOR	Zero-width / format chars	U+200B, U+FEFF (non-BOM), U+00AD	Warn by default; fail under `--strict`	Possible in legitimate authoring
MINOR	Mixed-script identifiers	Cyrillic `а`, Greek `α` in URLs / install lines	Warn in narrow contexts only	Low — scoped to package lines

The BOM exception

One codepoint sits on a fence. U+FEFF is a byte-order mark when it's the very first byte of a file — legitimate. The same codepoint anywhere else is a zero-width no-break space, which is exactly the kind of invisible an attacker reaches for. So the rule grants a pass to exactly one position and flags every other occurrence:

elif cp in ZERO_WIDTH_MAJOR:
    # A single U+FEFF at the very first byte of the file is a
    # legitimate BOM and gets a pass.
    if cp == 0xFEFF and line_no == 1 and col_idx == 1:
        continue
    findings.append(Finding(severity="MAJOR", ..., rule="zero-width-or-format", ...))

Position-aware, not codepoint-aware. The byte is fine at offset 0 and suspect everywhere else.

Make the invisible visible in the log

A finding that says "MAJOR at line 14, column 22" is useless if the reviewer opens the file and sees nothing there — because the offending character is, by definition, invisible. Every finding carries file:line:column, the codepoint's unicodedata.name() label, the rule name, and a ~32-character context window with every invisible escaped to <U+XXXX>:

def _escape_context(line, column, width=32):
    ...
    for ch in window:
        cp = ord(ch)
        if cp in TAG_CHARS or cp in BIDI_CONTROLS or cp in ZERO_WIDTH_MAJOR \
           or cp in OTHER_INVISIBLE or cp < 0x20:
            out_chars.append(f"<U+{cp:04X}>")
        else:
            out_chars.append(ch)
    return "".join(out_chars)

Now the CI log shows npm install p<U+0430>ckage instead of a line that looks identical to the clean one. The reviewer can actually see the attack.

Ship report-only, then ratchet

The gate has two rollout switches. --warn-only always exits 0 — it reports findings without failing the build, for the window where you're still learning what's in the corpus. --strict flips MAJOR findings into build failures once you've cleaned up the known-benign noise.

This is the same self-expiring report-only pattern we use elsewhere: land a gate in advisory mode, let it observe production traffic, then enforce once you've proven it won't false-positive your own contributors into a wall. BLOCKER fires from day one because it has no false-positive cost; MAJOR waits behind --strict until the corpus is clean.

The result

Wired into .github/workflows/validate-plugins.yml next to the existing schema validator. Scanned 4,776 files.

Zero blockers. Clean main.

Eight MAJOR findings — and the honest detail is the interesting one. All eight traced to a single community-contributed file that intentionally used a zero-width space inside a fenced code block as a rendering workaround. Not an attack. A legitimate-but-messy authoring choice. That is precisely why MAJOR sits behind --strict and isn't flipped on yet: the ratchet waits until that one file is cleaned up, so the first enforced run doesn't punish a contributor for a cosmetic hack.

Tests cover the boundaries with six byte-precise fixtures — blocker-tag-chars, blocker-bidi-override, bom-allowed, clean-skill, major-zero-width, minor-homoglyph-install — driving an 8-test regression suite in tests/test_validate_unicode_hygiene.py. The whole thing shipped as PR #777 closing #776: ~317 lines of validator, one workflow edit, one test file.

Also shipped

Same day, part of a wider CI-hardening campaign:

An after-action review (PR #775) closed a 2026-05-22→24 hardening sequence: 11 PRs landing v4.32.0 with 10 blocking required gates and zero report-only, plus cleanup of 974 Python errors, 223 shellcheck warnings, ~60k markdown issues, and 970 MB freed.
Doc-quality gate "round 2" in two other repos (intentional-cognition-os, qmd-team-intent-kb): fixed Vale by scoping via per-directory .vale.ini sections instead of the action's broken single-path files: input, and lychee by passing . as the required positional argument. Scoping refinements, not policy loosening — the gates stay BLOCKING.
intentional-cognition-os also got an ico audit verify SHA-256 chain verifier. The audit log had carried a tamper-evident hash chain since launch, but nothing actually walked it — so the tamper-evidence was theoretical. Now ico audit verify exits 2 on AUDIT_TAMPERED.

Self-Expiring Report-Only CI Gates — the --warn-only → --strict ratchet is the same advisory-to-enforced pattern, generalized.
Safety Model First: 16-Tool Ops MCP in One Day — designing the threat model before the surface, applied to an ops MCP server.

{
"@context": "https://schema.org",
"@type": "BlogPosting",
"headline": "The Unicode Layer Your Validator Can't See",
"description": "Schema validation can't see invisible Unicode. A stdlib-only CI gate that catches tag-char injection, Trojan Source bidi overrides, and homoglyph attacks.",
"image": "https://startaitools.com/images/og-image.png",
"datePublished": "2026-05-24",
"author": {
"@type": "Person",
"name": "Jeremy Longshore"
},
"publisher": {
"@type": "Organization",
"name": "Start AI Tools"
},
"url": "https://startaitools.com/posts/unicode-hygiene-gate-same-day-trapdoor-defense/"
}

Self-Expiring Report-Only CI Gates: From Advisory to Enforced

Jeremy Longshore — Wed, 27 May 2026 13:00:40 +0000

Advisory CI gates are where good intentions go to die. A team adds a linter "in warning mode for now," and "for now" becomes forever. The violations scroll past in PR reviews, nobody cleans them, the gate never goes blocking. Six months later the warnings are archaeological noise.

The pattern that breaks this cycle is simple but mechanical: every report-only gate carries a self-expiring deadline marker — REPORT-ONLY-UNTIL: <date> — and a meta-gate script fails the build if any gate outlives its deadline. Advisory mode becomes temporary by construction. The second part: organize around one logical concern per PR, bulk-cleaning all violations for that concern in the same commit. Tightly-coupled tools — like eslint and prettier, or ruff and ruff-format — flip together because they share a single run step and a single class of violation. Unrelated gates never share a PR. This keeps each change reviewable and the blame surface small.

What Is a Self-Expiring Report-Only CI Gate?

A self-expiring report-only CI gate is a quality check that runs in advisory mode — violations are warnings, not build failures — but carries an explicit deadline date. A meta-gate fails the build once that deadline passes, forcing the team to either clean the violations and flip the gate to blocking, or remove it. This prevents advisory gates from quietly becoming permanent technical debt.

The Meta-Gate: Enforced Deadlines

The enforcement script is small but decisive. It lives in the workflow as a step that runs before any code cleanup and fails the build if any gate is past its deadline:

- name: Check CI deadline compliance
  run: |
    python scripts/check-ci-deadlines.py
  env:
    GITHUB_REF: ${{ github.ref }}

The script (~150 lines) scans workflow files for REPORT-ONLY-UNTIL: <date> markers and compares them against datetime.now():

import re
import datetime
import sys

def check_deadlines(workflow_file):
    with open(workflow_file) as f:
        content = f.read()

    # Match: REPORT-ONLY-UNTIL: YYYY-MM-DD
    pattern = r'REPORT-ONLY-UNTIL:\s*(\d{4}-\d{2}-\d{2})'
    matches = re.finditer(pattern, content)

    today = datetime.date.today()
    expired = []

    for match in matches:
        deadline_str = match.group(1)
        deadline = datetime.date.fromisoformat(deadline_str)

        if today > deadline:
            expired.append((deadline_str, match.group(0)))

    return expired

if expired_gates := check_deadlines('.github/workflows/validate-plugins.yml'):
    print("FATAL: report-only gates past deadline:")
    for deadline, marker in expired_gates:
        print(f"  {marker} (expired {deadline})")
    sys.exit(1)

The hardcoded workflow path can be swapped for any project (or glob all workflow files in .github/workflows/).

The marker lives as a comment in the workflow, immediately preceding the gate it guards:

- name: Run ESLint
  # REPORT-ONLY-UNTIL: 2026-06-20
  run: npm run lint:js || true

When the deadline arrives, the meta-gate blocks the build. To flip the gate to blocking, you remove the marker AND remove the || true in the same commit:

  - name: Run ESLint
-   # REPORT-ONLY-UNTIL: 2026-06-20
-   run: npm run lint:js || true
+   run: npm run lint:js

This forces a deliberate choice: keep the gate advisory (extend the deadline or remove the marker entirely to soft-delete it), or flip it to blocking (bulk-clean the violations first, then remove both markers in one PR).

The Campaign: Nine PRs, Zero Rot

Claude-code-plugins is a 2000+ GitHub star monorepo with ~10,468 markdown files and plugins written in Python, TypeScript, shell, and markdown. The groundwork — the deadline meta-gate and one chastening moment where a freshly-blocking gate was flipped back to report-only after it caught a real failure on its first run — landed 2026-05-21. The campaign proper ran across 2026-05-22 and 2026-05-23, organized as nine sequential PRs (A through I), each addressing one logical concern.

PR A (#764): ESLint + Prettier

ESLint and Prettier flipped to blocking gates together. No backlog of violations; the gates were added with cleanup already done, so the flip was immediate. These tools are tightly-coupled (same run step, same class of violations).

PR B (#765): Ruff + ruff-format

Ruff (Python linter) and ruff-format flipped to blocking gates together. Bulk cleanup across ~180 Python files, then flip. This PR established the template: bulk cleanup + immediate gate, no deadline crutch. Again, tightly-coupled tools (same step, same violation class).

PR C (#767): Markdownlint (report-only)

Markdownlint added as a report-only gate. Added with a REPORT-ONLY-UNTIL: 2026-06-20 deadline because the backlog was large (80 violations across 10,468 files). This was deliberate: the gate exists, enforcement is temporary, and the deadline is explicit.

PR D (#766): Root directory hygiene

Root directory cleanup (removed stale license files, dead symlinks, misplaced config fragments). No gate change, just hygiene.

PR E (#768): Shellcheck — where the pattern earned its keep

Shellcheck cleanup and flip to blocking. This PR surfaced the real value of the pattern.

Shellcheck flagged 223 violations across 47 shell script files. These weren't nitpicks a reviewer would catch — they were silent runtime failures that report-only mode had been hiding for months. Flipping the gate to blocking forced them into the light.

First detail: 47 of the .sh files were actually Python scripts with a #!/usr/bin/env python3 shebang but .sh extension. Shellcheck flagged them as SC1071 (unknown shell dialect). The fix was git mv to rename them .py, which also correctly moved them under ruff's scope (discovered later in PR G when the widened-test-loop caught new Python lint).

The real bugs shellcheck surfaced:

SC2064 – trap expansion time. A cleanup handler was written as:

trap "rm -rf $temp_dir" EXIT

The issue: $temp_dir expands when the trap is SET (at script start), not when it FIRES (at exit). If the script changed the variable later, the trap deleted the wrong directory (or nothing at all). The fix:

trap 'rm -rf "$temp_dir"' EXIT

Single quotes defer expansion to fire-time. Shellcheck flags this by default; it's a real gotcha.

Unquoted redirection operator. A dependency pinning line read:

pip install some-package[extra]>=1.0

The >=1.0 is unquoted, so bash interprets > as file redirection. The script silently created a file named =1.0 instead of pinning the version. Quoting the entire argument:

pip install 'some-package[extra]>=1.0'

fixes it. Shellcheck flags the unquoted redirection.

SC2166 – POSIX -o operator. The script used:

[ -f "$file1" -o -f "$file2" ]

The -o operator (logical OR in [ ] test) is not well-defined in POSIX and can misparse in edge cases. The portable form:

[ -f "$file1" ] || [ -f "$file2" ]

is safer and clearer.

Other notable finds: SC2188 (no-command redirect — fix: : > "$LOG_FILE"), SC2213/SC2214 (getopts with duplicate option letters and unreachable case branches).

After cleanup, PR E added a .shellcheckrc file with project-wide disables for checks that were genuinely incompatible with the codebase style:

disable=SC1090,SC1091,SC2155,SC2034
severity=warning

These are set in the config, NOT in the workflow step. A gotcha: .shellcheckrc does not have a severity= directive (it only recognizes disable=). The severity policy is set at runtime in the CI step, so the workflow must include --severity=warning in the run: command.

PR F (#769): TypeScript coverage + codeblock syntax

TypeScript coverage audit and code-block syntax linting flipped to blocking together. These tools are tightly-coupled (both operate on codeblock content in TypeScript/SKILL.md).

PR G (#770): Widened test loop

Widened test loop (a 15-minute timeout gate that runs the full plugin suite). Flipped to blocking.

PR H (#771): Codeblock syntax cleanup

Codeblock syntax cleanup (SKILL.md and README fenced-code fence fixes across plugins) flipped to blocking.

PR I (#772): Markdownlint — the last gate flips

Markdownlint, the last report-only gate. 80 violations reduced to zero across 10,468 files. The largest categories were:

MD051 (broken anchor links): 33 violations. TOC link targets regenerated to match GitHub's auto-slug format.
MD056 (table column mismatch): 20 violations. Notable: 8 were a literal | inside a backtick code span that needed escaping to \|, and 12 were genuine header/row count mismatches.
MD001 (improper heading increment): 7 violations.
MD045 (missing alt text on shield badges): 3 violations.

The remaining ~17 violations were distributed across smaller categories (MD046 indented-code, MD003 setext-style, MD031 fence blank-line) and cascading --fix resolutions.

After PR I merged: 10 blocking required gates, zero report-only. The gates: eslint, prettier, ruff, ruff-format, shellcheck-skills, skill-codeblock-syntax, typescript-coverage-audit, widened-test-loop, markdownlint, and the base validate step.

The Proof: External Contributions Now Bind

The same day PR I merged (2026-05-23), a community-contributed plugin (agency-os, PR #709) landed. This PR had been open for review since before the campaign; when the 10 gates went blocking on 2026-05-23, the still-open PR's CI started failing, forcing a restructure. Its SKILL.md was 863 lines of auto-generated cruft. The marketplace-tier required fields (name, description, allowed-tools, version, author, license, compatibility, tags) were missing.

The CI ran. All 10 gates failed. The contributor restructured the SKILL.md from 863 lines to 168 and added the 6 missing marketplace fields. No human reviewer had to enforce the standard in the PR thread — the CI did. This is the payoff: the gates now bind external contributions automatically, without a human argument happening in the thread.

How to Adopt This

Add the meta-gate script (check-ci-deadlines.py) to scripts/. Swap the hardcoded workflow path (or glob all workflow files) and any project can adopt it.
Mark each report-only gate with a deadline comment before the run step. Use format # REPORT-ONLY-UNTIL: YYYY-MM-DD. Choose a date 4–6 weeks out.
Add the meta-gate to your workflow as an early step. It fails fast if any gate is past deadline.
When you're ready to flip a gate or tool-pair to blocking: bulk-clean all violations in one commit, then remove the deadline marker AND the || true in the same PR. One logical concern per PR (tightly-coupled tools flip together).
Never extend a deadline without a written plan. If the backlog is too large, the deadline was too aggressive; next time, choose 8–12 weeks instead. The discipline is in the deliberate choice, not in the calendar date.

The system ensures that advisory gates cannot quietly rot into permanent technical debt. They expire, they get re-evaluated, or they go blocking. Technical governance becomes automatic.

Safety Model First: 16-Tool Ops MCP, One Day

Jeremy Longshore — Mon, 25 May 2026 13:00:30 +0000

The Intent Solutions production stack now lives on a single Contabo VPS after a multi-week migration. Twenty-four containers across five stacks — Braves, Plane, Twenty, Umami, ntfy — sit behind one Caddy reverse proxy. Every day-to-day operational task touches that box: reload Caddy after a host-block edit, restart a stuck container, pull the last 200 lines of a service log, snapshot an instance before a risky change. Doing those by hand from a shell defeats the point of having Claude Code in the loop. Doing them through a sloppy MCP server is how you brick prod from a chat window.

So the question on day one wasn't "what tools do I want?" The question was "what would have to be true of those tools before I let a model fire them at a server I actually depend on?"

The answer became a 7-point safety model, written into the README on the very first commit, before any tool existed. That model — and the host registry that sits behind it — is the whole reason the rest of the work fit in one day.

The 7-point safety model

The model is what every tool in the catalog has to satisfy before it ships: a hard write denylist, dry-run-by-default for destructive operations, argument validation, key-based SSH only, backup-before-write, validate-before-reload, and capped output. No exceptions, no "we'll add that later."

Write denylist (hard). server.file.write always refuses /etc/passwd, /etc/shadow, /etc/sudoers[.d/...], ~/.ssh/authorized_keys, /boot/, /usr/. Per-host config cannot opt out.
Dry-run by default. Every destructive tool requires explicit apply: true to actually fire. Omitting it returns the command that would run.
Argument validation. Unit names, container names, service names, and paths are regex-validated before they ever reach a shell. /^[A-Za-z0-9_.@-]+$/ for systemd units, /^[A-Za-z0-9_.-]+$/ for container names, /^\/[A-Za-z0-9_./-]+$/ for absolute paths. Shell metacharacters never get the chance.
Key-based SSH only. The ssh2 Client opens with privateKey. No password fallback, no agent fallback. If the key isn't there, the call fails.
Backup before write. Real writes do cp -p <path> <path>.bak first. Recovery is one SSH command away.
Validate before reload. server.caddy.reload runs caddy validate first and refuses to reload if validation fails. A broken Caddyfile cannot leave the building.
Output capped. server.exec defaults to 8 KiB per stream; server.file.read to 1 MiB. A runaway log tail can't blow up the context window or the model's reasoning.

The thing to notice: none of these are tool-specific. They're cross-cutting invariants. Designing them first meant every tool I built afterward had a fixed checklist to satisfy, not a fresh argument to relitigate.

A tool author shouldn't be inventing safety policy at 3pm on commit four. By then, the pressure to "just ship this one helper" has won the argument. The 7-point list exists so that argument never starts — every tool either passes all seven gates or doesn't ship.

The host registry as the access-control boundary

Here's the architectural choice that did the most work: the tools enforce nothing host-level. The registry does.

Only hosts in ~/.config/server-ops/hosts.yaml are reachable. The tools take a host parameter, look it up, and refuse if it isn't there. Adding a new host is one YAML edit. Rotating a key is one YAML edit. Restricting which commands server.exec may run on a given host is one YAML edit.

hosts:
  intentsolutions:
    address: 167.86.106.29
    user: intentsolutions
    key: ~/.ssh/id_ed25519
    port: 22
    allowed_commands:
      - "^docker "
      - "^free"
      - "^df"
    write_allowed_paths:
      - "^/srv/braves/.env$"
      - "^/etc/caddy/Caddyfile$"

  dev-box:
    address: 100.x.x.x
    user: jeremy
    key: ~/.ssh/id_ed25519
    # No allowed_commands -> all commands allowed
    # No write_allowed_paths -> denylist only

Two optional per-host regex allowlists do the per-host narrowing:

allowed_commands — server.exec only runs commands matching one of these. Omit to permit anything.
write_allowed_paths — server.file.write only writes paths matching one of these. The hard write denylist is always enforced regardless.

The prod host gets the tight collar. The dev box stays loose because if I brick it, I haven't taken any customer down.

The asymmetry is the point — least privilege per environment, not a blanket policy. A flat policy that treated every host identically would either be too loose for prod or too tight for the dev box. Per-host config lets each environment carry exactly the constraints that match its blast radius. The Caddyfile and the Braves .env are the only two files I ever intentionally edit on the prod box; everything else stays read-only. That's enforceable in seven lines of YAML.

Why not let the tools do ACLs?

Tempting design: bake the allow/deny logic into each tool. Don't. Two reasons.

First, you end up with the same access policy duplicated across exec, file.write, systemd.restart, docker.restart, compose.up/down, and caddy.reload. Drift between those is inevitable, and one drifted tool is the one an attacker (or a confused model) walks through.

Second, the host registry is a much better trust seam. Reviewing one YAML file is a five-minute job. Reviewing seven tool implementations for consistent ACL enforcement is a meeting. The tools stay simple — they just do their one thing, validate their args, and ask the registry "am I allowed here?" The registry owns the answer.

The shared SSH client pool

The second choice worth calling out: one ssh2.Client per host, shared across exec and SFTP.

Every tool — server.exec, server.file.read, server.file.write — goes through the same pool. A first call to host:intentsolutions opens the TCP connection, authenticates, and caches the client. Every subsequent call against that host reuses it. The pool drops the cached client on close or error events so the next call reconnects cleanly. On SIGINT / SIGTERM the server calls closeAllClients() to drain on shutdown.

The alternative — opening a fresh SSH session per tool call — would have added a full TCP + TLS + auth roundtrip to every operation. With Claude Code firing a sequence of exec then file.read then systemd.restart calls against the same host, that's three connection setups where one would do. Worse, key-only auth is slow enough that the latency would show up in conversations.

Pooling is one of those "obvious in hindsight" decisions that only stays clean if you commit to it on day one. Retrofit-pooling is a rewrite — every tool that opened its own client now has to learn to ask the pool, and the lifecycle questions (who closes? who reconnects on error? what happens on shutdown?) get answered seven different ways instead of once.

Why not use ssh-agent?

Using the local SSH agent would have eliminated the need to handle a private key file directly. It would also have made the server's behavior depend on out-of-band ambient state — whether an agent is running, whether the right key is loaded, whether the user remembered to ssh-add after a reboot. For a server that needs to be predictable from a long-running Claude Code session, that ambient state is a liability. The privateKey path is explicit, reproducible, and the failure mode is loud: missing key, fail immediately.

The tool catalog

With the safety model and the host registry settled, the actual tools fell out fast. Six commits, 16 tools, 40 unit tests, ~1,950 lines added.

Tool	What it does	Destructive?
`server.exec`	Run shell command; capped output, optional allowlist	Configurable
`server.file.read`	SFTP read, 1 MiB cap	No
`server.file.write`	SFTP write with `<path>.bak` snapshot	Dry-run default + write denylist
`server.health`	Snapshot uptime, free, df, sensors, 7d OOM count	No
`server.systemd.status`	`systemctl status <unit>`	No
`server.systemd.restart`	`sudo systemctl restart <unit>`	Dry-run default
`server.caddy.reload`	Validate then reload Caddyfile	Dry-run default; refuses if validate fails
`server.docker.ps`	List running containers	No
`server.docker.logs`	Tail container logs (default 200 lines)	No
`server.docker.restart`	Restart container	Dry-run default
`server.compose.up`	`docker compose up -d` in a directory	Dry-run default
`server.compose.down`	`docker compose down`	Dry-run default
`server.compose.logs`	Tail one service's logs	No
`contabo.instance.list`	List Contabo instances	No
`contabo.instance.create`	Provision new instance (billed)	Dry-run default
`contabo.instance.snapshot`	Snapshot an instance	Dry-run default

The pattern is uniform: read-only tools just run; destructive tools default to dry-run and only fire with apply: true. A dry-run call returns the command that would execute, so the model (or the human reading the transcript) can verify before committing.

The Contabo wrapper has a quirk worth knowing

Contabo's identity server requires both OAuth2 (client_id, client_secret) and legacy (api_user, api_password) to mint a bearer token. This is non-standard — most OAuth2 flows want one or the other, not both. The wrapper validates all four env vars are present and surfaces a clean error if any are missing:

Missing Contabo credentials: CONTABO_API_USER, CONTABO_API_PASSWORD

The token is cached in-process until 30 seconds before expiry. Tests expose _clearTokenCache() to reset between cases. contabo.instance.create and contabo.instance.snapshot default to dry-run for the obvious reason: a create call books real money. Snapshot doesn't bill the same way, but the principle is consistent — if it mutates infrastructure, it asks before firing.

The wrapper is intentionally thin. It's not trying to be a full Contabo SDK. It exposes three operations because those are the three I actually need from a Claude Code session: list what's running, provision a new instance when scaling out, snapshot before doing something risky. Adding more later is cheap; getting the safety defaults right is the part that has to be right on the first commit.

Why not allowlist commands inside `server.exec` itself?

I went back and forth on this. The case for in-tool allowlists is "defense in depth — even if the registry is wrong, the tool refuses." The case against won, for the same reason the registry owns ACLs in general: every additional enforcement point is another spot where the policy can drift from intent.

The server.exec allowlist is a regex array — but it lives per-host in hosts.yaml, not hardcoded in the tool. The tool itself just asks the registry "is this command allowed on this host?" and refuses if no, runs if yes. One source of truth, reviewable in one place.

What this buys you

The day's stats are downstream of the design, not the other way around. Six commits, 16 tools, 40 passing unit tests, typecheck + build clean, CI matrix green on Node 20 and 22. v0.1.0 cut and registered in .mcp.json by end of day.

That pace is possible because the cross-cutting decisions were nailed before any tool got written. Every tool was a fill-in-the-blanks exercise: validate args, look up host, check allowlist, build command, run with output cap, return dry-run-or-result. No tool needed to argue with the safety model — it just had to satisfy it.

The opposite path — build tools first, retrofit safety after — is how MCP servers end up with five different error paths, three different ACL styles, and the one tool that forgot to cap its output. Designing the invariants before the surface area is what keeps the surface area honest.

There's a second-order benefit: the safety model is the spec. When a future tool gets added — say server.nginx.reload or server.zfs.snapshot — there's no design meeting. The author reads the 7-point list, picks the matching pattern (validate-before-reload for nginx, dry-run for zfs), drops it into a host's allowed_commands, and ships. The cognitive load of "is this safe enough?" got paid down once, on day one.

This matters more for a Model Context Protocol (MCP) server than for ordinary tools because the caller is a model, not a human. A human reading server.exec documentation would hesitate before running rm -rf /srv/braves. A model that just learned the tool exists will happily fire it if the prompt nudges that way. The guardrails are the layer that doesn't depend on the caller having good judgment.

Also shipped

claude-code-plugins (PRs #762/#763/#764): CI hardening — eslint + prettier added as blocking gates (first PR of a multi-PR cleanup), human-triggered auto-merge disabled (dependabot bumps still auto-merge), and nine historical AA-AACR audit files from December 2025 committed to the record.
intentional-cognition-os: v0.2 dogfood continued — paraphrase_robustness metric landed in verify.py, ask-loop.py extracted as a standalone helper with --paraphrases, bank.py schema library plus ADRs 029-032, release v1.2.5, wiki/citation resolution against the workspace cache (closes h99). Continuation of yesterday's zero-to-five FTS fallback arc.
pipelinepilot: Firebase Cloud Functions standardized on Gen2 Node 20 ESM (firebase-admin import fixed), orchestrator wrapper added with sync query(**kwargs) and pinned cloudpickle, Python smoke for the Vertex AI Reasoning Engine, beads tracking initialized, .env patterns added to .gitignore.
hybrid-ai-stack + intent-genai-project-template: Gemini PR-review fixups — flake8 violations cleared, three pre-existing security defects in redact/sanitize/routes closed, mypy + ruff cleanup.

Intent Catalog: Six Phases from Empty Repo to Production Control Plane — the sibling shape: empty repo to working control plane in a small number of clean phases.
Guidewire MCP v0.1.0: Carrier-Native Server Blueprint — another foundation-ship MCP post, same v0.1.0 framing.
A v1.0 Is a Gate, Not a Tag — release-gate discipline; the same logic that says "design the invariants first" applies to "earn the version number."

Five Tags, Zero Ships: How an Auto-Release Workflow Lied for a Whole Day

Jeremy Longshore — Sun, 24 May 2026 13:00:27 +0000

Five GitHub tags. v1.0.4 through v1.1.0. Five green checkmarks on the workflow. Five formatted release notes. The npm registry stayed at v1.0.5 the entire time.

This is what it looks like when a release workflow ships tags without shipping code. Every observable surface said "done" except the one that mattered — the registry. The bug wasn't in one place; it was three independent failures that combined to make the lie convincing.

What the Checkmarks Promised

gh release list showed all five tags with formatted changelogs. The workflow run logs were entirely green. If you ran npm install -g intentional-cognition-os, you got v1.0.5. No error. No warning. Silently wrong for anyone relying on v1.0.5+, silently right for everyone else.

The pattern repeated across the morning: commit → auto-release fires → tag appears → npm registry unchanged. The workflow was perfectly honest about tagging. It just wasn't releasing anything.

Bug 1: Tests That Passed by Lying

The "Verify readiness" step was:

- name: Verify readiness
  run: pnpm test || true

The || true is the tell. Every test failed. Failed to resolve entry for package @ico/types — the workspace packages hadn't been built yet, so pnpm test resolved nothing, threw hard errors, and the || true swallowed them all. The workflow saw exit code 0 and kept going.

In a monorepo, the build step is not optional ceremony. The test runner needs the workspace packages to be built first. The fix:

- name: Verify readiness
  run: |
    set -e
    pnpm build
    pnpm test
    pnpm lint
    pnpm typecheck

set -e means any non-zero exit stops the workflow. If tests fail after the build, you find out. If the build fails, you stop. Lint and typecheck went into the same step because they were already in the local pre-push hook; the only reason to keep them out of the release gate is laziness or speed, and a release gate is the wrong place to optimize either.

Bug 2: Nine Version Sources, Six Ignored

Nine surfaces emit a version string in this repo: root package.json, version.txt, CHANGELOG.md, the five workspace package.json files (packages/cli, packages/kernel, packages/compiler, packages/types, packages/benchmarks), and the runtime constant at packages/kernel/src/version.ts. The workflow bumped three of them — root, version.txt, CHANGELOG.md — and silently left the other six behind.

Result: root said 1.0.4, workspace packages said 1.0.3. Root said 1.0.5, workspace said 1.0.4. Drift every run. ico --version told users the workspace's number, not the tag's.

Lock-step monorepos need single-source-of-truth version sync. A helper that picks up the six the workflow was missing:

bump_pkg_json() {
  local file=$1
  local version=$2
  node -e "
    const fs = require('fs');
    const pkg = JSON.parse(fs.readFileSync('$file', 'utf8'));
    pkg.version = '$version';
    fs.writeFileSync('$file', JSON.stringify(pkg, null, 2) + '\n');
  "
}

bump_pkg_json package.json "$VERSION"
for pkg in packages/*/package.json; do
  bump_pkg_json "$pkg" "$VERSION"
done
sed -i "s/export const VERSION = '.*';/export const VERSION = '$VERSION';/" \
  packages/kernel/src/version.ts

All nine sources now move together. ico --version reports the truth.

Bug 3: The Step That Wasn't There

The workflow tagged releases. It never published to npm. There was no npm publish step. That's not a typo — the workflow was complete without it. Every release ran. Every release skipped the one thing that makes it a release.

Here's what belongs after "Create GitHub Release":

- name: Publish to npm
  env:
    NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
  run: |
    set -e
    if [ -z "$NPM_TOKEN" ]; then
      echo "NPM_TOKEN not set — skipping publish"
      exit 0
    fi
    if npm view "intentional-cognition-os@$VERSION" version 2>/dev/null; then
      echo "intentional-cognition-os@$VERSION already on npm — skipping"
      exit 0
    fi
    echo "//registry.npmjs.org/:_authToken=$NPM_TOKEN" > ~/.npmrc
    pnpm --filter intentional-cognition-os publish --no-git-checks
    sleep 5
    npm view "intentional-cognition-os@$VERSION" version

Three guards, all in the script — not in the step's if: condition. (Step-level env: isn't available to that step's own if: in GitHub Actions, so if: env.NPM_TOKEN != '' would always evaluate false. The check belongs inside run:, where the env is real.) Token presence fails safe if it's missing. Idempotency skips if already published (covers manual publishes). Post-publish verification re-queries the registry to confirm it landed.

A release workflow that doesn't end with a verifiable artifact in the registry isn't a release workflow. It's a tagging workflow with extra steps.

The State Behind the Process

Fixing the workflow forward didn't fix the present. When the workflow was corrected (commit 7681dd5), main was drifted: root at 1.1.0, workspace at 1.0.5. Users running ico --version got 1.0.5. One-time backfill in commit c651de8 aligned all nine version sources to 1.1.0. Then verified: pnpm build succeeded, pnpm test 1,210/1,210 passing, ico --version → 1.1.0.

Process bugs leave state behind. Fixing the process doesn't heal the damage. You clean it up separately.

The Three-Bug Pattern

Every CI/CD pipeline that ships has these three failure modes available:

Quality gates that pass on failure (|| true, swallowed errors). Fix: set -e and explicit step order.
Monorepo workspaces with distributed version state. Fix: single-source-of-truth version sync in the workflow.
A release workflow that doesn't end with verification the artifact reached the registry. Fix: final step that queries the registry and confirms.

The icos release workflow had all three. The checkmarks lied because the workflow wasn't designed to catch itself lying.

Also Shipped 2026-05-19

Daily-log convention — the rest of the day, in one paragraph each. Not connected to the release-workflow thread; logged here because they happened on the same git day.

claude-code-slack-channel v2 cluster — 4 PRs merged with enterprise governance substrate framing. RFC 8785 JCS interop vectors (#175), cross-tier shadow detection (#176), journal v2 Ed25519 signing (#177), strip denied tool-call detail (#178).
kobiton R3 close-out — deliverable final review, Blog 3 rewrite, 5 close-out PRs merged.
claude-code-plugins partner portal — Kobiton and Nixtla brand integration, Killer Skill of the Week refresh.
intentional-cognition-os test infra — Intent Solutions Testing SOP layers L0-L7 installed (.husky/, dependency-cruiser, stryker, RTM/PERSONAS/JOURNEYS docs). 3,447 insertions in commit e0efdee.

v1.0.0: Conditional GO Through a Release Gate — The gate that flagged this path.
Honest Performance Benchmarks for a Paid-API Compiler — Earlier icos work from this release cycle; same repo, different failure mode.

A v1.0 Is a Gate, Not a Tag

Jeremy Longshore — Thu, 21 May 2026 13:00:39 +0000

Two beads were open at the start of 2026-05-18. E10-B11 was the v1.0 release-readiness gate. E10-B12 was the v1.0 release cut, blocked-by-design on B11. Epic 10 was the last epic in intentional-cognition-os (ICO). The release pipeline was wired through /release. Everything that mattered had to clear one ritual.

Five npm releases shipped that day: v0.21.0 → v0.22.0 → v0.22.1 → v0.22.2 → v1.0.0 → v1.0.1. The interesting one is v1.0.0, because the gate said GO with conditions, not GO. And the same-day v1.0.1 is the proof that "GO with conditions" is the correct verdict shape for a real release, not a binary.

The 3× degradation gate

The release ran on top of fresh benchmark infrastructure. 625691e and f7bd287 closed out E10-B06 (performance profiling) with a 500-source large-corpus benchmark. The headline addition was a 3× degradation gate — a configurable cap (default 3.0) that fails the run if per-unit cost at large scale exceeds 3× the moderate-corpus baseline.

The gate is intentionally narrow:

// utils/degradation.ts — gate stays honest by NOT inferring per-unit costs
export function computeDegradation(
  moderatePerUnitMs: number,
  largePerUnitMs: number,
  cap = 3.0
): { ratio: number; pass: boolean } {
  if (moderatePerUnitMs === 0) {
    return { ratio: Infinity, pass: false }; // catch degenerate samples loudly
  }
  const ratio = largePerUnitMs / moderatePerUnitMs;
  return { ratio, pass: ratio <= cap };
}

The runner does per-unit derivation BEFORE calling the gate. Ingest's perFile.medianMs is already per-unit (each iteration was one file). Lint's result.medianMs is whole-workspace, so the runner divides by page count first. Putting that decision in the runner instead of the gate is the difference between "gate that knows what it's measuring" and "gate that guesses at the measurement units."

Results at 500 sources: ingest 1.25× (PASS), lint 0.33× (PASS — got faster at scale, likely amortized constants). The gate had teeth and the system passed cleanly.

The release-readiness checklist (E10-B11, PR #73)

Eight items, verified item-by-item, recorded honestly. No "looks good to me" entries:

CI passes — all 4 jobs green on last 3 main runs
Evals pass — smoke eval clean; retrieval/citation/compilation handlers wired with 30+ unit tests
Coverage targets — PARTIAL. Types 100%, kernel 84.6% (target 90%), compiler 62.3% (target 80%), CLI 45.2% (target 70%)
Docs updated — current per E10-B07/B08
CHANGELOG complete — auto-generated, current through v0.22.0
No critical beads open — only B11 (this) + B12 (release cut, blocked by design)
User journey walkthrough — ico init → status → 14-command CLI surface, live smoke-tested
Performance targets met — ingest 200× headroom, lint 3000× headroom, 3× degradation gate PASS

Verdict: GO with two conditions.

C1: ico --version reported 0.1.0 (a stale kernel constant) instead of the published 0.22.x. Fix in-cut.
C2: Coverage shortfall on kernel/compiler/cli. Documented as post-v1, not blocking. 1,210 passing tests, zero known bugs.

That verdict is the artifact. Most release rituals make GO/NO-GO a binary. The conditional verdict is honest: state the gap, decide if it blocks, ship if it doesn't, document the gap permanently if it doesn't.

What "GO with conditions" actually means

A conditional release verdict is the three-state model: fix what's fixable in-cut, document what isn't, ship anyway. Unlike a binary GO/NO-GO gate that forces a boolean choice, a conditional gate acknowledges that real releases ship with known imperfections. The conditions are documented forever in the release record — no lying about readiness, no pretending gaps don't exist, but no unnecessary delays waiting for the perfect threshold that never comes.

Why not GO/NO-GO binary?

Binary GO/NO-GO encourages two bad behaviors.

Behavior one: lower the bar to ship. "The version-string bug is fine, users will figure it out." The release ships, the operator-visible defect ships with it, and the next person debugging an environment ends up reading the wrong build into their incident postmortem.

Behavior two: delay until the gate is perfect. Coverage targets met on a Tuesday that never comes. Kernel at 84.6% is allegedly not 90%, so v1.0 slips. Then 90% becomes 95%, because some new code landed during the wait. The gate becomes a treadmill.

Coverage at kernel 84.6% / compiler 62.3% / CLI 45.2% with 1,210 passing tests and zero known bugs is shippable. Blocking v1.0 on coverage uplift would have been a bigger lie than shipping with documented shortfalls. The AAR opens C2 as a post-v1 bead for the next planning cycle. The truth is in the record.

C1 is the inverse case — ico --version reporting the wrong number is shippable but ugly, and the fix is small. So fix it in-cut, document it, move on. The gate didn't pretend C1 was fine; it just didn't pretend it was a v2.0-blocker either.

The prescription is a three-part rule, not a two-part one: fix what's fixable in-cut, document what isn't, ship anyway. Binary GO/NO-GO collapses three states into two and loses the most useful one — the "shippable with known imperfections" state where most real releases actually live.

C1 fix: read your own version (PR #74)

packages/cli/src/index.ts had been importing version from @ico/kernel, which exported a hardcoded string. The kernel constant was never maintained in lock-step with the published CLI package — and shouldn't be, since they are independent artifacts on independent release cadences.

// packages/cli/src/index.ts — read from CLI's own package.json
function readCliVersion(): string {
  try {
    const pkgPath = new URL('../package.json', import.meta.url);
    const pkg = JSON.parse(readFileSync(pkgPath, 'utf-8'));
    return pkg.version;
  } catch (err) {
    console.error('[ico] failed to read CLI package.json:', err);
    return '0.0.0-unknown'; // sentinel — CLI keeps working, operator sees clear msg
  }
}
export const cliVersion = readCliVersion();

The try/catch is load-bearing. readCliVersion() runs at module load, BEFORE the process-level error handlers are installed further down the file. An uncaught throw here would surface as a raw Node stack trace and bypass the friendly [ico]-prefixed message convention every other CLI error uses. The sentinel path is what makes this safe to call at import time — the CLI keeps working, the operator gets a legible message, and the bug is visible without crashing.

The test was tightened in the same PR. /^\d+\.\d+\.\d+/ (no end anchor — would accept nonsense like 0.22.1.99) became:

expect(cliVersion).toMatch(/^\d+\.\d+\.\d+(-[\w.-]+)?$/);

Strict semver core plus optional pre-release tag. The previous regex was a one-character bug; the fix is one character plus an opt-in pre-release group.

The cut itself (52fa7a4 → v1.0.0)

The cut commit was tiny: 11 files, +54/-10 lines. It did one thing: aligned all 6 workspace package.json + version.txt + kernel/src/version.ts at 1.0.0.

The auto-release workflow had been bumping the root package.json and version.txt only — internal packages had drifted to 0.1.0 or 0.22.1 depending on history. /release Phase 3 caught the drift. Phase 5 required explicit SHA approval before any push (f1a627b). Phases 6-8 ran atomically.

Verified at v1.0:

1,210 / 1,210 tests pass across 5 packages
Lint + typecheck clean
escape-scan REFUSE=0 CHALLENGE=0 FLAG=0
ico --version reports 1.0.0

The tarball turned out incomplete (v1.0.1, same day)

During the actual npm publish flow, the pack dry-run reported 7 files when expected was 9: dist + package.json, no README, no LICENSE. The CLI's package.json declared:

"files": ["dist", "README.md", "LICENSE"]

But the CLI directory didn't OWN those files. The canonical README.md and LICENSE live at the monorepo root.

Fix landed inline before the real publish:

// packages/cli/tsup.config.ts — copy README + LICENSE at build time
export default defineConfig({
  // ... entry, format, dts, sourcemap ...
  onSuccess: 'cp ../../README.md ../../LICENSE ./',
});

The copies are gitignored (their source of truth is the repo root). v1.0.0 on npm now includes both. No version bump for the build-infra fix itself, but the same day shipped v1.0.1 for the next user-visible change.

This is the test of whether "GO with conditions" was the right shape. A binary GO/NO-GO ritual would have caught the version string (C1) and either fixed it before re-running the whole gate or punted to v1.0.1. The conditional model said: ship, here's what we know is imperfect. When the tarball turned out incomplete during the actual publish — a discovery that couldn't have been made during gate verification, because it only surfaces in the publish pipeline itself — the answer was just: ship v1.0.1 the same day. No drama. No "release is broken" panic. The model already accepted that real releases generate follow-on releases.

AAR same day

d17e10e docs(aar): v1.0.0 release after-action report landed within hours. Three lessons-for-next-release, captured while they were still warm:

Beads JSONL/Dolt sync flapping during multi-PR sessions — repeated need to re-close beads after merges. Filed as a follow-up to investigate the sync ordering.
Auto-release workflow bumps root + version.txt only — should bump packages/*/package.json in lock-step. The 11-file cut commit was entirely correcting drift the workflow could have prevented.
/release skill execution worked as designed — Phase 0 surfaced no blockers, Phases 1-3 caught the version drift, Phase 5 required SHA approval, Phases 6-8 atomic.

Same-day AAR is non-negotiable. The version-drift issue, the tarball issue, the conditional-verdict pattern — all of them lose 80% of their teaching value if you write the AAR a week later, after the warm memory of "wait, why didn't the workflow catch that?" has faded into "yeah, we shipped, it was fine."

Also shipped

The release gate constrained the v1.0.0 cut, not the working day. Three other repos kept moving in parallel — exactly the behavior the conditional-verdict model is designed to enable. A release that takes the whole org offline isn't a release ritual; it's an outage.

hustle: Phase 3 auth landed in three commits — NextAuth + Drizzle/SQLite infrastructure, dashboard cutover, password reset flow. Coordinated migration from the previous auth stack on a single feature branch.
claude-code-slack-channel: ACP session/cancel boundary adapter extracted into a module, and JSON-RPC id widened to nullable per spec §5.1 (#172, #173).
claude-code-plugins: Six PRs — repo quality audit, private vulnerability reporting enabled, validator discovers root-level SKILL.md (Anthropic-spec layout), slack-channel mirror stopped stripping upstream tests, blog cross-post infra fix.

Honest perf benchmarks for a paid-API compiler — yesterday's post on the benchmark infrastructure that fed this release gate
Five releases in fifteen minutes: Mandy cutover and freeze break — earlier five-releases-in-a-day pattern
GitHub release workflow: uncommitted changes and semantic versioning — related release-engineering theme

Honest Perf Benchmarks for a Paid-API Compiler

Jeremy Longshore — Wed, 20 May 2026 13:00:40 +0000

intentional-cognition-os is a TypeScript "compiler" — markdown sources go in one end, a structured artifact comes out the other, and several of the middle stages call paid Claude APIs to do the cognitive work. Up to today there were zero performance gates on any of it. No baseline, no regression alarm, no "did that refactor make ingest 4× slower" check.

The benchmark suite that landed across four PRs answers two design questions that had to be settled before a single line of timing code got written:

How do you compare numbers across machines when half the corpus is randomly generated text?
What do you do about the steps that cost real money on every run?

Get either answer wrong and the benchmark suite is worse than no benchmark suite — it produces numbers that look authoritative and aren't.

The corpus has to be byte-identical

The first scenario — ingest — needs a corpus. Hand-curated fixtures committed to disk were considered and rejected: they don't scale, they go stale, and they encode whoever-wrote-them's idea of "representative." A generator is the right answer, but a generator has to be deterministic or before/after diffs are noise.

The generator uses a seeded mulberry32 PRNG and pulls UUIDs from the same stream:

function mulberry32(seed: number) {
  return function () {
    let t = (seed += 0x6d2b79f5);
    t = Math.imul(t ^ (t >>> 15), t | 1);
    t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

function seededUuidV4(rand: () => number): string {
  // 16 bytes from the seeded stream, version + variant nibbles set per RFC 4122
  const bytes = new Uint8Array(16);
  for (let i = 0; i < 16; i++) bytes[i] = Math.floor(rand() * 256);
  bytes[6] = (bytes[6] & 0x0f) | 0x40;
  bytes[8] = (bytes[8] & 0x3f) | 0x80;
  return formatUuid(bytes);
}

The non-obvious trap is crypto.randomUUID. It would have looked correct, passed every unit test, and silently produced different UUIDs on every run — so every "identical" corpus would have differed in the front-matter id field. That breaks ingest's content-hash cache in different ways on different machines. Same seed, same count, same body-word count yields byte-identical output everywhere. That's the contract.

One more gotcha worth a sentence: the corpus generator writes front matter through gray-matter, which quotes string values. The compiler's wiki-page validator uses a hand-rolled YAML parser that does NOT strip quotes — so wiki fixtures emit all values unquoted. A quoted compiled_at would arrive at Zod's datetime check with literal " characters in it and fail. Two parsers, two rules, documented inline at the parser boundary.

An API key is not consent

The render, compile, and ask scenarios call Claude. Running them on every CI pass would either drain a budget or quietly stop running when the budget hit zero. Neither is acceptable.

The gate is two env vars, both required:

ANTHROPIC_API_KEY=sk-ant-... \
ICO_BENCH_INCLUDE_CLAUDE=1 \
pnpm --filter @ico/benchmarks bench

From PR #70's design notes, kept verbatim because the framing matters:

The double gate is intentional. An API key alone is not consent — many developers have it set for normal CLI use. ICO_BENCH_INCLUDE_CLAUDE is the explicit "yes, burn tokens on this benchmark run" signal.

This pattern shows up elsewhere — CI=true plus RUN_E2E=1, prod credentials plus --really-really-yes. The shape is the same: one signal proves capability, the second proves intent. A single-gate design fails open the first time someone forgets which shell they're in.

Skipped is not zero

The interesting design call was what to do when the gate is closed. The wrong answers:

Don't run, don't record. Trend tooling then can't tell "we stopped running render" from "render still passes."
Record a zero. Trend tooling thinks render got infinitely fast and stops alarming.

The right answer: record the scenario as skipped: true with a stable skipReason. ScenarioRecord is Partial<CommonTiming> so the timing fields legitimately don't exist on skipped records:

{
  "name": "render",
  "skipped": true,
  "skipReason": "ICO_BENCH_INCLUDE_CLAUDE not set",
  "git_sha": "9c14f02",
  "node": "v22.21.0",
  "platform": "linux-x64"
}

A baseline-comparison script can now answer three different questions instead of two: did this scenario regress, did it improve, or did it not run? Skipped runs stay visible in the JSON timeline. They don't pollute the histogram, but they prove the scenario still exists and the runner saw it.

The four PRs, briefly

PR #68 scaffolded the packages/benchmarks/ workspace, the corpus generator, a bench() timer with warmup + N-iteration median + RSS delta, and the runner that captures git SHA, Node version, and platform into results/<iso>-<sha>.json. The results/ directory is gitignored except .gitkeep — baselines get tracked explicitly, not by accident.
PR #69 added the lint scenario and moved runLint, scanWikiPages, extractWikilinks, detectOrphans, LintResult, and SchemaError out of packages/cli/src/commands/lint.ts into a new packages/compiler/src/lint.ts. The function only composes compiler + kernel primitives and has no CLI dependency — it belonged in the compiler the whole time. The CLI's lint command shrunk to a thin wrapper around commander wiring and renderLintReport. Side fix: extractWikilinks had a module-level /g regex whose lastIndex carried state between calls — the same class of bug that landed in PR #67 the day before. Fixed by constructing the regex per call.
PR #70 added the render scenario and the double-gate.
PR #71 added compile and ask, each using the same gating pattern. Roughly 70 lines of additions across both files — the gate had already done the hard work.

Why not the obvious alternatives

Vitest's built-in bench was considered. It does microbenchmarks well and integrates with the existing test runner. It does not produce the JSON timeline shape needed for cross-run comparison, and bolting that on means owning the storage layer anyway. Build it once, build it right.

Committing fixture corpora to disk was considered. They go stale, balloon the repo, and encode one author's idea of "moderate." The seeded generator is reproducible AND parameterizable — same determinism guarantee, no committed binary blobs.

Running Claude scenarios always was considered for about a minute, then rejected on cost grounds. Even with caching, a benchmark suite that costs $2 per run on a busy day stops getting run.

What the numbers say

Three scenarios ran on the dev box this afternoon (Claude-gated ones skipped because the opt-in wasn't set):

Scenario	Median	Target	Headroom
ingest (per-file, 50 sources × 500 words)	~9 ms	< 2 s	220×
lint (50 sources + 30 wiki pages)	~12 ms	< 30 s	2400×
render	SKIPPED (no opt-in)	—	recorded

The headroom isn't the point — those targets are deliberately generous because the goal is regression detection, not perf bragging. The point is that there are now numbers to regress against.

Also shipped today

claude-code-plugins repo audit. A 232-line audit landed at 266-RA-AUDT-repo-quality-audit-2026-05-17.md cataloguing a broken /about route, missing 404 handling, 14 stale MS-OLDV files still claiming v1.0.0 while the repo is at v4.30.0, and notebook content teaching the old 6-required-fields skill spec when the current spec requires 8. The first commit incorrectly flagged the wiki as empty, because gh api repos/.../wiki returns 404 even when the wiki has content — that endpoint isn't a content probe, it's a metadata probe with bad error semantics. Followup commit cloned the wiki, found 23 pages, and refreshed all of them with current numbers. Lesson noted inline: don't use API existence probes as content probes. Clone and read.

claude-code-slack-channel threat model. Added T11 (EchoLeak — instructions exfiltrated via legitimate-looking message replies) and invariant #7: admin verbs are not chat content. An operational key-management doc for the audit-signing key landed alongside the threat model update.

The transferable pattern

Five scenarios in source tree, three actively measured, two gated behind explicit consent. The numbers that get reported are honest because the inputs are reproducible and the skipped runs are visible. Forget the opt-in flag and three scenarios show up as skipped in the JSON — they don't disappear, and they don't pretend to be zero.

Any benchmark suite that mixes deterministic and paid steps needs all three pieces: a deterministic corpus that survives machine swaps, an opt-in gate strong enough to mean something, and a record shape that distinguishes "didn't run" from "ran fast." Miss one and the suite will quietly lie to you the first time someone forgets which mode they're in. The lie is worse than the gap it filled.

Five Silent Failures in One Day — the regex lastIndex bug that re-appeared in PR #69 was one of these.
Deterministic-First, LLM-Advisory CI — same principle: the deterministic gate decides, the paid gate informs.
Transitive CVE Clearance: A Dual-Layer Pattern — the double-gate is the same shape as that two-layer defense.

Five Silent Failures in One Day

Jeremy Longshore — Tue, 19 May 2026 13:00:41 +0000

A silent failure is when a tool reports PASS without doing the work it was supposed to do — the legitimate empty-set case and the broken-but-silent case produce identical output, and nothing downstream can tell them apart.

A green check is not evidence of work. It is evidence that whatever ran did not raise an error. Those are different claims, and on 2026-05-16 the difference surfaced five times in five unrelated systems before lunch.

The pattern is the same in all five: a tool reported PASS without doing the work it was supposed to do. Not a wrong answer — no answer, dressed up as a correct one. The legitimate empty-set case and the broken-but-silent case produced identical output. CI was green. Reviewers saw nothing to push back on. The signal that something was wrong came from downstream consumers noticing the work was missing.

The five instances, in the order they were found:

A CI prescreen that ran on zero plugins and called itself green
A .gitignore rule that silently dropped plugin configs from every commit
Prettier that reformatted an 11,000-line catalog and exited 0
An SSH deploy that succeeded by doing nothing
A regex that quietly skipped matches because the /g flag left state behind

Each one shipped past code review. Each one was caught by a downstream user, not by the gate that was supposed to catch it. Each one has now been re-armed with a guard whose job is to assert the work actually happened — not to assert that the command exited zero.

1. The prescreen that ran on zero plugins

Repo: claude-code-plugins, PR #730.

The pr-prescreen.yml workflow's "Compute changed plugin paths" step combined gh api --paginate with --jq in a single pipe:

- name: Compute changed plugin paths
  run: |
    gh api --paginate \
      "/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files" \
      --jq '.[].filename' \
      | grep -E '^plugins/[^/]+/' \
      | cut -d/ -f1-2 \
      | sort -u > changed-plugins.txt || true

This works on every local shell. On the GitHub Actions runner, the --paginate + --jq combination silently produced empty stdout. No error. No exit code. Just nothing on the pipe. The downstream grep | cut | sort -u happily processed zero lines and wrote an empty file. The trailing || true swallowed any failure that might have escaped the pipeline.

The classifier then read changed-plugins.txt, saw zero entries, and emitted PASS: no plugin paths matched the PR diff. Two external PRs — #726 and #728, the first contributions through the new pipeline — both landed false PASS verdicts on PRs that obviously added new plugin directories.

The fix is two changes and a guard:

- name: Fetch changed files
  run: |
    gh api --paginate \
      "/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files" \
      > pr-files.json

- name: Extract plugin paths
  run: |
    jq -r '.[].filename' pr-files.json \
      | grep -E '^plugins/[^/]+/' \
      | cut -d/ -f1-2 \
      | sort -u > changed-plugins.txt

- name: Sanity guard
  run: |
    if jq -r '.[].filename' pr-files.json | grep -qE '^plugins/'; then
      if [ ! -s changed-plugins.txt ]; then
        echo "HARD_BLOCK: PR touches plugins/ but extraction produced zero dirs"
        exit 1
      fi
    fi

Splitting gh api --paginate from jq removes the pipe-buffering interaction that ate stdout. Dropping the blanket || true lets real errors propagate. The third step is the actual fix: it asserts that if the PR diff touched any plugin path, the extraction must have produced at least one row. "I found nothing" becomes "I would have found something — fail loud."

2. The gitignore that ate plugin configs

Repo: claude-code-plugins, PR #733.

The root .gitignore contained one line that was never meant to apply globally:

.mcp.json

The original intent was dev-local — devs sometimes drop a .mcp.json at the repo root for personal MCP servers. The pattern matched everywhere. Three plugins — slack-channel, pr-to-spec, x-bug-triage — had a .mcp.json on disk because the mirror sync wrote them, and git silently never tracked any of the three. The mirror produced the file. The working tree showed the file. git status showed it as ignored. Nothing red anywhere.

Plugins without their .mcp.json fail the MCP handshake at install time. Claude Code can't determine how to spawn the server. The plugin loads, registers nothing, and the user sees commands that do nothing.

A second silent failure lived in the same PR. The mirror's sources.yaml listed source files explicitly:

plugins/x-bug-triage:
  sources:
    - server.ts
    - lib.ts

server.ts imports journal.ts, manifest.ts, policy.ts, supervisor.ts — none of which were in the allow-list. The mirror shipped a non-functional server, not because anything errored, but because the include list silently skipped the missing files. No "file not in sources" warning. No diff check. Just a partial build that compiled because the imports themselves were valid module references at type-check time but missing at runtime.

The fix:

# .gitignore
.mcp.json
!plugins/**/.mcp.json

plugins/x-bug-triage:
  sources:
    include: "*.ts"
    exclude: ["*.test.ts", "*.spec.ts"]

The negation rule re-tracks plugin configs. The glob-with-exclude replaces named-file allow-lists with a pattern that can't silently miss a new file. The three affected .mcp.json files were force-added in the same commit.

3. Prettier that reformatted 11,000 lines and exited 0

Repo: claude-code-plugins, PR #730 (same PR as the prescreen failure).

.claude-plugin/marketplace.extended.json is the canonical plugin catalog — eleven thousand lines, hand-formatted with deliberate multi-line keywords arrays for git-diff hygiene:

{
  "name": "example-plugin",
  "keywords": [
    "ci",
    "validation",
    "marketplace"
  ]
}

A contributor's format-on-save action ran prettier across the catalog. Prettier collapsed every keyword array to a single line:

{
  "name": "example-plugin",
  "keywords": ["ci", "validation", "marketplace"]
}

The JSON was still valid. Prettier exited 0. The validate-plugins.yml workflow loaded the catalog, parsed it, ran every entry through the schema — all green. The actual diff was +1 plugin entry, -1,200 lines of reformatted catalog. Every other in-flight PR's merge base was now unrecoverable without rebase-and-reformat.

The fix has two parts. First, .prettierignore:

.claude-plugin/marketplace.extended.json

Second, an active line-budget guard at scripts/check-catalog-format.py:

def expected_line_delta(base_catalog, head_catalog):
    with open(base_catalog) as f:
        base = json.load(f)
    with open(head_catalog) as f:
        head = json.load(f)
    base_by_name = {p["name"]: p for p in base["plugins"]}
    head_by_name = {p["name"]: p for p in head["plugins"]}

    added = set(head_by_name) - set(base_by_name)
    removed = set(base_by_name) - set(head_by_name)
    modified = {n for n in head_by_name & base_by_name
                if head_by_name[n] != base_by_name[n]}

    # Average plugin block is ~30 lines.
    return (len(added) + len(removed) + len(modified)) * 30

actual_delta = abs(file_line_count(head) - file_line_count(base))
expected = expected_line_delta(base, head)
budget = expected + 300  # slack for inline edits

if actual_delta > budget:
    sys.exit(f"FAIL: catalog diff {actual_delta} lines, budget {budget}")

The guard parses both catalogs structurally, computes the expected line delta from the actual content changes, and rejects PRs where the file delta exceeds that by more than 300 lines. "The file is still valid" becomes "the diff is the size we expected from the work that was claimed."

4. The SSH deploy that succeeded by doing nothing

Repo: hustle, PR #40. Documented in the intentsolutions-vps-runbook AAR for Phase 2.5 of the VPS migration.

The new Hustle VPS deploy workflow merged green. The first auto-deploy reported success. The container on the VPS was untouched.

The canonical reusable VPS deploy workflow is one SSH call:

- name: Deploy
  run: ssh ${{ env.DEPLOY_USER }}@${{ env.DEPLOY_HOST }}

There is no command argument. The whole architecture relies on a command="..." force-command directive in authorized_keys to bind the deploy key to a specific script. Connect with the key, the forced command runs, deploy happens, connection closes.

The hustle-deploy user's authorized_keys had no force-command. Plain ssh user@host with no command and no force-command opens an interactive session. The runner has no TTY. The session sits idle for a moment, the server times out the silent connection, exit 0. From the runner's perspective: SSH connected, SSH closed cleanly, deploy step SUCCESS. From the VPS's perspective: a key authenticated, nothing happened, the session ended.

The fix is a deploy script and a force-command lock:

# /usr/local/sbin/deploy-hustle
#!/bin/bash
set -euo pipefail
cd /srv/hustle
git fetch origin
git reset --hard origin/main
docker compose pull
docker compose up -d --remove-orphans
docker compose ps

# /home/hustle-deploy/.ssh/authorized_keys
command="/usr/local/sbin/deploy-hustle",no-port-forwarding,no-X11-forwarding,no-pty ssh-ed25519 AAAA... deploy@github

Now there is no path where the SSH channel can do nothing. The forced command runs or the key fails to authenticate. The second deploy ran the script end-to-end, recreated the container, and produced visible log output the runner could grep.

The generalization matters more than the fix. Every Docker-variant deploy in the fleet that depends on a force-command and doesn't have one is silently broken in the same way. lilly-75-holy and braves-booth are flagged for audit; partner-portals and claude-code-plugins-plus-skills are safe — both have the force-command directive in place. The fleet sweep is tracked as a follow-up bead off the P7 Stage C epic, not folded into this post.

5. The regex that skipped matches because `/g` left state behind

Repo: intentional-cognition-os, PR #67 (a Gemini review followup on E10-B03).

Two module-level constants:

const SOURCE_RE = /\[\^src:([^\]]+)\]/g;
const WIKILINK_RE = /\[\[([^\]]+)\]\]/g;

Used in two back-to-back RegExp.exec loops to iterate citation markers in a body of text:

export function extractCitations(body: string): Citation[] {
  const out: Citation[] = [];
  let m;
  while ((m = SOURCE_RE.exec(body)) !== null) {
    out.push({ kind: "source", id: m[1] });
  }
  while ((m = WIKILINK_RE.exec(body)) !== null) {
    out.push({ kind: "wikilink", id: m[1] });
  }
  return out;
}

RegExp instances with the /g flag carry a mutable lastIndex between calls. The exec loop is supposed to walk it to the end and let the final non-match reset it to 0 — but any code path that exits the loop early, throws mid-iteration, or runs concurrently on the same regex object leaves lastIndex mid-string. The next call to extractCitations starts searching from wherever the last one stopped.

The citation handler kept reporting "verified" because the missed citations were not checked at all — not flagged as missing, not flagged as wrong. They were invisible. Whichever entries fell before the carried-over lastIndex were skipped silently, every time.

The fix:

export function extractCitations(body: string): Citation[] {
  // Required: SOURCE_RE and WIKILINK_RE are module-level /g regexes.
  // Reset lastIndex on entry so prior loop state cannot cause this call
  // to start mid-string and silently skip matches.
  SOURCE_RE.lastIndex = 0;
  WIKILINK_RE.lastIndex = 0;

  const out: Citation[] = [];
  let m;
  while ((m = SOURCE_RE.exec(body)) !== null) {
    out.push({ kind: "source", id: m[1] });
  }
  while ((m = WIKILINK_RE.exec(body)) !== null) {
    out.push({ kind: "wikilink", id: m[1] });
  }
  return out;
}

The comment is load-bearing. Without it, the next refactor pulls the resets out as "redundant" and the silent skip comes back. Six regression tests pin the invariant: prebuilt-index honored, batch aggregation correct, 100 sequential calls return identical output, two interleaved bodies (one long, one short) stay independent of each other.

The shape of a silent failure

All five share the same anatomy. There exists a legitimate no-op outcome — no plugin paths matched, no files to include, no formatting changes needed, no command to run, no remaining matches in the string. The error path produces an observable state identical to the legitimate no-op. The downstream consumer cannot tell which one it got.

The fixes are not better error handling. The fixes are active assertions about the work that was claimed:

prescreen: if files matched the trigger, the extraction must have produced rows
gitignore + allow-list: plugin configs must reach the tree, not just the working directory — and source allow-lists must fail on missing imports, not silently ship a partial build
prettier: the diff size must match the structural work
SSH deploy: bind the command to the key — make it impossible for the channel to do nothing
regex: reset state to a known precondition before every call, and pin that contract with a test

The common verb in every fix is assert, not handle. The bug was not that errors weren't caught. The bug was that there was no point in the pipeline where the system stated, in code, what counted as the work actually being done.

The hardest silent failures to catch are the ones where the tool's success state and its silent-failure state are observationally identical. That is the category. Once auditing for it begins, more keep surfacing — most CI pipelines have at least one step that exits 0 whether or not it did anything, and most of them are downstream of a step that can legitimately produce empty output.

Silent failures don't get worse over time. They get more confident. Each green check trains the audit instinct to skip them, and the audit instinct is the only thing standing between the build status and the truth.

Deterministic-first, LLM-advisory CI — the broader argument for keeping reject/accept decisions in code that can be reasoned about, with model output as advisory signal
Three guards against shipping slop — earlier examples of the same assert-the-work pattern in plugin merges
Two false-positive fixes, same root cause — when two unrelated bugs share an underlying shape

DEV Community: Jeremy Longshore

Honor the Gate When the Verdict Is Inconvenient

The Thesis

Story 1: Semantic-Flux—Honoring a Pre-Registered STOP on a $4 ML Signal Test

Story 2: agent-governance-plane—Deleting a CI Gate Rather Than Faking It Green

The Parallel

Also Shipped

The Cost Asymmetry

Related Posts

Human-in-the-Loop Is a Delivery Guarantee, Not a UI Feature

The reframe: this is distributed systems, not UI

The reply outbox: CCSC's durable delivery pattern

Why not the obvious approach?

The approval receiver: AGP's Socket Mode pattern

The transferable rule

Also shipped

Related Posts

The Wrong Product, Built Perfectly

The build that worked

The detour

The reversal

What the reversal cost

Why a clean pipeline can't save you

The principle: couple to what you trust

Tradeoffs, honestly

Also shipped

Related posts

From one adopter to two: the discovery-affordance spec just got named

The validation moment

Why "spec, not opinion"

What this means for tonsofskills.com users

What comes next: install_source_url

The take-home

Vite Dev Server in Production: The 871-Byte Tell

The 871-byte tell

Three smoking guns (the 60-second checklist)

Why this happens

The fix: a two-stage Dockerfile

The compose change

Verification: predicted before / after

What would have caught this earlier

The transferable lesson

The Unicode Layer Your Validator Can't See

The threat model has three shapes

The artifact

Tiered severity, not a single verdict

Severity tiers at a glance

The BOM exception

Make the invisible visible in the log

Ship report-only, then ratchet

The result

Also shipped

Related posts

Self-Expiring Report-Only CI Gates: From Advisory to Enforced

What Is a Self-Expiring Report-Only CI Gate?

The Meta-Gate: Enforced Deadlines

The Campaign: Nine PRs, Zero Rot

PR A (#764): ESLint + Prettier

PR B (#765): Ruff + ruff-format

PR C (#767): Markdownlint (report-only)

PR D (#766): Root directory hygiene

PR E (#768): Shellcheck — where the pattern earned its keep

PR F (#769): TypeScript coverage + codeblock syntax

PR G (#770): Widened test loop

PR H (#771): Codeblock syntax cleanup

PR I (#772): Markdownlint — the last gate flips

The Proof: External Contributions Now Bind

How to Adopt This

Safety Model First: 16-Tool Ops MCP, One Day

The 7-point safety model

The host registry as the access-control boundary

Why not let the tools do ACLs?

The shared SSH client pool

Why not use ssh-agent?

The tool catalog

The Contabo wrapper has a quirk worth knowing

Why not allowlist commands inside server.exec itself?

What this buys you

Also shipped

Related Posts

What comes next: `install_source_url`

Why not allowlist commands inside `server.exec` itself?

5. The regex that skipped matches because `/g` left state behind