Your Agent Pipeline's Review Gate Should Be Code, Not a Prompt Convention

Ammar — Tue, 25 Aug 2026 19:18:30 +0000

Your Agent Pipeline's Review Gate Should Be Code, Not a Prompt Convention

ZOdyssey is a code-enforced orchestration pipeline for ZCode (for now — the pattern is harness-portable), and its center of gravity is not dispatch: it is getting the plan right before any code moves. Acceptance criteria are executable and confirmed with the user; a plan-linted, nonce-bound review verdict is the only key that unlocks product code. Every code block below is quoted from the repo, with its source lines named.

The pipeline shape is not the news — orchestrators already share it, learned from omo and credited. The news is that most leave planning and review as prompt conventions: the model is told to plan carefully and not to edit before review passes, but nothing actually stops it.

The 8-phase pipeline. The REVIEW node is the only door to EXECUTE — and the hook, not the prompt, holds it shut.

The failure that motivated it: a gate deleted twice

The roadmap is blunt: the Bash write-gate was deleted twice — v0.2.0 replaced 170 lines with if (isBash) exit(0); — and three independent external audits missed it (docs/ROADMAP.md:22-25). Audits verify the diff in front of them; none re-checks an invariant established two releases earlier (docs/ROADMAP.md:27-28).

The regression suite now carries its justification in its header — skills/odyssey/hooks/pre-tool.bash-gate.test.mjs:4-17:

// WHY THIS FILE EXISTS: the Bash gate has been silently deleted TWICE.
//   v0.1.1 (5c99927) shipped it deleted — the author's local ZODYSSEY_UNGATE_BASH=1 copy was
//                    mirrored to the public repo verbatim.
//   v0.1.2 (433c037) restored it and wrote a public post-mortem.
//   v0.2.0 (e57b01b) deleted it AGAIN (-170 lines -> `if (isBash) exit(0);`), and three
//                    independent external audits did not notice, because each audit reviews the
//                    diff in front of it and none re-checks an invariant established two
//                    releases earlier.
//
// Audits verify that code matches its documentation locally. They are not a regression suite.
// This is the regression suite. If the gate is removed a third time, this file fails.
//
// Every assertion below is an invariant the README/DESIGN.md already CLAIM. Nothing here is new
// policy — it is the existing promises, made executable.

The pipeline is the part you already know

Eight phases, nothing exotic, quoted from the README (README.md:82-91):

  -1  PRIME        prompt-master refines the raw task into a sharp brief;
                    measurable criteria get ONE user-confirmation round (never blocks)
   0  TRIAGE       trivial → just answer; standard → single-track; architecture → full pipeline
   1  CONSULT      metis classifies intent, surfaces questions/risks
   2  PLAN         prometheus writes <repo>/.zcode/plans/<slug>.md  (cannot edit product code)
   3  REVIEW       momus returns OKAY | REJECT + blockers   ←  THE ENFORCED GATE
   4  EXECUTE      sisyphus-junior per todo, parallel-by-default, scope-locked to the plan's Files:
   5  VERIFY       run each todo's executable acceptance criteria
      automatic   compact.mjs — fires at final entry for large runs; F1–F5 consume a brief, not the full doc set
   6  FINAL WAVE   F1 plan-compliance · F2 code-quality · F3 manual-QA · F4 scope-fidelity · F5 capability-routing

The delta is phase 3: elsewhere "momus returns OKAY" is an instruction; here it is a state field the hook reads.

Accurate plans come before enforcement

Everything upstream of review exists to make the plan accurate, not just reviewed. In PRIME, measurable criteria get one user-confirmation round before planning starts. The plan is then linted by scripts/parse-plan.mjs (DESIGN.md:445), and the review dispatch is gated on an evidence chain — nonce → momus-artifact → plan-sha → lint → verdict (docs/DESIGN.md:456). The verdict is nonce-bound to the exact reviewer dispatch, so it is unforgeable; REJECT loops are bounded (max_rounds, default 3).

Downstream, verification is not a claim: record-verify executes each todo's acceptance criteria and records evidence per criterion; record-todo refuses done without those passing records (README.md:139). On its own launch: this run was REJECTED in round 1 — an invalid branch name and a diff-grep bug in one acceptance criterion — and 83 criteria were executed and recorded as evidence (.zcode/state/launch-v0-7-3.json).

What the hook actually checks

The PreToolUse decision tree: first match wins, every other branch blocks.

The gate is a PreToolUse hook (skills/odyssey/hooks/pre-tool.mjs) reading state.json and the tool-call payload, blocking calls that would break an invariant — the review gate itself, pre-tool.mjs:876-884:

  // Outside PROJECT_DIR entirely and not bookkeeping → treat as product code (gated).
  // Review gate: product-code edits require verdict == OKAY.
  if (state.review?.verdict !== "OKAY") {
    block(
      `edits to product code are blocked until the plan passes review ` +
        `(current verdict: ${state.review?.verdict ?? "none"}, round ${state.review?.round ?? 0}/${state.review?.max_rounds ?? 3}). ` +
        `Complete the review phase first. (slug=${state.slug})`
    );
  }

Once the verdict is OKAY, supporting invariants take over: executors edit only the plan's Files: union, fail-closed on unreadable plans (README.md:130); dispatches are ledger-counted, capped at 4 (README.md:132); write-capable Bash is gated like Edit (README.md:133).

The subtle one is self-modification: the plan lives under .zcode/plans/, agent-writable bookkeeping, so an executor could widen its scope by editing the plan after the verdict. An external audit (SEC-4) closed that — pre-tool.mjs:916-927 re-hashes the plan against the sha bound to the OKAY verdict; drift blocks the edit, and a plan edit must re-run review to re-bind the sha:

      const boundSha = state.review && state.review.plan_sha256;
      if (boundSha) {
        const diskSha = createHash("sha256").update(planText).digest("hex");
        if (diskSha !== boundSha) {
          block(
            `PLAN TAMPERED: the on-disk plan (${planPath}) no longer matches the plan-sha bound to ` +
            `the OKAY verdict (expected ${boundSha.slice(0,12)}, got ${diskSha.slice(0,12)}). ` +
            `.zcode/plans/ is writable, so scope must be re-authorized by re-running momus + ` +
            `record-review (which re-binds the sha). (slug=${state.slug})`
          );
        }
      }

Verdicts are read from the artifact, not assumed — ambiguous or absent fails (README.md:135). All of it is inert in normal editing: hooks are NO-OP unless a run is active (README.md:167).

Documentation that fails the suite

Documentation is lock-pinned too: every file:line is range-checked and content-pinned in scripts/anchors.lock.json — 743 citations across 68 documents (README.md:143) — so a drifted anchor fails npm test. The tool, scripts/check-anchors.mjs:46-49:

// Usage:
//   check-anchors.mjs [--root <dir>] [--json]        verify against the lock
//   check-anchors.mjs [--root <dir>] --update        deliberate re-baseline
//   exit: 0 clean · 2 bad args · 4 zero citations discovered · 9 problems found

With the stated limit, kept on purpose: the lock proves unchanged since seeding, never correct (README.md:143).

The limits we kept on purpose

The self-grading loop. Absent the opt-in post-done /orchestrate-consult, the pipeline writes and grades its own exam (docs/ROADMAP.md:51-54).
The disarm window. Hooks disarm during consult remediation because done is terminal; phase: "remediate" re-arms them (docs/DESIGN.md:298-302).
"What it is NOT." A standing README section: not a replacement for normal agent operation, not multi-model, not harness-agnostic, not team-mode in v1 (README.md:253-258).
Registry inaction. The reviewer-reliability registry flags an unreliable config at n=9; nothing acts on it (README.md:316).
Model-routing honesty. "It routes models, not agents" — one connected model in v1 (docs/DESIGN.md:356-359).

An orchestration layer that promised enforcement it did not have would repeat the failure that motivated it.

What the new version adds

One thing: the external auditor's read-only window is verified, not promised. Every spawn window records a tri-state readOnlyViolation — true on any work-path change or HEAD move, null fail-closed on unreadable git (README.md:152; CHANGELOG.md:7-12).

Status, stated honestly: the changelog entry is dated 2026-08-24, but the newest tag is still v0.7.2 — available on GitHub, not a shipped release.

What a closed run leaves behind: plan, nonce-bound verdict, executed criteria, run report.

Where to look

Repository: https://github.com/amartinawi/zodyssey. docs/DESIGN.md carries the enforcement principle and state model; docs/ADAPT.md covers porting the delta to other hook-capable orchestrators. The gate is skills/odyssey/hooks/pre-tool.mjs — for ZCode, for now — crib freely.

DEV Community: Ammar

Your Agent Pipeline's Review Gate Should Be Code, Not a Prompt Convention

Your Agent Pipeline's Review Gate Should Be Code, Not a Prompt Convention

The failure that motivated it: a gate deleted twice

The pipeline is the part you already know

Accurate plans come before enforcement

What the hook actually checks

Documentation that fails the suite

The limits we kept on purpose

What the new version adds

Where to look