Put an LLM QA gate in front of your paid deliverable: fail-closed review, one self-heal, no retry loops

TAGS: ai, node, automation, playwright

I built and shipped a compliance-report service — domain registered, Stripe live, fulfillment automated — in under a week of calendar time. Not a quarter. Not a team. One person with an AI-native toolchain. The point of this post isn't the speed; it's the specific piece of engineering that made the speed possible, because it's the piece most solo builds skip and then get burned by.

The pipeline is the easy part

The product audits public-sector websites for ADA Title II accessibility and produces an evidence binder. The pipeline is boring on purpose — a chain of CLI stages, each writing JSON to disk:

crawl-scan.mjs   → Playwright + axe-core over N pages → findings.json
pdf_census.py    → every linked PDF, tagged/untagged   → pdf-census.json
summarize.mjs    → severity-weighted score             → summary.json
evidence.mjs     → annotated screenshots of failures   → evidence/*.png
binder.mjs       → 13-page PDF                         → readiness-binder.pdf

Files on disk, not in-memory state. Every stage is resumable, inspectable, and re-runnable against a single town's directory. That's not architectural purity — it's what lets one person debug page 9 of a PDF at 11pm without re-crawling a municipal website.

None of that is the hard part. AI-assisted, that chain came together fast.

The part you can't hire

Here's what actually blocks a solo operator from charging money: nobody checks the work before it goes out. In a company that's a role. Someone reads the deliverable, notices the screenshot is missing its markers, and stops the send. Alone, you're the generator and the reviewer, and you cannot review every artifact of an automated pipeline forever.

So the reviewer became a subprocess. Before any binder is emailed, a headless model reads the actual PDF, the scan data, and — with vision — the evidence screenshots:

const prompt = `You are the pre-delivery QA reviewer, reviewing a PAID deliverable.
Read: readiness-binder.pdf (every page), summary.json, evidence/*.png (LOOK at them).
Check strictly:
 1. Entity name + domain correct on cover/headers — no other town's name anywhere.
 2. Counts in Section 1 match summary.json.
 3. Every screenshot shows numbered flag markers; captions match what's visible.
    A screenshot with no visible markers is a FAIL.
 4. No overflow into footers, blank pages, unreplaced {{PLACEHOLDER}} tokens.
Output ONLY: {"pass":bool,"issues":[...],"notes":"..."}`;

let verdict = qaReview(...);                 // reviewer errored? → pass:false
if (!verdict.pass) {
  if (selfHeal(verdict.issues)) {            // fixes src/, commits, ONE attempt
    regenerateArtifacts();
    verdict = qaReview(...);
  }
  if (!verdict.pass) throw new Error(...);   // send stays blocked, owner notified
}
await sendBinder();

Three things in there are load-bearing, and I got each of them wrong first.

Fail closed, including on the reviewer itself. If the model times out, crashes, or returns garbage, the verdict is pass: false. A broken referee must never become a green light. Every failure path routes to a human inbox with both verdicts attached.

Parse tolerantly or you'll false-fail good work. "Output only JSON" is a suggestion. Code fences, a stray sentence, a lone { in prose — early on, perfectly good binders got blocked by formatting quirks. The fix is a scanner that walks the string for the first balanced {...} that parses and contains a pass key, ignoring fences and preamble. Strict on the deliverable, forgiving on the envelope.

Self-heal fixes the generator, not the artifact — exactly once. When the gate rejects, a headless coding session gets the findings plus repo access, with hard rules: only edit src/, never hand-patch the generated PDF, never touch the email or payment logic, commit the fix. Then artifacts regenerate and the gate re-reviews. One attempt. On the money path, a retry loop is just a slower way to ship something broken.

What it caught

The gate earned itself in its first days. evidence.mjs was placing its numbered flags before the final scroll, so badges landed outside the captured frame and visible numbering started at #2 — a defect I'd have shipped, because the JSON manifest said four annotations and four annotations existed. Only something looking at the PNG catches that. The fix went into the generator with a comment naming the date it was caught.

That's the real speed argument. It isn't that AI writes code fast. It's that the review loop that normally costs a headcount and a week of calendar time is now a subprocess you can call from a fulfillment script — so an idea gets to actually deployed and taking payments in days, and stays trustworthy after you stop watching it.

That's how we built CivicBinder.