The Pain: Your agent system runs — and you have no idea how well. When something slips through, you find out from reader complaints or user tickets, and "improvement" means hoping to be more careful next time.
What You'll Learn: How the observability trio works together in production — Gate finds problems (11 real gates), Audit records them (a replayable ledger), Correction turns each fix into a permanent rule (a 5-step loop) — with real log entries as evidence, plus a 30-minute install recipe for your own agent.
Opening: Last Night at 21:00, the Audit Caught a Fish That Slipped Through the Net
On 2026-08-24 at 21:00, the nightly distribution job ran as usual. The three-way comparison (local drafts ∩ publish queue ∩ series knowledge table) found something: 2 articles finished earlier that day were already sitting in the WeChat draft box, but not a single word had been published overseas. Nobody forgot — the publish queue simply had no entry for them.
That fish was not caught by a human. It was caught by the audit — the second piece of the trio, checking the books while nobody was watching.
In the previous article, Golden Dataset: Turn Agent Regression Testing into a CI Gate, we talked about the gate. This one answers what happens after the gate: how do the mistakes a gate blocks, the entries an audit records, and the rules a correction loop sediments actually work together in production? The trio is not three boxes on an architecture diagram — it is three mechanisms running every day. This is their production retrospective, and every claim below has a real record behind it.
One: Roles of the Trio — Find, Record, Fix
Let's pin down the responsibilities first:
- Gate: find problems — every output is 100% checked; if it does not pass, it is not delivered
- Audit: record problems — every call, every block, every publish goes into the books
- Correction: fix problems permanently — after an error is logged, extract the root cause, physicalize it into scripts and gates, and never make the same mistake twice
F1 covered "test trajectories, not final answers", F2 covered calibrating the judge, F3 covered hanging a golden dataset on a CI gate. The trio is the engineering foundation under all three: without Gate, nobody stops errors; without Audit, nobody records errors; without Correction, recording is wasted.
Two: Gate in Production — 11 Gates That Have Blocked Real Things
The gate here is not an "electronic version of human review". It is a real shell script mounted in front of every task output: gate-check.sh.
# gate-check.sh skeleton (simplified from the real script)
# Gate 0: STANDING.md compliance check
# Gate 1: task context existence check
# Gate 2: skill file existence check
# Gate 3: rule execution evidence — full verify suite
run_verify "$HARNESS_HOME/verify/quote_format.py" "Rule A + Rule B format/wording" || \
block "⛔ Gate 3/8: quoting format/wording non-compliant — revise per rules/rule_A_quoting_format.md + rule_B_quoting_terms.md."
# Gate 4: task not timed out (2 hours)
# Gate 5: report quality + artifact provenance marker
# Gate 6: Evals golden-set regression
# Gate 7: scene structured-output validation
# Gate 8: email draft quality
# Gate 9: audit_fail auto-improvement gate
# Gate 10: quote_validation_missing auto-improvement gate
# Gate 11: entry convergence validation
11 gates. Each one is an independent script that reads only real file evidence — never an LLM's self-report. A gate that relies on self-discipline is not a gate; it is a decoration.
The gate has blocked real things in production. Three real examples:
Example 1 — the format gate. When a quoting output did not follow the format rules, Gate 3 blocked the output outright and pointed at the rule files to fix. What it caught was a deliverable that "looked finished but was wrong in every format detail".
Example 2 — the image gate. One article referenced an image path containing a space; the markdown parser mangled the path and the upload failed. After that pitfall was recorded in the error ledger, the image reference rule became "absolute paths, no spaces inside parentheses" — and no later article has hit it again.
Example 3 — the regression gate. In the F3 regression incident, the v1 baseline passed 100%; after a one-line logic change, v2 dropped to 50% — the gate rejected it on the spot. The change never shipped; the error was stopped before it went live.
The gate also has an anti-deadlock mechanism: the same task may retry at most 3 times; after 3 failures the gate releases the output and flags human intervention. The purpose of a gate is to block errors — not to deadlock the system.
Three: Audit in Production — What the Ledger Actually Contains
Blocking an error is only the first step. If an error is never recorded, next time it is the same error again. What Audit does: every block, every call, every publish leaves an entry.
The audit entries are physical:
# every gate block writes one audit log line
python3 /root/hermes-harness/pipeline/audit_log.py "gate:block" "$task_type:$task_id" "blocked" "$msg"
# every gate pass also writes one (sampled to avoid bloat)
python3 /root/hermes-harness/pipeline/audit_log.py "gate:pass" "${TASK_TYPE:-}:${TASK_ID:-}" "success" ""
The ledger holds more than block records. In production I keep four kinds of books every day:
| Ledger | What it records | Real example |
|---|---|---|
| Audit log | every gate:pass / gate:block | Gate 9 audit_fail blocks recorded in a row |
| Publish snapshots | per-draft verification: content_len, image count, imgur leftovers | 08-25: 14 drafts checked, B6 content_len=25658 intact |
| Run traces | full output of every cron run, archived with timestamps | 08-24 21:00 distribution report retrievable in full |
| Three-way comparison | local drafts ∩ publish queue ∩ series table | caught H5/H6 missing from the queue |
The most valuable audit was the night of 08-24: three tables compared, and 2 new articles were found missing from the publish queue. Without that ledger, the miss would only have surfaced when readers asked "why no update?". The value of audit is not how complete the logs are — it is that you can look things up when things go wrong.
Four: Correction in Production — How Errors Become Rules
Gate blocks, Audit records — still not enough. Correction is the step that gives the system a memory: turn a one-off error into a permanent rule.
My correction loop has five steps:
① incident or block → ② error-ledger entry → ③ fix with verification → ④ rule re-injection → ⑤ gate immunity
The error ledger has sedimented 31 errors, each in four parts: time, scene, root cause, fix. But an entry is only the beginning — the key is step ④, rule re-injection, physicalized into the system. On 2026-08-25, CHANGELOG v6.8.1 carried three maintenance patches, exactly three real correction loops:
| Patch | Incident | Fix | Physicalized into |
|---|---|---|---|
| ① | An artifact bypassed the pipeline and was delivered directly | New entry-convergence validation: write draft file → verify → deliver only if passed | verify/entry_convergence.py + Gate 11 |
| ② | Report without provenance marker, suspected of bypassing the generation pipeline | Reports must carry a mechanism-chain injection marker; missing marker = rejected | verify/provenance_marker.py + Gate 5 |
| ③ | Gate 9 false positive: business warnings judged as audit failures, blocking all system output for a stretch | Only failed/error/exception/critical/blocked count as failures | verify/audit_fail.py fix + CHANGELOG entry |
Patch ③ deserves an extra paragraph: that time the gate itself had a bug, misjudging warnings as failures and blocking the system every 30 minutes. The fix was not "just tweak it" — it was drawing a precise boundary for failure: a warning is a business notice, not an audit failure. That is the essence of correction sedimentation: turning a vague "something feels off" into an exact "this counts as a failure".
Correction also has a discipline: last round's lesson goes straight into this round. On the night of 08-24, Hashnode publishing reported "Draft not found". Following the earlier E1 lesson, there was no blind retry: wait 60 seconds, then independently verify with a title counter — count=1, judged successful. The same pit, the second time, does not get stepped in.
Beyond the Basics: The Essence of the Trio Is Turning Incidents into Data
After running this for over a month, I re-understood the word "observability".
Observability is not "being able to see" — it is being able to change. Pile up all the logs you want; if reading them does not produce one fix or one rule, it is self-comfort. The real value of the trio is splitting one incident into three actionable stages: Gate says "here is the mistake", Audit says "this time it is recorded", Correction says "next time it will not happen".
That is also why the trio is indispensable. If Gate does not block, Audit has nothing to record; if Audit does not record, Correction has no input; if Correction does not re-inject, Gate blocks the same error forever. Once the loop breaks, the system starts running sick.
Zoom out: this is the physical form of Loop Engineering on an agent system. Errors go from "incidents" to "data", from "blame" to "input for improvement". F3 said regression is memory — every golden case added is one more "never make this mistake again" memory; the trio turns that memory into a complete production flow: occur → record → fix → immunize.
The system will not become perfect after one fix. But it will run steadily more stable, because we turn every error into a rule.
Conclusion
What you learned today: how the trio works together in production — Gate's 11 gates block, Audit's ledger records, Correction's loop turns errors into rules. Everything blocked, recorded, and fixed is physicalized — never left to LLM memory.
The call to action is simple — 30 minutes tonight is enough to equip your agent:
# ① gate: mount gate-check.sh in front of task output (trim gates 0-11 as needed)
# ② audit: write one audit_log line per block/pass
python3 pipeline/audit_log.py "gate:block" "$task_type:$task_id" "blocked" "$msg"
# ③ correction: create error-ledger.md, four-part entries: time/scene/root cause/fix
Then pick one mistake you made recently and walk it through the five-step loop, physicalizing it into a verify script. You will feel it for the first time: the same error, stopped by the system before it even happens the second time.
Next, we zoom out from "how one agent gets stable" to "how multiple agents cooperate" — Multi-Agent Is Not the Default: The Production Consensus Behind "Avoid Multi-Agent Early": why do top teams advise "do not go multi-agent yet"? Where is the boundary between a single agent and a swarm?
About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.
Top comments (0)