The Pain: Your agent system runs — but you have no idea how well. When it fails, you find out from user complaints, and "improvement" means telling yourself to be more careful next time.
What You'll Learn:
- The observability trio: gate + audit + correction sedimentation — the feedback loop of a commercial agent
- Why the gate is a 100% full check (a hard constraint), not a sampling-based soft reminder
- Why every call is written to an audit log — data for the system to read back, not for humans
- How correction sedimentation turns each error into a physical rule, so the same mistake can never recur
- The real difference between a lab prototype and a commercial system: LLM memory vs. engineering hardening
1. The Problem: The System Runs, but You Don't Know How Well
Scene routing, two-layer classification, the composite flow container — the last three articles covered how to make an agent work steadily. But one more fundamental question remains:
When the system makes a mistake, how do you know?
The reality:
- The agent occasionally calls the wrong tool — nobody notices
- The output format is occasionally off — downstream parsing fails
- The same error recurs over and over — "be more careful next time" never works
A commercial system has no "one-shot perfection" — only continuous improvement. Observability is the foundation of that improvement.
💡 Core insight: no observation, no improvement. The first step of getting better is knowing where you are wrong.
▲ The three-piece loop: the gate finds problems → the audit records them → correction sedimentation hardens the solution.
2. The Trio Architecture
A commercial agent's observability system has three layers:
① Gate → finds problems (100% full check)
② Audit log → records problems (persisted to SQLite)
③ Correction → hardens solutions (never the same mistake twice)
Piece One: The Gate — 100% Full Check, Not Sampling
Every output passes through predefined validation checkpoints. Not sampling — 100% full check.
def run_gates(scene_id, output):
gates = SCENE_CONFIG[scene_id]["gates"]
for gate in gates:
result = gate(output) # every gate is a validation function
if not result["passed"]:
log_gate_failure(scene_id, gate, result["reason"])
return False, result["reason"]
return True, "passed"
The gate log records the reason for every interception — the data foundation for improvement.
The core: the gate is a hard constraint, not a soft reminder. If an output does not pass the gate, it is not delivered. Period.
Piece Two: The Audit Log — Every Call Is Recorded
Every call is written to a SQLite database: who called what tool, how many tokens were spent, whether the output passed the gate, whether retries happened.
# audit_log table fields (SQLite)
id — auto-increment primary key
timestamp — record time
scene_id — scene identifier (email/quoting/report...)
tool_calls — tool call record (JSON array)
tokens_used — tokens consumed by this call
gate_passed — did the gate pass (0/1)
retry_count — number of retries
error_message — error message (optional)
from dataclasses import dataclass, asdict
from datetime import datetime
@dataclass
class AuditRecord:
"""One audit record: who called what tool, how many tokens,
did the gate pass."""
timestamp: str
scene_id: str
tool_calls: str
tokens_used: int
gate_passed: bool
retry_count: int
error_message: str = None
class AuditLogger:
"""Audit log: records the key facts of every Agent call."""
def __init__(self):
self.records = [] # in-memory queue
self._persist = [] # persistence queue
def log(self, scene_id, tool_calls, tokens, passed, retries, error=None):
record = AuditRecord(
timestamp=datetime.now().isoformat(),
scene_id=scene_id,
tool_calls=tool_calls,
tokens_used=tokens,
gate_passed=passed,
retry_count=retries,
error_message=error,
)
self.records.append(record)
self._persist.append(asdict(record)) # persisted to SQLite / file
def failed_ratio(self) -> float:
"""Gate interception rate: a measure of system health."""
if not self.records:
return 0.0
failed = sum(1 for r in self.records if not r.gate_passed)
return failed / len(self.records)
This data is not for humans to read — it is for the system to read back, automatically analyzing patterns and spotting improvement points.
Piece Three: Correction Sedimentation — Errors Become Rules
Error → record → extract root cause → harden the rule → never the same mistake again.
Every manual correction is a rule hardening — not telling the LLM to "remember to be careful next time", but directly modifying the verify script or adding a gate check, so the system itself blocks that class of error forever.
# capture the error
failure_capture.py --record "user asked to forward the mail to sunny, but the Agent looked up freight rates"
# extract the root cause -> harden the rule
# -> add a gate to the email scene: forwarding intent MUST call the smtp tool
This is the most essential difference between a commercial agent and a lab prototype:
- A prototype improves through the LLM's "memory" — a soft constraint that the model forgets
- A commercial system improves through engineering-level "hardening" — a hard constraint that cannot be forgotten
▲ The closed loop: pass → deliver; blocked → record → retry or escalate; every correction hardens a rule that intercepts the next round.
3. The Complete Loop
Agent executes
│
├─→ Gate check
│ ├─ pass → deliver
│ └─ blocked → record reason → retry / escalate to human
│
├─→ Audit log (every call recorded)
│
└─→ Correction sedimentation (human correction + rule hardening)
│
└─→ update gate / verify scripts → next round auto-intercepts
With every cycle, the system becomes more reliable. This is not a feature of some future release — it is the underlying mechanism by which the system continuously evolves.
▲ From error to rule: prototype relies on LLM memory (soft, forgets); a commercial system hardens the fix into the gate (physical, cannot forget).
4. The Trio, Compared
| Layer | Role | Problem it solves | Implementation |
|---|---|---|---|
| Gate | Finds problems | Non-compliant output | verify script, 100% full check |
| Audit log | Records problems | No way to replay what happened | SQLite, full capture of every call |
| Correction | Hardens solutions | Recurring mistakes | error → rule, made physical |
✅ Verified: this trio has run in our system for months. After each rule hardening, the same class of error is automatically intercepted by the gate — no human intervention needed.
🩸 Pitfall: early on we had the gate but no audit — the gate blocked errors but recorded nothing, so we could not analyze why something was blocked or how often. Only after adding the audit log could we see which gates had a high hit rate (the rule works) and which had a low one (the rule is useless).
💼 Value: a system that observes itself, records itself, and evolves itself is what deserves to be called "commercially viable." Anything less is just a demo that runs.
▸ Cognitive leap: observability is not monitoring — it is a learning mechanism. This is how the system keeps evolving.
5. The You Right Now
You are no longer the pragmatist who said "as long as the system runs." You are becoming an engineer who lets the system observe itself and evolve itself.
No observation, no improvement. The first step of getting better is knowing where you are wrong. The gate finds problems, the audit records them, correction sedimentation hardens the solutions — these three pieces form the feedback loop of a commercial agent.
Next up: correction sedimentation, in depth — how does a system never make the same mistake twice? The complete mechanism: error → record → root cause → hardened rule.
About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.
Top comments (0)