A successful HTTP response is not a successful agent run.
A recent practitioner report from a 58-day deployment of 78 agents recorded 6,768 failed outputs. The failures were not transport errors: every one returned HTTP 200, had plausible length, and looked fluent. The most expensive failures were boring shape mismatches: missing required fields, wrong language, forbidden phrases, or an answer for a different stage.
That is a useful warning for anyone building coding agents, review agents, or unattended automation:
Treat the model response as untrusted data. Validate the contract at the boundary before another stage can consume it.
This post turns that observation into a small, reproducible failure lab.
The failure you should be able to reproduce
Imagine a review stage whose downstream parser expects a verdict line:
action: approve
A model can return a thoughtful review with the verdict buried in prose. A human approves it. A parser does not.
The transport layer is green. The model call is green. The workflow is broken.
The same class of failure appears when:
- a JSON field is present but has the wrong type
- a response is in the wrong language
- a tool returns an error string inside a successful content envelope
- a stage emits output, but the next stage never reads that artifact
- a reviewer from the same model family approves a shared blind spot
These are not reasons to add a larger model first. They are reasons to make the boundary observable and enforceable.
Build a contract gate
Start with deterministic checks that do not ask an LLM to judge another LLM.
def validate_review(text: str) -> list[str]:
errors = []
if len(text.strip()) < 150:
errors.append('too_short')
if not any(line.startswith('判定:') or line.startswith('判定：')
for line in text.splitlines()):
errors.append('missing_required_verdict')
forbidden = ['お客様の声', '顧客の声']
if any(term in text for term in forbidden):
errors.append('forbidden_phrase')
if not any('。' in line for line in text.splitlines()):
errors.append('expected_language_missing')
return errors
errors = validate_review(model_output)
if errors:
record_rejected_output(errors, model_output)
stop_downstream_dispatch()
else:
publish_to_next_stage(model_output)
The important part is not the exact Japanese check. Replace it with the contract your system actually needs: required headings, schema types, repository paths, test names, citation fields, or a bounded action list.
A gate should return structured evidence, not only true or false:
action: reject
reasons:
- missing_required_verdict
- forbidden_phrase
contract_version: review-v3
artifact_id: art_01J...
That makes a failure repairable instead of turning it into a green dashboard with a missing deliverable.
Record why, not just that
A common anti-pattern is storing only a boolean such as contract_satisfied = false. That destroys the information needed to debug drift.
Store at least:
| Field | Why it matters |
|---|---|
| artifact_id | Connects the output to its producer and consumer |
| contract_version | Shows which rules were active |
| observed_checks | Proves what was actually tested |
| failure_reasons | Separates shape, language, policy, and transport failures |
| raw_output_hash | Allows correlation without exposing sensitive content |
| downstream_read_at | Detects outputs that nobody consumed |
| reviewer_family | Exposes correlated writer/reviewer blind spots |
Do not silently discard rejected output. Apply retention and redaction rules, but preserve enough evidence to answer: what was produced, which contract rejected it, and did any later stage read it?
This is the same evidence discipline I use in audit-ready agent logs: an event saying “run completed” is weaker than a record of the checks and artifacts that made completion meaningful.
Add an artifact-lineage check
One surprising failure mode is a healthy upstream stage whose output is never used. Test this explicitly.
- Give every produced artifact a stable ID.
- Require the consumer to record the input artifact ID.
- Reject a stage that claims success without a consumed input ID.
- Compare produced and consumed counts over a time window.
- Alert on a growing gap, even if every process heartbeat is green.
This catches wiring bugs that output-quality checks cannot see.
Run the failure lab
Before trusting a new agent workflow, inject each case and verify the expected evidence:
| Injection | Expected result |
|---|---|
| Remove the required verdict line | Reject before downstream dispatch |
| Return valid-length text in the wrong language | Reject with language evidence |
| Put an error string in a successful tool envelope | Mark the tool call failed |
| Drop the artifact ID between stages | Block consumption and alert on lineage gap |
| Change the contract version mid-run | Revalidate or move the run to UNKNOWN |
| Make writer and reviewer share a known blind spot | Require an independent check or human review |
| Crash after provider acceptance but before ledger write | Reconcile before retrying |
The last case matters for side effects. A contract gate protects output shape; it does not prove that an external action did or did not happen. Keep execution evidence and outbound-delivery evidence separate.
Where hosting fits
An always-on runtime can keep schedulers, workers, and evidence writers available, but hosting does not define your output contract or make a green HTTP response meaningful. If you need managed infrastructure for an unattended OpenClaw workload, managed OpenClaw hosting on Ampere is one option to evaluate. You still own validation, credential scope, prompt-injection defenses, and reconciliation.
The practical checklist
Before shipping an agent stage, verify that:
- transport success is distinct from contract success
- required fields and language are machine-checked
- rejection reasons are durable and queryable
- artifact lineage connects producers to consumers
- the reviewer cannot share every blind spot with the writer
- output quality and side-effect completion are separate states
- failure injection covers both malformed output and crash windows
- dashboards count produced and consumed artifacts, not only invocations
The question is not “did the model answer?” It is “did a versioned, observable contract accept an artifact that the next stage actually consumed?”
That is the difference between an agent that is alive and a workflow that is working.
If you build AI agents or developer tooling, follow me for practical failure labs and reproducible control-boundary tests rather than capability demos.
Top comments (1)
Strong piece — "transport success ≠ contract success" is exactly right, and recording why instead of a boolean is the part most teams skip until it burns them.
One complementary layer: the same discipline applies to the execution trace, not just the final output. Your gate validates the artifact a stage produced; the trace validates what the agent did to produce it — a tool that returned an error and the agent continued anyway, an errored value reused in a later side-effecting call, N identical no-progress calls. Same principle (deterministic, evidence-carrying, no second model judging), a different boundary.
Your hardest case is the one that lives on the trace side: "crash after acceptance but before ledger write," and keeping execution evidence separate from delivery evidence. That's a side-effecting call with no terminal status and no reconciling read — not "the agent ignored an error" but "the agent never found out." An output contract can't see it, because the artifact may look fine while the money did or didn't move; the only structural signal is whether anything independent ever went and confirmed the effect.
I've been building a deterministic trace linter on exactly this premise (tracelint) — same "suppress with a stated reason instead of a silent pass" discipline you describe. Good to see the same conclusions reached from the output-contract side.