Green Agent Trace, Wrong Business Result: A 12-Case Outcome Test

An agent trace can show a model response, a tool call, the tool's recorded result, timing, and a final status. That is valuable evidence about execution. It is not, by itself, evidence that the requested business record changed correctly or that the final message described the change truthfully. A green span can coexist with a wrong customer, a stale value, a rejected side effect, or a completion claim made before the system of record commits.

If your team is deciding whether an agent is ready for a pilot, add an outcome contract beside the trace: define the precondition, the permitted action, the expected postcondition, the system that can prove it, and the exact user-facing claims that are allowed after each state. Then test both the telemetry path and the business-state path. This article provides a reusable synthetic 12-case acceptance suite, a small JSONL schema, and a Python checker that compares the expected business result with an independently observed receipt.

For the broader task of choosing a bounded SME use case, the professional AI-agent guide for SMEs gives the selection context; the control below addresses a narrower question that selection guides and tracing dashboards do not settle: what evidence makes one completed run safe to call successful?

Decision in one minute

Treat three statements as different things. “The call succeeded” is a transport or tool-execution observation. “The intended state now exists” is a business-state assertion. “The user was told what actually happened” is a communication assertion. A reliable release decision requires evidence for all three when the workflow can change an external record. For read-only tasks, the state assertion may instead be a versioned source snapshot and a content check.

The practical decision rule is simple: do not use a trace status as the only acceptance signal for a task whose result has consequences outside the model response. Define one or more postconditions in the system of record, capture a redacted receipt or state version, and compare it to the contract after each test run. Block or escalate if the observed result is unknown. Do not infer “no side effect” from a missing log event; absence of telemetry may mean the event was dropped.

This is a release-control proposal, not a claim that every AI workflow needs a heavyweight ledger. A low-risk drafting assistant may need only a source snapshot and a reviewer decision. An agent that sends messages, edits account details, changes orders, or triggers payments needs stronger evidence at the action boundary. The control depth should follow the side effect, reversibility, data sensitivity, and cost of a false completion.

What a trace proves—and what it does not

OpenAI’s tracing documentation describes a trace as the steps within an agent turn, including model responses, tool calls, and delegated work. Its dashboard records details such as inputs, outputs, duration, and status for spans; tool spans can include a request, a recorded result, and an error outcome. That description is useful and intentionally about recorded execution. It does not define whether the business outcome was correct. The distinction is visible in the product documentation itself: the trace gives an inspectable account of steps that were recorded, while the task owner still has to define what “correct” means for the workflow. OpenAI Agents tracing documentation

This is not a criticism of tracing. Traces answer questions that are otherwise difficult to answer: which tool ran, in what order, under which parent operation, how long a step took, and where an error appeared. They help locate execution failures and compare runs. But a tool can return HTTP 200 while refusing a business request in its payload; a database update can affect the wrong row; a write can be accepted asynchronously and later fail validation; or the model can summarize a successful read with an unsupported claim. A span’s status and the application’s desired state are different objects.

A span can be consistently named and richly attributed while still answering only the question its instrumented operation represents. OpenAI’s documentation describes agent, generation, and tool spans, but the business postcondition remains an application-level assertion. A naming convention can help correlate records across services; it cannot invent the predicate for a company’s workflow. Keep that boundary clear when teams discuss “observability”: consistent recording is necessary for analysis, but it does not make the business assertion true.

The evidence gap is operational, not theoretical

A 2026 arXiv paper on Procedure-Aware Evaluation argues that outcome-only benchmark scoring can count a task as successful even when the procedure violated a policy or the agent’s communication did not match what it observed. Its experiments use particular benchmark tasks, models, and evaluation rules; its reported 27–78% range applies to benchmark-reported successes in those evaluated settings, not to production agents in general. The useful takeaway is the structure of the problem: task utility and procedural integrity are separate dimensions, and a final success label can hide a violation if the test only checks terminal completion. Cao, Driouich, and Thomas, “Beyond Task Completion,” arXiv, submitted 3 March 2026

Another research report, published by Amazon Science, analyzes 4,671 traces across five agent domains and describes behavioral signals associated with failures, including disproportionate effort relative to outcome. The report says these signals can identify some failures without a judge model in the evaluated data. That is promising for triage, but it is not a substitute for an application’s own ground truth: a high retry count may be an anomaly, while a single successful-looking tool call may still update the wrong record. The authors’ measured effects and thresholds belong to their studied tasks and data; do not copy their rates as a forecast for your workload. Amazon Science, “Failing agents leave traces”

NIST describes its AI Risk Management Framework as a voluntary framework for incorporating trustworthiness considerations into AI design, development, use, and evaluation. It does not prescribe this particular event schema or acceptance suite. It does support the broader management principle that teams should make risk controls part of the system lifecycle rather than treat evaluation as a one-time demo. Apply the framework to your own risk context, and do not present a locally designed test as a NIST requirement. NIST AI Risk Management Framework

A real practitioner question

In a LangChain community thread, one developer described an agent that told the user it needed more information, then proceeded to make a tool call within the same request. The post is one person’s debugging report, not a prevalence estimate and not proof of a platform-wide defect. It is still a useful example of an unresolved operational question: when the user-facing status, the agent’s decision, and the tool execution diverge, which record tells an evaluator what happened and what may be claimed? LangChain Forum discussion, 25 November 2025

That question appears in many forms: did the action run, did it run once, did it run against the intended entity, did the returned status mean committed or merely queued, and did the assistant accurately describe the result? A detailed trace can help answer the first questions when instrumentation is complete. The application owner must define the business invariant and determine which system is authoritative for the later questions. A support thread can reveal a failure mode worth testing; one thread cannot tell you how often the failure occurs.

Define three independent green checks

Use a three-layer acceptance model. The first layer is execution evidence: the run has a valid root, the required spans or events are present, parent-child relationships make sense, and the tool call reached a terminal outcome. The second layer is state evidence: the authoritative system reports the required postcondition for the intended entity, and the observation is tied to the same operation or transaction. The third layer is claim evidence: the final message is consistent with the state evidence and does not claim more than the system has established.

Write the release result as a tuple rather than a single green badge: execution_ok, state_ok, claim_ok, and evidence_complete. The first three are independent pass/fail/unknown values. The last value indicates whether the evidence is sufficient to decide. An unknown is not a pass. It is a request for a stronger observation, a human review, or an explicit “cannot confirm” user message.

For a read-only research agent, the postcondition might say that every key answer sentence cites a source version available at run time and that the final answer does not claim a write. For a CRM assistant, the condition might say that a note was attached to the selected customer record and that the response says “saved” only after the CRM’s committed record is read back. For an agent that merely drafts a purchase order, “draft created” and “purchase approved” must be separate outcomes. The contract should mirror what the system actually does, not the verbs the product team wishes it did.

EY describes AI as a strategy and service capability, while consulting guidance on agentic governance emphasizes accountability, observability, and oversight. Such guidance is useful for executive ownership and operating-model choices, but it does not establish the correctness of an individual run. A team should connect those governance responsibilities to testable controls: a named owner for each postcondition, a source of truth, and a decision for missing evidence. EY AI insights provide the broader consulting context; the detailed pass criteria below are an engineering design recommendation, not a firm’s published standard.

The compact outcome contract

Create one contract for each business action class, not one vague contract for “the agent.” Give each contract a version and an owner. A useful record includes: contract_id, workflow_version, intent_id, entity_ref, precondition, allowed_action, expected_postcondition, authoritative_system, receipt_type, allowed_user_claims, and failure_route. Use a pseudonymous or encrypted entity reference in telemetry; keep identity resolution in a protected system.

The precondition says what must be true before the agent acts. The allowed action bounds the operation and its arguments. The expected postcondition is a predicate over authoritative state after execution. The receipt explains where to observe that state and how to correlate it to the requested operation. The allowed-claim list maps state to truthful user language. The failure route identifies whether the system retries, reconciles, asks the user, or transfers to a human.

For example, a service agent that updates a delivery preference could require an authenticated session, a customer-selected address, and a current order identifier before writing. Its action contract could permit an update only to the preference field for that order. The postcondition could require the record to contain the selected delivery preference and the version to be newer than the read version. A success receipt would include the operation ID, entity reference, final state version, and commit status. The user may hear “updated” only if the read-back confirms the new preference. If the API returned “accepted” without a committed record, the permitted claim is “request submitted; confirmation pending.”

This contract avoids a common category error: treating the tool’s acknowledgement as the task’s postcondition. A queue can acknowledge a message before a worker applies it. A browser automation can report a click without proving the resulting page state. A ticketing API can return an object ID whose status is still pending. The receipt must be selected from the source that owns the business truth, not from the agent’s narrative.

A reusable 12-case synthetic test pack

The following matrix is an original, hand-authored acceptance asset designed on 8 November 2026. It contains twelve scenarios only. It is not a sample of production traffic, a user study, a model benchmark, or a measured failure rate. Each case specifies an input condition, the evidence a test harness should inspect, and an expected decision. Adapt it to one workflow and replace the illustrative fields with the system’s real state transitions.

Case	Scenario	Expected execution evidence	Expected state / claim result
1	Correct record, committed update	Required tool span and matching operation ID	State read-back matches; “updated” allowed
2	Tool returns transport success but business rejection	Span is successful at transport layer; payload has rejection	State unchanged; “not updated” or escalation
3	Wrong entity selected	Tool call references a different entity than the contract	Fail closed even if that record changed; no success claim
4	Write accepted asynchronously	Tool result is queued or pending	State remains unknown until correlated commit receipt
5	Commit succeeds but trace exporter drops the final event	State read-back shows committed operation; telemetry is incomplete	State may pass; evidence completeness fails and telemetry incident is recorded
6	Agent claims completion without a write	No valid action span or receipt	State fails; claim fails
7	Correct action, stale read-back	Write is recorded; read is from an earlier version	State unknown; retry only under the workflow’s safe reconciliation rule
8	Duplicate call after timeout	Two attempts share one business operation ID	Exactly one final effect required; otherwise fail and reconcile
9	Permission denied before side effect	Denial event is tied to principal and action	No write expected; user is told permission blocked the action
10	Partial success across two records	One state predicate passes and another fails	Report partial completion precisely; do not use a global “done” claim
11	Tool returns a plausible but stale value	Source version is older than the contract’s freshness bound	Answer is blocked or marked stale; no unsupported “current” claim
12	Human handoff is required	Escalation event and named queue or owner recorded	Agent says it handed off only after queue acceptance; resolution stays pending

These cases intentionally separate failure classes that often look alike in a dashboard. Case 2 is not a transport failure; the business operation rejected the request. Case 5 is a telemetry failure even though the external state is correct. Case 7 is a consistency problem rather than proof that the write failed. Case 10 forbids collapsing partial state into one boolean. Case 12 distinguishes the agent’s intention to hand off from an accepted handoff. If your test runner reports only “tool succeeded” and “agent finished,” it cannot express these distinctions.

Why twelve cases are enough for a first review—and not enough for release

The number twelve is a design choice that makes the initial review manageable. It is not a statistically justified sample size, a coverage guarantee, or evidence that twelve tests catch a fixed fraction of failures. A production suite needs cases derived from real workflow rules, incidents, boundary values, entity types, locales, timeouts, retries, and permissions. Begin with one scenario per distinct invariant, then add variants where the same visible response can conceal a different state.

To expand the suite, create a cross-product only when interactions are plausible. For example, a permission denial during a retry after an asynchronous write deserves its own test if the product can encounter that sequence. Do not create a combinatorial explosion of arbitrary combinations. Rank candidate tests by side-effect risk, reversibility, customer harm, data sensitivity, and whether the current evidence can distinguish “did not happen” from “happened but was not observed.” A test is valuable when it could change a launch decision or prevent an incorrect claim.

Preserve both the expected and observed evidence for every case. The expected values belong to a reviewed fixture. The observed values must come from the agent runtime, the authoritative application system, or a test double that behaves according to a declared contract. Never hand-edit a test result to match what the agent said. If a business state cannot be read after the action, label the outcome unknown and design a receipt path before claiming that the agent is production-ready.

Event schema: tie execution to a business result

Do not overload a span with every business field. Keep the trace useful for timing and causal analysis, and attach a separate outcome event or evaluation record linked by trace_id, run_id, and operation_id. The minimum useful distinction is between what the agent requested, what the tool attempted, what the system committed, what the evaluator observed, and what the assistant told the user.

An outcome record might contain contract_id, contract_version, intent_id, run_id, trace_id, operation_id, entity_ref_hash, attempt, workflow_version, tool_name, decision, precondition_result, expected_state_digest, observed_state_digest, state_version_before, state_version_after, commit_status, claim_class, evidence_status, checked_at, and checker_version. The digest should be computed over a carefully defined canonical representation. A hash of arbitrary JSON is not stable if field order, whitespace, normalization, or volatile values change. Document which fields are included and which are intentionally ignored.

Use enumerated states such as pass, fail, unknown, and not_applicable; do not encode unknown as false when false means “verified absent.” Preserve the attempt number and the parent operation. When a retry occurs, a new attempt may share the business operation ID but must have its own execution span. That lets the team ask whether two tool calls were attempts at one operation or two separate requested actions.

Avoid logging raw prompts, full customer records, access tokens, payment details, or unrestricted tool arguments merely because tracing makes them easy to capture. OpenAI’s tracing documentation calls out recorded inputs and outputs and describes trace export permissions; teams should inspect what is captured and who can export it before enabling broad retention. The source’s export example requires trace-read permissions, which is a reminder that trace access is an authorization decision, not a neutral debugging convenience. Keep the outcome record minimal, use redacted references, separate high-sensitivity payloads from routine telemetry, set retention periods, and test deletion paths.

A small executable checker

The code below validates one JSON Lines file against the most important invariants in this article. It uses only Python’s standard library. Each line is one observed test result; expected fields are included in the fixture so the checker can run offline. In a real system, generate expected fields from reviewed test cases and observed fields from protected test adapters. This script is a starter control, not a production-ready audit system: it does not authenticate receipts, validate cryptographic signatures, connect to a database, enforce access control, or assess semantic truth in free text.

Save as check_outcomes.py and run python check_outcomes.py outcomes.jsonl. A nonzero exit code means one or more records failed or remain unknown. In CI, fail closed for state-changing tasks when the result is unknown. For a non-side-effecting research task, the contract may define a different policy, but that choice should be explicit.

#!/usr/bin/env python3
"""Validate trace-linked business outcomes from synthetic JSONL fixtures."""
import json
import sys
from pathlib import Path

REQUIRED = {
    "case_id", "contract_id", "run_id", "trace_id", "operation_id",
    "attempt", "entity_ref_hash", "expected_state", "observed_state",
    "execution_status", "commit_status", "claim_class", "evidence_status",
}
SUCCESS_CLAIMS = {"updated", "booked", "sent", "approved", "completed"}

def validate(row):
    errors = []
    missing = REQUIRED - row.keys()
    if missing:
        return ["missing fields: " + ", ".join(sorted(missing))]
    for key in ("case_id", "contract_id", "run_id", "trace_id", "operation_id"):
        if not isinstance(row[key], str) or not row[key].strip():
            errors.append(f"{key} must be a non-empty string")
    if not isinstance(row["attempt"], int) or row["attempt"] < 1:
        errors.append("attempt must be a positive integer")
    if row["execution_status"] not in {"pass", "fail", "unknown"}:
        errors.append("invalid execution_status")
    if row["commit_status"] not in {"committed", "rejected", "pending", "unknown", "not_applicable"}:
        errors.append("invalid commit_status")
    if row["evidence_status"] not in {"complete", "partial", "missing"}:
        errors.append("invalid evidence_status")
    state_ok = row["expected_state"] == row["observed_state"]
    committed = row["commit_status"] == "committed"
    success_claim = row["claim_class"] in SUCCESS_CLAIMS
    if success_claim and not (state_ok and committed):
        errors.append("success claim lacks matching committed state")
    if row["execution_status"] == "fail" and success_claim:
        errors.append("failed execution cannot carry a success claim")
    if row["commit_status"] == "rejected" and state_ok and row["expected_state"] != row["observed_state"]:
        errors.append("rejected operation unexpectedly matches target state")
    if row["evidence_status"] != "complete" and success_claim:
        errors.append("success claim requires complete evidence")
    return errors

def main(path):
    failures = 0
    for line_no, text in enumerate(Path(path).read_text(encoding="utf-8").splitlines(), 1):
        if not text.strip():
            continue
        try:
            row = json.loads(text)
            errors = validate(row)
        except Exception as exc:
            errors = [f"invalid JSON or fixture: {exc}"]
        if errors:
            failures += 1
            print(f"FAIL line {line_no}: " + "; ".join(errors))
        else:
            print(f"PASS line {line_no}: {row.get('case_id', 'unknown')}")
    print(f"summary: failures={failures}")
    return 1 if failures else 0

if __name__ == "__main__":
    if len(sys.argv) != 2:
        raise SystemExit("usage: python check_outcomes.py outcomes.jsonl")
    raise SystemExit(main(sys.argv[1]))

There are deliberate limits in the example. Equality between expected_state and observed_state assumes both are normalized, versioned snapshots and that the predicate is simple. A real contract should use a specific predicate, such as “the active delivery preference equals pickup, the order ID is the requested order, and the record version is at least the acknowledged version.” It should not compare an entire volatile record if timestamps or unrelated fields change. A payment workflow might need a settlement event, not a UI message. A message workflow might need a provider receipt plus a policy check, and “accepted by provider” still may not mean “read by recipient.”

The checker also trusts the JSONL fixture’s source. In production, use a trusted test harness to query the authoritative system, verify the receipt’s signature or server identity where available, and preserve the query time and source version. A fabricated or stale observation can pass any local comparison. The code is meant to make the contract visible and runnable, not to create cryptographic assurance by itself.

How to author expected and observed fixtures

Keep test fixtures small and legible. A passing example might look like this:

{"case_id":"C01","contract_id":"delivery-pref-v2","run_id":"r-001","trace_id":"t-001","operation_id":"op-001","attempt":1,"entity_ref_hash":"h:order-7","expected_state":{"preference":"pickup","version":18},"observed_state":{"preference":"pickup","version":18},"execution_status":"pass","commit_status":"committed","claim_class":"updated","evidence_status":"complete"}

For this fixture, expected and observed state use the same normalized snapshot shape, so the equality check is meaningful. If your contract uses a bound such as “version at least 18,” replace whole-object equality with an explicit predicate such as expected.preference == observed.preference and observed.version >= expected.version_min. Keep that predicate in version-controlled code and record the checker version. Do not weaken a test after seeing a failure without a reviewed change to the business contract.

When you produce a synthetic suite, label it synthetic in the README and in reports. Do not write “we found 10% of agent tasks fail” if the suite contains ten hand-built fixtures. The number of authored cases describes the test pack, not the behavior of a population. A pass rate over a small, intentionally selected fixture set is a coverage result for those scenarios only. If you want to estimate real-world reliability, design a representative sampling method, define the denominator, account for repeated entities and correlated runs, and disclose how unknown outcomes were handled.

For each fixture, preserve the expected outcome separately from the captured result. A test operator should be able to inspect the request, the trace link, the authoritative receipt, and the final user claim without modifying any of those fields. If a human adjudicates an ambiguous case, record who reviewed it, what evidence was available, the decision, and the policy version. The adjudicator’s conclusion is an evaluation label, not an event that magically changes the original state.

A decision matrix for rollout

Use the following matrix to decide how much evidence to require. It is an engineering recommendation, not a standard or universal risk score. “High risk” is defined locally by the workflow owner based on potential harm and recoverability.

Workflow shape	Minimum state evidence	When to permit automatic completion language	Default unknown-result behavior
Read-only answer from fixed documents	Document IDs, versions, retrieval record, answer-source mapping	After sources are current and claims pass the task’s review rule	Mark answer unverified or ask for review
Draft only; no external write	Draft ID, owner, version, and explicit draft status	Say “draft prepared,” not “sent” or “approved”	Keep draft in review queue
Reversible internal update	Operation ID, target entity, committed read-back, version	After target and state predicate match	Stop automatic retries; reconcile state
External communication	Recipient, approved content digest, provider acceptance receipt	Say “sent” only at the provider’s defined acceptance point	Do not resend until duplicate/receipt status is reconciled
Financial or sensitive change	Strong identity, permission decision, bound approval, committed state and independent verification	Only when all policy and state predicates pass	Block, preserve evidence, and route to an authorized human
Multi-record workflow	Per-record predicates and an explicit partial-completion state	Only when all required records pass or the user is told exactly what remains	Report partial progress and create an owner handoff

The matrix prevents a low-risk success criterion from being applied to a high-impact action. A “tool call returned” check may be adequate for an internal, disposable draft in a sandbox. It is not adequate for a money movement or a customer record change. The same model and agent can serve both workflows; the action contract and release gate should differ because the consequences and authoritative evidence differ.

Metrics that connect traces to outcomes

Start with counts that have explicit denominators. state_verified_rate is the number of eligible completed runs whose required postcondition was verified divided by the number of eligible completed runs. truthful_completion_claim_rate is the number of completion claims supported by matching state evidence divided by all completion claims. unknown_outcome_rate is the number of state-changing runs whose final state could not be established divided by all state-changing runs. Report these by workflow version, action type, environment, and time window; do not merge read-only answer tasks with write tasks into one opaque score.

Track telemetry completeness separately: required spans present / required spans expected. A missing span is a measurement-quality defect. It should not be silently counted as a business failure if the state source independently confirms the operation, nor as a business success if the state source is missing. This distinction is especially important when sampling or asynchronous export is enabled. OpenAI notes that traces are built after a turn ends and can appear after the agent’s answer; an absent trace immediately after a response can therefore be a timing issue, not evidence that the agent did nothing. The telemetry and outcome windows must be aligned before calculating rates. OpenAI tracing lifecycle details

Do not optimize the agent against one metric alone. Lower latency can coincide with more unknown outcomes if the system stops checking; more tool calls can be legitimate for complex cases; a high verified-state rate can hide poor user communication. Pair outcome measures with quality and safety guardrails, and inspect changes to task mix. If a deployment shifts from routine to exceptional cases, a raw before/after percentage may move even when the system did not regress. Stratify, preserve definitions, and show sample counts with every rate.

Use traces to explain a metric, not to replace it. If verified outcome rate falls while the trace error rate stays flat, inspect tool payload semantics, entity selection, source freshness, and commit behavior. If trace completeness falls while state verification remains stable, inspect instrumentation and export delay. If truthful completion claims fall while state writes remain correct, inspect the message-generation boundary and status mapping. The point of an outcome contract is to make these diagnoses possible without asking an LLM judge to infer system state from a polished sentence.

Safe rollout sequence

First, define the action boundary and the system of record. Name who owns the workflow predicate and who is authorized to change it. Write the allowed action and non-goals. If a tool can create an external side effect, establish a test environment or non-production entity before running negative cases. Confirm that the fixture can inspect the result without relying on the agent’s own description.

Second, run the 12 synthetic cases against a deterministic harness or mock to verify the evaluator itself. Intentionally alter the fixture to create an incorrect entity, an uncommitted result, and a missing observation; confirm that each one produces the expected failure or unknown. If all test fixtures pass before the evaluator has been challenged, you have only shown that the fixtures are internally consistent. Verify that the checker rejects malformed input, duplicate operation IDs where they are forbidden, and contradictory claim/status combinations.

Third, run shadow evaluations on representative, permission-limited cases. Shadow mode means the agent proposes actions but does not execute the side effect, or operates on a test tenant with reversible data. Compare proposed actions, expected state, and observed state from the baseline process. Record how often a human must correct the intent, entity, or claim. Do not convert shadow agreement into a causal productivity claim: it establishes behavior under the observed sample and conditions, not the counterfactual business benefit of full automation.

Fourth, use a narrow pilot with a named operator, a rollback plan, a stop threshold, and a way to inspect every unknown result. Keep the first action class bounded. Ensure the human can intervene before irreversible actions. If the agent is allowed to write, the adapter should return a structured status distinguishing rejected, pending, committed, and unknown. A single boolean success collapses states that need different recovery. For each unknown, define whether the safe next action is to query, wait, stop, or transfer ownership; blind retry is not a general reconciliation strategy.

Fifth, review incidents and near misses as test additions. Preserve a redacted trace, the relevant outcome receipt, the contract version, and the human adjudication. Turn the failure into a regression fixture only after removing sensitive content and agreeing on the expected behavior. Update the contract if the business rule truly changed, and retain a history of the old rule so that old runs can still be interpreted. A new prompt version without a corresponding contract/version update makes comparisons ambiguous.

Failure modes that a green dashboard can hide

Wrong entity, correct tool. The call reaches the API and the API returns success, but the agent selected the wrong customer or order. Catch this by binding the approved entity reference to the action and verifying it again in the authoritative receipt. Do not rely on a model-generated entity name when a stable identifier is available.

Accepted is mistaken for committed. A queue or provider accepts a request, but a later worker rejects it. Model the intermediate state explicitly. A trace can be fully successful for the enqueue operation while the business request remains pending. The user-facing language should reflect the stage that was verified.

A missing event is mistaken for no side effect. Sampling, exporter lag, retries, or instrumentation bugs can remove evidence. If the state source is authoritative and says committed, the action may have happened despite the missing event; if neither source is available, the result is unknown. Do not automatically rerun a potentially non-idempotent action just because the trace is incomplete.

The check reads a stale replica. A write commits but a replica has not caught up. Immediate read-back may show the prior value. Define consistency expectations, use an operation receipt or read-your-write path where supported, and set a bounded reconciliation window. If the state remains unknown after the window, stop and escalate instead of converting an eventual-consistency delay into a false failure or a duplicate write.

The evaluator repeats the agent’s assumption. A test harness that derives expected state from the tool arguments can bless a wrong decision. The source of expected state should come from the reviewed fixture or business rule, while observed state comes from the authoritative system. Keep those data paths separate. A judge model may help label ambiguous language, but it should not be the only source that claims the database was changed.

The final message overstates partial completion. One of three records may update successfully. A summary saying “all set” is false even if two traces are green. Store a per-item outcome and compute the aggregate only from the required set. The UI should present partial completion with the remaining items and owner.

The log captures too much. Recording every prompt and tool payload can expose personal or confidential data to operators and vendors who do not need it. OpenAI’s trace/export capability and NIST’s lifecycle-oriented risk framing make access and retention part of the control design, but your organization must decide the lawful basis, data minimization, retention, and deletion rules. Prefer identifiers, digests, narrow fields, and sampled access to raw payloads. Redaction must happen before exporting to a third-party destination when policy requires it.

A success metric shifts with task mix. A change in eligible task types, entity volume, user population, or human fallback changes the denominator. Preserve the eligibility rule and stratify results. A metric that moves from 92% to 95% may describe a different population if difficult cases were silently routed to humans before the agent began. Measure both the agent path and the total workflow path.

Contradictions and limits

There is a real tension between rich traces and data minimization. Debugging can improve when inputs, outputs, and tool results are recorded; privacy, security, contractual, and operational risks can increase with the same collection. The right choice depends on sensitivity, access controls, retention, deployment model, and whether a redacted receipt can answer the incident question. “Log everything” is not a neutral default. “Log nothing” makes accountability and diagnosis difficult. Choose fields based on a stated use, restrict access, and test deletion.

There is also a tension between automated outcome scoring and human judgement. Structured state checks are preferable when the business predicate is explicit. They can miss semantic questions such as whether an answer was confusing, whether a user’s intent was actually met, or whether a source is persuasive enough. A human or model evaluator can contribute a label for such questions, but should not overwrite authoritative state. The evaluation record should show which parts were deterministic and which involved judgement.

Research findings should not be generalized beyond their setup. The procedure-aware paper analyzes benchmark scenarios, and the Amazon Science report analyzes a defined collection of traces across five domains. Both motivate richer evaluation, but they do not establish the failure rate of your application. The LangChain forum post is a single anecdote. The GSC signal behind this article is also modest: for the tracked query “agent intelligence artificielle pro,” the account’s Search Console history showed position 26.5, two impressions, and zero clicks in the 7-day window ending 5 November 2026. That is a tiny site-query observation, not evidence of broad demand, future ranking, or a causal content opportunity. It was used only as a topic-selection clue, and the window may not align exactly to the reader’s local timezone.

A YouTube search surfaced a relevant OpenTelemetry talk on distributed tracing for multi-agent systems. A direct caption retrieval attempt returned page boilerplate rather than transcript text, so the video was excluded from factual support. This article relies on readable documentation, papers, and one community report instead. That limitation matters: a video title or description is not a substitute for hearing its claims or inspecting the demonstrated system.

The protocol fails when the team cannot define an authoritative state, cannot safely observe it, or cannot make the agent’s action idempotent or reconcilable. For some workflows, the outcome is inherently delayed or subjective. In that case, the correct result may remain pending until a human or downstream event closes the loop. Do not force a binary pass/fail merely because a dashboard wants one. Use “unknown” as an honest operational state with an owner and a next review time.

Practical checklist before a pilot

• [ ] The action class and excluded actions are written down.
• [ ] A named owner has approved the business postcondition and version.
• [ ] The test fixture defines expected state independently from the model output.
• [ ] The authoritative system can confirm or refute the state change.
• [ ] The action, entity, and business operation ID remain correlated across retries.
• [ ] Tool acknowledgement states distinguish rejected, pending, committed, and unknown.
• [ ] The final user message is checked against observed state, not only tool status.
• [ ] Partial completion and human handoff have explicit representations.
• [ ] Missing telemetry is separated from verified absence of side effects.
• [ ] Unknown outcomes do not trigger an unsafe blind retry.
• [ ] Sensitive payloads are minimized, access-controlled, and retained only as needed.
• [ ] The checker is tested with negative fixtures, malformed input, and stale state.
• [ ] Metrics show denominators, task mix, workflow version, and unknown rate.
• [ ] Stop conditions, reconciliation ownership, and rollback paths are assigned.
• [ ] Synthetic case counts are never presented as production failure rates.

What to do next

Start with one side effect and one authoritative record. Write the contract in plain language, translate its postcondition into a deterministic predicate where possible, then attach that check to a trace-linked result record. Run the 12-case synthetic suite against your evaluator, add cases from known incidents, and test the cases where telemetry and business state disagree. Only then decide whether the agent may execute the action, merely draft it, or must hand it to a person.

Keep trace status, state status, and claim status visible as separate signals. Use a trace to reconstruct a run, a receipt to establish what the business system accepted or committed, and the final-message check to ensure the user was not promised more than the evidence supports. If any one of these sources is missing, report the uncertainty and assign a reconciliation owner. A green dashboard is useful; a verified, bounded, and truthfully described outcome is the release criterion.

Source and method notes

Research cutoff: 8 November 2026. The primary site page used for topic context is an evergreen SME agent guide. The official tracing documentation was read for its description of trace structure, span details, tool results, timing, and export permissions. The NIST page was read for the framework’s stated voluntary scope and lifecycle framing. The arXiv paper was read as a preprint; its reported benchmark ranges are explicitly kept inside its tested setup. The Amazon Science research summary was read as a research report and its trace count and domains are not treated as a general population estimate. The LangChain forum thread is a single user report used to identify a question, not to estimate prevalence. EY’s page is consulting context rather than primary evidence about technical correctness.

The original 12-case matrix and code were authored for this article on 8 November 2026. The method was to translate the execution-versus-state-versus-claim distinction into hand-written failure cases and a standard-library JSONL checker. Sample size for this original asset: 12 designed scenarios. Exclusions: no production traces, users, organizations, agent framework, benchmark runs, or measured error rates. The checker is illustrative and should be adapted and independently tested before production use. The GSC note reflects one query-site row for one seven-day window; its low impressions mean it cannot support a market-size or ranking conclusion. Video captions were unavailable through the direct retrieval path and the video was excluded from evidence.