When a Failed Agent Step Looks Finished

When a Failed Agent Step Looks Finished

A multi-agent research workflow can fail in a quiet way: a step returns nothing useful, the orchestration keeps going, and the run ends with a polished report. The logs still know something went wrong. The surfaces people actually read — the summary, the terminal output, the exit path — do not.

I evaluated OpenAI's official financial-research example in openai/openai-agents-js, frozen at commit 710cccfd8fd26b395f8e3470419852d76de80967. A six-case harness caught two fail-open paths. A one-file patch closed them. Upstream fixed the example about ten hours after the report. This is a case study of that path — not a critique of the SDK, and not a claim about how often the pattern shows up in production.

What was measured

The target slice is examples/financial-research-agent. The example plans web searches, runs search agents concurrently, drops failed results, writes a structured report, asks a verifier for { verified, issues }, revises at most twice, and then emits terminal output.

Every result here is labeled synthetic-orchestration: the real FinancialResearchManager control flow runs, but the model and web-search boundaries are deterministic fakes. No model API key is required. That keeps the evidence cheap and repeatable. It also means this is not a model-quality benchmark, a financial-accuracy study, a security finding, or evidence about the Agents SDK as a whole.

The six cases cover search aggregation, partial search failure, complete search failure, an accepted first draft, a report that passes on the final retry, and retry exhaustion.

Baseline: 4/6

Canonical baseline run: 4 pass, 2 fail.

The two failures share one control-flow pattern. A negative prerequisite exists in state, but no guard connects that state to the terminal report path:

1. Complete search failure. Both searches fail. search() turns exceptions into null; performSearches() filters them out; run() still calls the writer. The manager reports success and emits a report with no usable sources.
2. Verification exhaustion. The verifier rejects the initial report and both permitted revisions. The revision loop stops correctly after two attempts, then output proceeds anyway. The manager ships the still-rejected report.

Two of the four baseline passes reproduce the example's own shipped tests (accepted first draft; pass on final retry). Across the four cases this teardown designed, the baseline was therefore 2 pass / 2 fail. The headline 4/6 includes the two replicated cases.

After a one-file patch: 6/6

The patch changes only examples/financial-research-agent/manager.ts:

1. throw before the writer when zero usable search summaries remain;
2. throw before terminal output when verification is still negative after the bounded revision loop.

Same corpus, same fakes, same control flow:

Metric	Baseline	Patched
Corpus pass	4/6	6/6
Corpus fail	2/6	0/6
Upstream files changed	0	1

Behaviorally:

• FR-003 (all searches fail): baseline called the writer and emitted a report; patched stops with a controlled error before the writer.
• FR-006 (retry budget exhausted): baseline kept three negative verifications and two revisions, then still emitted the report; patched keeps the same loop shape and stops with a controlled error instead of emitting.

The other four cases stayed green. The example's own two focused tests still pass under the patched run.

Upstream closed the path

The behavior was reported as openai/openai-agents-js#1544. Maintainer pull request #1546 ("fail closed without financial research sources") merged into main as 6483fef about ten hours after the report and closed the issue as completed.

The reproduction in the teardown repo stays pinned to 710cccf on purpose — a commit that predates the fix — so the recorded baseline remains checkable after the code moved on. The path described here is closed in current main.

The same class — a failed step treated as a finished one — also turned up in a scheduled maintenance agent in google/adk-python. A collaborator reproduced it; a fix pull request is still under review. That is a separate frozen path, not evidence that anything about ADK is already resolved.

Reproduce without a key

npm ci
npm run verify

Prerequisites: Git, Node.js 22+, Corepack, and network access for the initial clone and locked install. No model API key, paid API, or database. On a fresh clone the full gate was measured at 184 seconds, which includes cloning the frozen target; with that clone already cached, three later runs measured 90, 117 and 119 seconds. Timings move with machine load, so treat these as observations rather than guarantees.

npm run verify sets up the detached frozen target, reproduces the canonical baseline (4/6), applies the patch for the remediation run (6/6 plus upstream compatibility checks), reverses the patch, and validates hashes and claims. Full constructive analysis lives in the teardown repository's TEARDOWN.md.

Probe your own agent in ten minutes

You do not need this harness. The method is one sentence: break a step on purpose with a deterministic fake, and see whether anything downstream turns red.

Work through these against one real code path:

1. Make the tool boundary raise. Replace one client or wrapper with a stand-in that raises immediately. If the summary still counts the item as processed, or a final artifact is still produced, the path is fail-open.
2. Read the exit status, not the log. Run the entry point the way your scheduler does and check the status. Status zero after logged errors usually means a catch-and-fall-through.
3. Exhaust the retry budget. Force verification or retry to stay negative on every attempt. If the loop ends and the next stage still runs, exhaustion was treated as permission to continue.
4. Return empty instead of raising. Many pipelines only handle exceptions. An empty list or null flowing into a report is the same defect with quieter symptoms.
5. Run zero units of work. Point the runner at an empty input set. A green pass rate with nothing measured is not a pass — it is an unmeasurable run reported as success.
6. Fail one item out of several. Partial failure is where "attempts" and "successes" diverge. If the summary counts the whole batch, the accounting is wrong.
7. Blind a guard. Delete the file a check reads, revoke a credential, or hand it malformed input. A guard that still passes with nothing to inspect was decoration.
8. Aggregate over nothing. Ask for a metric over an empty result set. A number where the honest answer is "not measured" — especially when lower-is-better metrics report an ideal empty value — is fail-open one level up.

If any probe stays green where it should turn red, you already have a deterministic reproduction: the fake you just wrote.

What this does not say

These results do not measure model quality, financial correctness, security, production frequency, or the Agents SDK beyond one frozen example. Fake boundaries make every figure synthetic-orchestration evidence. That limitation is deliberate: the failure under study is in the accounting of failed steps, not in model intelligence.

The general lesson

A system that cannot tell a failed step from a finished one fails open. The most persuasive form of that failure is not a crash. It is a fluent report written after the evidence is gone.

Define the minimum usable evidence before synthesis. Treat retry exhaustion as a terminal state. Assert side effects — downstream calls, emission, exit status — not only returned objects. And when you break a step on purpose, believe the surface that says "done" only if every other surface agrees.

---

Kerem Turhan. Reproduction and write-up: agent-reliability-teardown-openai-agents-js.