I make an open-source bug-reproduction workflow for coding agents. I wanted to test whether it changed the completeness of a Codex investigation, rather than relying on the workflow sounding sensible.
The result was 6/14 for the concise control and 14/14 for the corrected workflow-assisted run.
That headline needs context. This was one synthetic Python fixture, one model and a self-authored rubric. The workflow used more tokens and produced a much longer answer. The first workflow run did not score 14/14 either: it scored 12/14, exposed two weaknesses, and led to a revision before the final run.
Everything needed to inspect the comparison is public:
https://github.com/skyestrela/ai-agent-skill-preview/tree/main/evidence/bug-reproduction-benchmark
The question
Does supplying a structured Bug Reproduction Brief make a vague-bug investigation more complete and auditable than a concise task instruction alone?
The test was not designed to measure implementation speed or whether Codex could patch the bug. Both prompts explicitly prohibited modifying files or proposing a fix. The boundary was reproduction and evidence collection.
Controls
Both runs used:
- the same authenticated Codex model:
gpt-5.4-mini; - the same committed Python fixture:
a4b9eab9dcc7b2ebbfe5f5d0502d4866cefd36ce; - read-only Codex sandboxes;
- ephemeral sessions with user configuration and repository rules ignored;
-
PYTHONDONTWRITEBYTECODE=1; - clean working trees before and after;
- the same instruction not to modify files or propose or implement a fix.
The workflow-assisted run additionally received Bug Reproduction Brief v1.0.1.
The fixture contains an intentionally faulty invoice calculation. Its existing tests pass because they do not cover the explicit-zero edge case described in the support report.
What both runs found
Both the control and workflow-assisted run:
- identified the explicit-zero input that fails;
- stated the expected and observed output;
- ran the existing tests;
- provided a runnable reproduction;
- left the fixture unchanged;
- avoided proposing a fix.
This matters because the 6/14 control was not useless. It found the central failure quickly.
What the workflow added
The workflow-assisted answer also recorded:
- the immutable commit;
- the inspected runtime environment;
- that the intake report was second-hand and unverified;
- an explicit minimal fixture;
- two separate reproductions;
- unresolved unknowns;
- a safe next hypothesis;
- a strict boundary between reproduced observations and diagnosis.
The concise control jumped from the reproduction to a root-cause statement. The final workflow answer deliberately stopped at observable evidence and a testable next hypothesis.
A deterministic 14-point rubric scored those criteria. The criterion-level result is committed as score.json, alongside the exact prompts and final outputs.
The failed first workflow run
The first workflow-assisted run scored 12/14, not 14/14.
It exposed two weaknesses in the workflow:
- the answer did not explicitly preserve the report's second-hand, unverified provenance;
- it stated a causal explanation despite the reproduction-only boundary.
I tightened the public skill to v1.0.1 and reran it against the unchanged fixture. The 14/14 result came from that corrected workflow.
That makes this an iterative product-development comparison, not a blinded academic study. I am disclosing the iteration because hiding the 12/14 run would make the final result look cleaner than the process actually was.
Token and verbosity cost
The additional completeness was not free.
| Run | Input tokens | Cached input | Output tokens | Reasoning output |
|---|---|---|---|---|
| Control | 68,604 | 62,720 | 1,400 | 255 |
| Workflow | 74,547 | 67,328 | 4,471 | 2,661 |
The workflow used 8.7% more input tokens and produced a substantially longer answer.
That trade-off will not suit every task. For a low-risk bug where speed matters more than auditability, the concise instruction may be sufficient. Teams adopting a structured brief should shorten its output contract rather than assuming maximum detail is always better.
What this result does not prove
It does not prove that:
- every coding agent improves with this workflow;
- the result repeats across repositories or models;
- a 14/14 reproduction brief leads to a better patch;
- the extra tokens provide positive economic value;
- the workflow is universally better than a concise prompt.
Outputs can vary between repeated model runs. The rubric rewards reproduction-brief completeness, not bug-fix quality or developer productivity.
A stronger follow-up would run multiple trials across several repositories and models using a rubric fixed before execution.
Reproduce or criticise it
The repository contains:
- the exact control and workflow prompts;
- the intentionally faulty fixture and passing test suite;
- the final control and workflow outputs;
- criterion-level scoring and token usage;
- the complete MIT-licensed workflow.
Benchmark and reproduction files:
Workflow source:
The most useful feedback would be criticism of the controls, rubric or reproduction boundary—not generic agreement with the score.
Disclosure
I created this workflow and sell an optional wider engineering pack containing ten editable workflows. The complete Bug Reproduction Brief used here is already public and MIT licensed; buying anything is not required to inspect or reproduce this comparison.
Optional wider pack:
Top comments (0)