DEV Community: skyestrela

A Transparent Codex Bug-Reproduction Comparison: 6/14 vs 14/14

skyestrela — Thu, 30 Jul 2026 15:06:39 +0000

I make an open-source bug-reproduction workflow for coding agents. I wanted to test whether it changed the completeness of a Codex investigation, rather than relying on the workflow sounding sensible.

The result was 6/14 for the concise control and 14/14 for the corrected workflow-assisted run.

That headline needs context. This was one synthetic Python fixture, one model and a self-authored rubric. The workflow used more tokens and produced a much longer answer. The first workflow run did not score 14/14 either: it scored 12/14, exposed two weaknesses, and led to a revision before the final run.

Everything needed to inspect the comparison is public:

https://github.com/skyestrela/ai-agent-skill-preview/tree/main/evidence/bug-reproduction-benchmark

The question

Does supplying a structured Bug Reproduction Brief make a vague-bug investigation more complete and auditable than a concise task instruction alone?

The test was not designed to measure implementation speed or whether Codex could patch the bug. Both prompts explicitly prohibited modifying files or proposing a fix. The boundary was reproduction and evidence collection.

Controls

Both runs used:

the same authenticated Codex model: gpt-5.4-mini;
the same committed Python fixture: a4b9eab9dcc7b2ebbfe5f5d0502d4866cefd36ce;
read-only Codex sandboxes;
ephemeral sessions with user configuration and repository rules ignored;
PYTHONDONTWRITEBYTECODE=1;
clean working trees before and after;
the same instruction not to modify files or propose or implement a fix.

The workflow-assisted run additionally received Bug Reproduction Brief v1.0.1.

The fixture contains an intentionally faulty invoice calculation. Its existing tests pass because they do not cover the explicit-zero edge case described in the support report.

What both runs found

Both the control and workflow-assisted run:

identified the explicit-zero input that fails;
stated the expected and observed output;
ran the existing tests;
provided a runnable reproduction;
left the fixture unchanged;
avoided proposing a fix.

This matters because the 6/14 control was not useless. It found the central failure quickly.

What the workflow added

The workflow-assisted answer also recorded:

the immutable commit;
the inspected runtime environment;
that the intake report was second-hand and unverified;
an explicit minimal fixture;
two separate reproductions;
unresolved unknowns;
a safe next hypothesis;
a strict boundary between reproduced observations and diagnosis.

The concise control jumped from the reproduction to a root-cause statement. The final workflow answer deliberately stopped at observable evidence and a testable next hypothesis.

A deterministic 14-point rubric scored those criteria. The criterion-level result is committed as score.json, alongside the exact prompts and final outputs.

The failed first workflow run

The first workflow-assisted run scored 12/14, not 14/14.

It exposed two weaknesses in the workflow:

the answer did not explicitly preserve the report's second-hand, unverified provenance;
it stated a causal explanation despite the reproduction-only boundary.

I tightened the public skill to v1.0.1 and reran it against the unchanged fixture. The 14/14 result came from that corrected workflow.

That makes this an iterative product-development comparison, not a blinded academic study. I am disclosing the iteration because hiding the 12/14 run would make the final result look cleaner than the process actually was.

Token and verbosity cost

The additional completeness was not free.

Run	Input tokens	Cached input	Output tokens	Reasoning output
Control	68,604	62,720	1,400	255
Workflow	74,547	67,328	4,471	2,661

The workflow used 8.7% more input tokens and produced a substantially longer answer.

That trade-off will not suit every task. For a low-risk bug where speed matters more than auditability, the concise instruction may be sufficient. Teams adopting a structured brief should shorten its output contract rather than assuming maximum detail is always better.

What this result does not prove

It does not prove that:

every coding agent improves with this workflow;
the result repeats across repositories or models;
a 14/14 reproduction brief leads to a better patch;
the extra tokens provide positive economic value;
the workflow is universally better than a concise prompt.

Outputs can vary between repeated model runs. The rubric rewards reproduction-brief completeness, not bug-fix quality or developer productivity.

A stronger follow-up would run multiple trials across several repositories and models using a rubric fixed before execution.

Reproduce or criticise it

The repository contains:

the exact control and workflow prompts;
the intentionally faulty fixture and passing test suite;
the final control and workflow outputs;
criterion-level scoring and token usage;
the complete MIT-licensed workflow.

Benchmark and reproduction files:

https://github.com/skyestrela/ai-agent-skill-preview/tree/main/evidence/bug-reproduction-benchmark?utm_source=devto&utm_medium=article&utm_campaign=bug-reproduction-benchmark

Workflow source:

https://github.com/skyestrela/ai-agent-skill-preview/blob/main/bug-reproduction-brief/SKILL.md?utm_source=devto&utm_medium=article&utm_campaign=bug-reproduction-benchmark

The most useful feedback would be criticism of the controls, rubric or reproduction boundary—not generic agreement with the score.

Disclosure

I created this workflow and sell an optional wider engineering pack containing ten editable workflows. The complete Bug Reproduction Brief used here is already public and MIT licensed; buying anything is not required to inspect or reproduce this comparison.

Optional wider pack:

https://ai-agent-skills-pack.vercel.app/?utm_source=devto&utm_medium=article&utm_campaign=bug-reproduction-benchmark

Six Checks Before You Trust an AI Agent Skill

skyestrela — Sun, 26 Jul 2026 07:10:00 +0000

Agent skills are becoming a convenient way to reuse instructions across coding sessions. Instead of pasting the same review, debugging or deployment prompt repeatedly, you put the process in a readable SKILL.md file and load it when the job matches.

That convenience creates a new review problem: a polished workflow can still be unsafe, vague or impossible to verify.

Before installing a third-party skill—or trusting one written inside your own team—I look for six things.

1. It starts with discovery, not a guessed fix

A trustworthy workflow tells the agent to inspect the real target before making claims.

For a bug, that might mean:

identify the repository and commit;
capture the exact failure;
record the runtime and environment;
reduce the failing input;
reproduce it before editing code.

For a deployment check, discovery should identify the actual platform, project, branch, build command and production URL. “Check whether the app is deployed” is too vague if the agent can silently choose the wrong environment.

Weak skills jump from the request to an action. Strong skills establish what is true first.

2. Success is observable

“Make the code better” is not a useful completion condition.

A skill should define outputs that another person can inspect. Examples include:

- failing test reproduced before the patch;
- focused regression test passes after the patch;
- full relevant test suite returns exit code 0;
- live endpoint returns the expected status and body;
- rollback command is recorded and tested where safe.

This changes the agent’s job from producing a plausible answer to producing evidence.

The evidence does not need to be elaborate. A command, exit code, short log excerpt and exact URL are often enough. What matters is that the workflow distinguishes “I changed it” from “I exercised the result”.

3. Permissions and destructive boundaries are explicit

A reusable skill should say what the agent must not do automatically.

Look for boundaries around:

deleting files or data;
rotating or exposing credentials;
changing production infrastructure;
creating charges or purchases;
force-pushing or rewriting history;
sending messages or publishing content;
disabling security controls.

“Use caution” is not a boundary. A better instruction is: “Do not delete, force-push or mutate production data without explicit approval. Prefer read-only inspection and reversible changes first.”

The workflow should also refuse to guess missing credentials. Asking the operator or using an already-authorised session is safer than inventing an access path.

4. Reversible actions come first

Good operational workflows order actions by risk.

A dependency triage should inspect the lockfile, changelog and usage before upgrading. A migration workflow should copy, verify and only then consider removal. A deployment workflow should preserve the previous known-good release and record the rollback path before promotion.

This is not bureaucracy. It reduces the cost of being wrong.

A useful test is: if the agent’s leading hypothesis is false, does the workflow leave you with a clean way back?

5. Verification reaches the real target

Local validation is necessary, but it may not prove the thing the user asked about.

If the task is to fix a production page, a green local build does not prove production changed. If the task is to publish a package, creating an archive does not prove the registry serves it. If the task is to repair an API, a mocked unit test does not prove the live route is healthy.

A sound skill separates layers:

static checks;
focused tests;
broader regression tests;
build or package validation;
deployment confirmation;
live-target verification.

Not every task needs every layer. The skill should choose the smallest set that actually proves the requested result.

6. Unknowns stay labelled as unknowns

Agent output often becomes misleading when assumptions are written with the tone of facts.

A trustworthy workflow requires a final report that separates:

observed facts;
evidence;
assumptions;
unresolved risks;
actions not taken;
the next safe step.

If a network call failed, the report should say it failed. If the agent did not measure user comprehension, it should not convert a design estimate into a measured result. If no sale occurred, it should not call a checkout test “revenue”.

Plain uncertainty is more useful than confident fiction.

A compact review contract

You can use this checklist before adopting any agent skill:

- [ ] Inspects the real repository, environment or live target
- [ ] Defines observable success criteria
- [ ] Protects credentials, production data and destructive actions
- [ ] Uses read-only or reversible actions first
- [ ] Runs verification appropriate to the requested outcome
- [ ] Separates facts, assumptions, blockers and remaining risks

Because skills are plain text, you can review and version these rules like code. That is the main advantage: the workflow is not hidden behind a confident interface.

Four complete workflows you can inspect

I have published four MIT-licensed examples so the method can be reviewed before installing anything:

Code Review Gate
Bug Reproduction Brief
Dependency Risk Triage
Rollback Readiness Card

Source and install instructions:

https://github.com/skyestrela/ai-agent-skill-preview?utm_source=devto&utm_medium=article&utm_campaign=trustworthy-agent-skills

Download all four complete workflows in one verified MIT-licensed ZIP:

https://github.com/skyestrela/ai-agent-skill-preview/releases/tag/free-workflows-bundle-v1.0.0

List the available skills without installing them:

npx skills add skyestrela/ai-agent-skill-preview --list

The free workflows are complete on their own. An optional £19 pack contains ten engineering workflows for review, debugging, TDD, security, APIs, migrations, deployment, refactoring, PR shipping and incident analysis:

https://ai-agent-skills-pack.vercel.app/?utm_source=devto&utm_medium=article&utm_campaign=trustworthy-agent-skills

The pack launches on Product Hunt on 26 July. If you use coding agents in real repositories, direct feedback on the inspection contract is more useful than generic praise:

https://www.producthunt.com/products/ai-agent-skills-pack?launch=ai-agent-skills-pack

Stop Asking AI Coding Agents to Fix Vague Bugs

skyestrela — Sat, 25 Jul 2026 15:34:38 +0000

A coding agent can produce a confident patch for the wrong problem when the input is only:

The upload sometimes fails. Please fix it.

That sentence does not identify the smallest failing input, exact error, environment, expected result, frequency, or even whether the reporter reproduced it personally.

If the first instruction is “fix it”, the agent has room to turn a suspected cause into a fictional fact. The safer sequence is:

preserve the observed failure;
record the real environment;
state expected versus actual behaviour;
reduce the failing case;
prove whether it repeats;
diagnose and repair only after that evidence exists.

1. Preserve the observed failure

Capture the exact error, status code, incorrect output, affected route or command, timestamp, and smallest known input.

Remove secrets and personal data before putting logs into an agent context. If the report came from another person and you have not reproduced it, label it as second-hand rather than silently upgrading it to fact.

Weak:

CSV imports are broken.

Useful:

At 14:22 UTC, POST /imports returned HTTP 500 for minimal.csv.
Response: "column index out of range".
The same account can import one-column.csv successfully.

2. Record the environment you can prove

Inspect rather than guess:

repository and commit;
runtime version;
operating system or container image;
package lockfile;
relevant feature flags;
local, test, staging, or production target.

Do not infer production configuration from your laptop. Environment differences are often part of the bug.

3. Separate expected and actual behaviour

Write two observable statements:

Expected: POST /imports accepts the smallest valid two-column CSV and returns HTTP 201.
Actual:   The same fixture returns HTTP 500 with "column index out of range".

Neither statement should include the suspected root cause.

“Expected: parser handles the off-by-one bug” already assumes the diagnosis. You have not earned that conclusion yet.

4. Reduce the reproduction

Start with the reported path. Remove unrelated records, services, fields, and steps one at a time.

Keep the smallest fixture that still fails. When removing one condition stops the failure, restore it and record the boundary.

Useful reductions include:

replacing a customer export with a sanitised two-row fixture;
removing unrelated network requests;
disabling background jobs that are not required;
running one focused test instead of an entire suite;
replacing a UI journey with the smallest direct request that proves the same fault.

Do not mutate production data merely to make reproduction convenient.

5. Prove repeatability

Run the minimal case at least twice where safe.

If it fails two times out of ten, report 2/10 over 90 seconds. Do not call it deterministic. Frequency and duration are evidence too.

A useful handoff looks like this:

# Bug Reproduction Brief
- Target and commit:
- Environment:
- Expected:
- Actual:
- Minimal steps:
- Minimal fixture:
- Reproduced: yes / no / intermittent
- Evidence:
- Unknowns:
- Safe next hypothesis to test:

Keep reproduction separate from repair

A verified reproduction is a stable contract for the next step. Now the agent can inspect the responsible code, rank hypotheses, create a failing regression test, apply the smallest fix, and prove that the fixture passes.

Without that contract, a green test may only prove that the agent tested its own assumption.

Install the complete free workflow

The MIT-licensed Bug Reproduction Brief is public and readable before installation:

npx skills add skyestrela/ai-agent-skill-preview \
  --skill bug-reproduction-brief --agent codex --global --yes --copy

Source and three other free engineering workflows:

https://github.com/skyestrela/ai-agent-skill-preview?utm_source=devto&utm_medium=article&utm_campaign=bug-reproduction-guide

Prefer one download? The verified MIT-licensed bundle contains all four public workflows:

https://github.com/skyestrela/ai-agent-skill-preview/releases/tag/free-workflows-bundle-v1.0.0

The full Markdown workflow can also be read directly:

https://ai-agent-skills-pack.vercel.app/preview/bug-reproduction-brief.md?utm_source=devto&utm_medium=article&utm_campaign=bug-reproduction-guide

If you want the wider engineering set, the £19 AI Agent Skills Pack contains ten editable workflows for review, debugging, TDD, security, APIs, migrations, deployment, refactoring, PR shipping, and incident analysis:

https://ai-agent-skills-pack.vercel.app/?utm_source=devto&utm_medium=article&utm_campaign=bug-reproduction-guide

The free workflow is complete on its own. The paid pack is optional, sold once with no subscription.