DEV Community

skyestrela
skyestrela

Posted on • Originally published at ai-agent-skills-pack.vercel.app

Stop Asking AI Coding Agents to Fix Vague Bugs

#ai #devops #productivity #testing

A coding agent can produce a confident patch for the wrong problem when the input is only:

The upload sometimes fails. Please fix it.

That sentence does not identify the smallest failing input, exact error, environment, expected result, frequency, or even whether the reporter reproduced it personally.

If the first instruction is “fix it”, the agent has room to turn a suspected cause into a fictional fact. The safer sequence is:

  1. preserve the observed failure;
  2. record the real environment;
  3. state expected versus actual behaviour;
  4. reduce the failing case;
  5. prove whether it repeats;
  6. diagnose and repair only after that evidence exists.

1. Preserve the observed failure

Capture the exact error, status code, incorrect output, affected route or command, timestamp, and smallest known input.

Remove secrets and personal data before putting logs into an agent context. If the report came from another person and you have not reproduced it, label it as second-hand rather than silently upgrading it to fact.

Weak:

CSV imports are broken.
Enter fullscreen mode Exit fullscreen mode

Useful:

At 14:22 UTC, POST /imports returned HTTP 500 for minimal.csv.
Response: "column index out of range".
The same account can import one-column.csv successfully.
Enter fullscreen mode Exit fullscreen mode

2. Record the environment you can prove

Inspect rather than guess:

  • repository and commit;
  • runtime version;
  • operating system or container image;
  • package lockfile;
  • relevant feature flags;
  • local, test, staging, or production target.

Do not infer production configuration from your laptop. Environment differences are often part of the bug.

3. Separate expected and actual behaviour

Write two observable statements:

Expected: POST /imports accepts the smallest valid two-column CSV and returns HTTP 201.
Actual:   The same fixture returns HTTP 500 with "column index out of range".
Enter fullscreen mode Exit fullscreen mode

Neither statement should include the suspected root cause.

“Expected: parser handles the off-by-one bug” already assumes the diagnosis. You have not earned that conclusion yet.

4. Reduce the reproduction

Start with the reported path. Remove unrelated records, services, fields, and steps one at a time.

Keep the smallest fixture that still fails. When removing one condition stops the failure, restore it and record the boundary.

Useful reductions include:

  • replacing a customer export with a sanitised two-row fixture;
  • removing unrelated network requests;
  • disabling background jobs that are not required;
  • running one focused test instead of an entire suite;
  • replacing a UI journey with the smallest direct request that proves the same fault.

Do not mutate production data merely to make reproduction convenient.

5. Prove repeatability

Run the minimal case at least twice where safe.

If it fails two times out of ten, report 2/10 over 90 seconds. Do not call it deterministic. Frequency and duration are evidence too.

A useful handoff looks like this:

# Bug Reproduction Brief
- Target and commit:
- Environment:
- Expected:
- Actual:
- Minimal steps:
- Minimal fixture:
- Reproduced: yes / no / intermittent
- Evidence:
- Unknowns:
- Safe next hypothesis to test:
Enter fullscreen mode Exit fullscreen mode

Keep reproduction separate from repair

A verified reproduction is a stable contract for the next step. Now the agent can inspect the responsible code, rank hypotheses, create a failing regression test, apply the smallest fix, and prove that the fixture passes.

Without that contract, a green test may only prove that the agent tested its own assumption.

Install the complete free workflow

The MIT-licensed Bug Reproduction Brief is public and readable before installation:

npx skills add skyestrela/ai-agent-skill-preview \
  --skill bug-reproduction-brief --agent codex --global --yes --copy
Enter fullscreen mode Exit fullscreen mode

Source and three other free engineering workflows:

https://github.com/skyestrela/ai-agent-skill-preview?utm_source=devto&utm_medium=article&utm_campaign=bug-reproduction-guide

The full Markdown workflow can also be read directly:

https://ai-agent-skills-pack.vercel.app/preview/bug-reproduction-brief.md?utm_source=devto&utm_medium=article&utm_campaign=bug-reproduction-guide

If you want the wider engineering set, the £19 AI Agent Skills Pack contains ten editable workflows for review, debugging, TDD, security, APIs, migrations, deployment, refactoring, PR shipping, and incident analysis:

https://ai-agent-skills-pack.vercel.app/?utm_source=devto&utm_medium=article&utm_campaign=bug-reproduction-guide

The free workflow is complete on its own. The paid pack is optional, sold once with no subscription.

Top comments (0)