Before Asking an AI Coding Agent to Fix a Bug, Verify the Bug Exists

Before asking an AI coding agent to “fix this bug,” there is one question worth asking first:

Does the bug actually exist?

An issue says it does. A review points it out. Someone reports seeing it in another environment.

Any of those is a good reason to investigate. None of them, by itself, proves that the problem currently exists exactly as described.

It may already be fixed. It may only happen in a particular version or environment. The reported behavior may even be expected.

As I have started relying more on AI coding agents, I have become more deliberate about checking this step.

A wrong premise can still produce a perfectly reasonable implementation

Suppose you give an agent this task:

Problem X is happening. Find the cause and fix it.

The agent can inspect the code, reason under the assumption that X exists, and produce a plausible fix.

That is not necessarily an AI capability problem.

If the input says “X is happening,” solving the problem under that premise is a reasonable thing to do.

The part I want to question happens one step earlier.

Bug report
 ↓
Fix

becomes:

Bug report
 ↓
Does the reported problem actually occur?
 ↓
Fix

Before implementation, verify the premise the implementation depends on.

I had a review finding that changed after I measured it

This happened publicly while I was working on PR #28 in my OSS project, spec-lane.

The PR added a gate that can execute an external verification command. During review, a strong claim came up around timeout behavior:

Even with SIGKILL, the CLI timeout itself may hang.

If I had accepted that statement as the premise, the next task would have been straightforward: implement a fix for a hanging timeout.

Instead, I measured it first.

Under the three conditions I tested, I could not reproduce a call exceeding the configured deadline.

That did not mean there was no problem.

I found a different boundary.

Even after the direct child process exits, a descendant process that still holds the inherited stdout/stderr pipes can affect how long spawnSync waits.

In one measurement, the direct child exited immediately while a grandchild remained alive for four seconds. The call took 3112 ms.

But when the grandchild remained alive for ten seconds, a two-second timeout returned in 2035 ms, and a 0.5-second timeout returned in 501 ms.

In the conditions I measured, the configured deadline still held.

So these were two different statements:

Original strong claim:
SIGKILL can allow the timeout itself to hang past its deadline.

What I could actually observe:
Descendants can influence waiting and timeout reporting within the deadline.

They sound similar, but they lead to different implementation work.

I did not add a new “hang fix.”

Instead, I changed the specification and tests to describe the boundary I could actually measure.

Measuring the premise changed the problem I was about to implement.

One important qualification: this incident was not automatically stopped by spec-lane's premise_evidence gate. It was a manual review-and-measurement decision made while developing the project.

Put “verify” between “reported” and “fix”

I now think about the flow like this:

Issue / review finding
          ↓
Verify the premise
          ↓
      Reproduced?
       ↙       ↘
     No         Yes
     ↓           ↓
Stop or        Spec
re-scope         ↓
              Implement

If the issue reproduces, proceed with the specification and implementation.

If it does not, do not force yourself—or an agent—to find a fix for the problem you expected to see.

Sometimes the right answer is to stop.

Sometimes, as in PR #28, the investigation reveals a different, measurable problem. In that case, change the scope to the problem that actually exists.

This extra step matters more as implementation becomes cheaper.

spec-lane can record that decision before implementation

I wanted this decision to exist in the development workflow rather than only in someone's memory.

In spec-lane 0.9.0, an intent can contain an optional premise_evidence record.

For example, when you decide that a change requires premise verification but fail to reproduce the reported problem:

premise_evidence:
  required: true
  method: live
  reproduced: false
  evidence: "Reported steps were executed against the current build, but the reported behavior was not observed."

Here, required: true records the decision that premise verification applies to this change.

reproduced: false records that the premise was not confirmed.

With that current record, spec-lane refuses the 1_intent -> 2_spec transition. A failed lane advance does not modify lane-state.json.

The purpose is not to have a CLI magically detect whether an AI is wrong.

The purpose is simpler.

Once the workflow has explicitly recorded “we could not confirm this problem,” do not silently continue into specification and implementation as if it had been confirmed.

Recorded evidence is not the same thing as truth

There is an important limit to this mechanism.

Suppose the file says:

reproduced: true

The CLI can verify that true was recorded.

It cannot verify that someone honestly reproduced the bug.

The same applies to:

method: live

The CLI can verify the field and its value. It cannot know whether somebody actually inspected a live system.

spec-lane also enforces a minimum length for the evidence text. That is a structural threshold, not a quality score.

A longer evidence string is not automatically better evidence.

The boundary I use is:

CLI proves recorded shape and transition behavior, not real-world truth.

The system can mechanically inspect the record and decide whether a transition is allowed.

The truth of the real-world observation remains outside that guarantee.

Missing premise evidence does not automatically block a lane

premise_evidence itself is optional.

If it is absent, the current CLI emits a warning rather than a hard error. A warning alone does not prevent the transition to the next phase.

That is deliberate.

The CLI cannot determine whether every change actually requires premise verification.

A bug already observed directly by a human is different from an unverified report. A feature addition may not have a “does this bug exist?” premise at all.

Making the CLI decide applicability on its own would introduce another form of false certainty.

So spec-lane starts enforcing after a human or agent has decided that premise verification applies and recorded the result.

It is a mechanical backstop for an explicit decision, not an oracle.

Faster implementation makes the question before implementation more important

AI coding agents have made the implementation step dramatically faster.

That makes the decision immediately before implementation more consequential.

What are we changing?

Why are we changing it?

Does the problem this change depends on actually occur?

If that premise is wrong, faster implementation only lets us move in the wrong direction faster.

So before handing an issue or review finding to an agent and saying “fix it,” I now want one more step:

Does this problem actually exist?

If it reproduces, proceed.

If it does not, stop.

If the investigation reveals a different problem, change the scope to what you can actually observe.

That is the role premise evidence plays in spec-lane: preserve that decision before implementation and provide a mechanical backstop against ignoring an explicitly failed premise.

Project: shiki-yusuke/spec-lane

Japanese version: Zenn

When you use AI coding agents, where in your workflow do you verify that the problem they are about to fix is real?