How I Taught My AI Coding Agent to Say "I Don't Know" Instead of Guessing

TL;DR

I spent months watching my autonomous coding agent confidently tell me things that weren't true — "this function is called from three places," "the bug is in the auth middleware" — when it hadn't actually checked. So I built an explicit uncertainty layer: the agent now has to ground every factual claim in a tool call, or flag it as unverified. Confident-but-wrong dropped hard. Here's how I built it and what it cost me in speed.

The Problem

Early on, my agent felt smart. It would read a stack trace, glance at a file, and immediately explain what was wrong with total confidence. Half the time it was right. The other half, it was making something up that sounded exactly as confident as the correct answer — same tone, same certainty, zero difference in how it was phrased.

That's the actual danger. A wrong answer that sounds uncertain, you double-check. A wrong answer that sounds like a fact, you ship.

The worst instance: I asked it to explain why a test was flaky. It gave me a clean, plausible paragraph about a race condition in a specific function. I believed it, "fixed" the race condition, shipped it, and the test kept failing — because the function it named didn't even call the code path in question. It had pattern-matched "flaky test" to "probably a race condition" and backfilled a story that fit, without ever grepping the actual call chain.

That's not a reasoning failure. It's a calibration failure — the agent had no mechanism to distinguish "I traced this and I'm sure" from "this is my best guess based on the vibe of the code." Both came out as the same declarative sentence.

How I Solved It

The fix wasn't a smarter model. It was forcing a structural separation between claims backed by a tool call and claims that are inference, and making the second category visibly different in the output.

1. Every factual claim needs a citation or a flag

I rewrote the agent's system prompt to enforce a simple rule: any sentence that states a fact about the codebase (what a function does, what calls it, where a bug lives) must either:

• reference a specific tool call result (a grep match, a read file, a trace) from this session, or
• be explicitly marked as an assumption.

GOOD:
"`validateToken()` is called from 3 places (confirmed via grep):
`auth/middleware.ts:42`, `auth/refresh.ts:18`, `test/auth.spec.ts:91`."

BAD (now blocked):
"validateToken() is used throughout the auth flow."

FLAGGED (allowed, but marked):
"[unverified] This is likely a race condition based on the error
pattern — I haven't traced the actual call sequence yet."

The second example isn't banned — sometimes a hunch is useful — but it can't be dressed up as a fact anymore.

2. A verification gate before "done"

I added a step between "agent thinks it's finished" and "agent reports done" that scans its own output for factual-sounding claims and cross-checks them against the tool calls actually made in that session.

# simplified version of the check
def find_unverified_claims(response_text, tool_calls_made):
    claims = extract_factual_sentences(response_text)
    unverified = []
    for claim in claims:
        if not is_flagged(claim) and not has_supporting_tool_call(claim, tool_calls_made):
            unverified.append(claim)
    return unverified

If find_unverified_claims returns anything, the agent doesn't get to just say it — it either goes back and verifies (runs the grep, reads the file) or downgrades the sentence to an explicit [unverified] tag. No silent middle ground.

3. Confidence as a first-class field, not a vibe

I stopped letting confidence live in word choice ("probably," "likely," "definitely") because those words are cheap and the model uses them inconsistently. Instead, every diagnosis or fix proposal now carries a structured confidence field alongside it:

{
  "claim": "Bug is caused by unhandled promise rejection in retry logic",
  "confidence": "verified",
  "evidence": ["read: retry.ts:12-40", "trace: 3 call sites, none catch rejection"]
}

versus:

{
  "claim": "Bug might be related to caching, based on symptom pattern",
  "confidence": "hypothesis",
  "evidence": []
}

Same downstream consumer (me, or the next step in the workflow) can now branch on confidence instead of parsing tone. verified claims get acted on. hypothesis claims trigger a "go verify this first" loop instead of a fix attempt.

flowchart TD
    A[Agent makes a claim] --> B{Backed by a tool call this session?}
    B -- Yes --> C[Tag: verified + cite evidence]
    B -- No --> D{Explicitly flagged as guess?}
    D -- Yes --> E[Tag: hypothesis]
    D -- No --> F[Blocked, go verify or downgrade]
    C --> G[Safe to act on]
    E --> H[Triggers verification step before acting]
    F --> B

4. Teaching it when to say "I don't know"

The hardest part wasn't blocking false confidence — it was getting the agent to actually stop and say "I need to check this" instead of finding a way to phrase around the gate. Early versions would technically satisfy the rule by adding a fig-leaf citation that didn't really support the claim.

What actually worked: I added a handful of few-shot examples in the system prompt showing the contrast — the same diagnosis done lazily (confident, wrong) versus done properly (verified, cited, sometimes slower). Seeing the paired examples side by side did more than any instruction to "be careful" ever did.

5. Making verification cheap enough that it doesn't get skipped

None of this works if verifying a claim is expensive, because then the incentive is to skip it. So I made the common verification paths as cheap as possible: a targeted grep for "does X call Y" is a few hundred milliseconds, not a full-file read. Once checking a claim costs less than writing the confident-sounding sentence for it, there's no reason left to skip the check. That sounds obvious in hindsight, but early on my verification step was doing a broad codebase scan every time, which was slow enough that the agent (and honestly, me too) started looking for shortcuts around it.

A concrete before/after

Here's the same bug report, from before and after this change, so the difference isn't abstract:

Before:

The intermittent 500 errors are almost certainly caused by the connection pool exhausting under load — the retry logic doesn't back off properly, so failed requests pile up and starve the pool.

Confident, specific-sounding, and entirely unverified. It reads like something someone checked. Nobody checked it.

After:

[hypothesis, unverified] The intermittent 500s could be connection pool exhaustion — this pattern (spikes under load, recovers on its own) is consistent with that. I haven't confirmed the pool size or checked whether retries are actually piling up.

Verifying now: read pool.ts for the configured max connections, then traced live connection count during a failure window.

[verified] Pool max is 20. During the failure window, active connections peaked at 19 with 40+ queued requests — this is pool exhaustion, not a red herring. Retry backoff is fixed-delay, not exponential, which is why failed requests re-queue faster than the pool drains.

The second version takes longer to produce and read. It's also the one where I can actually trust the fix that follows it, because I can see exactly what was checked and what was assumed going in.

The cost, in numbers

I tracked this for about three weeks before and after turning the gate on, same mix of task types both times as best I could control for it:

Metric	Before	After
Median time to first diagnosis	~40s	~65s
Diagnoses later found to be wrong	~1 in 4	~1 in 12
Diagnoses explicitly flagged as unverified	0% (no mechanism)	~30%
Fixes shipped based on a wrong diagnosis	not tracked, but it happened often enough to prompt this whole project	0 in the tracking window

Slower per diagnosis, way fewer wrong ones shipped. That trade was an easy call once I actually had numbers instead of a vague feeling that "sometimes it's wrong."

Lessons Learned

1. Confidence and correctness are different axes, and your agent's language conflates them by default. A model that's 90% sure and a model that's 10% sure will both write "the issue is X" unless you force a structural difference. Fix the structure, not the wording.

2. "I don't know" has to be a cheap, normal output, not a failure state. If flagging uncertainty feels like the agent is admitting defeat, it'll avoid doing it. I had to make hypothesis a totally normal, expected tag, not something that reads as an apology.

3. Verification gates slow things down, and that's the point. My agent got measurably slower on ambiguous tasks after I added this. I was tempted to loosen the gate. I didn't, because the alternative was shipping the fast, wrong answer instead.

4. Citations have to point at this session's tool calls, not general plausibility. I initially let the agent cite "based on typical patterns in codebases like this" as evidence. That's not evidence, that's a prior. It took an embarrassing amount of debugging to realize I'd left that loophole open.

5. You will find out how often your agent was already wrong, and it's higher than you think. Once claims are tagged, you get to see the actual verified-vs-hypothesis ratio over time. Mine was worse than I expected in week one — a genuinely humbling number to stare at.

What's Next

I'm working on letting the hypothesis tag drive actual behavior — instead of just being visible metadata, an unverified claim should automatically trigger a cheap, targeted verification step (one grep, one read) before the agent is allowed to act on it, rather than relying on me reading the tag and deciding manually. Right now that loop is still half-manual, and closing it fully is the next milestone.

Wrap-up / CTA

If you're building anything agentic and it's making claims about a codebase, a system, or data it hasn't directly checked, go look at how those claims are phrased. If a guess and a fact read the same, that's the bug to fix first.

Curious what other calibration tricks people are using — drop a comment if you've solved this differently. And if you're deep in Claude Code agent design, follow me here, I write about this stuff regularly.