Let your LLM say I don't know instead of guessing

Your user asks a factual question. The model returns a polished paragraph. Nothing in the stack threw an error. That is exactly when you are most exposed. A language model generates the most plausible next words, not the most true ones, so a wrong answer can look identical to a right one. That mechanism is what people call AI hallucination: same token prediction, missing or weak facts, fluent fiction at the end.

Dan Klein, co-founder and CTO of Scaled Cognition, put the internals this way on Chain of Thought: we label outputs after the fact, but the model never splits “truth” from “guess” on its own.

If they're right, we call them truth. And if they're wrong, we call them hallucination. But the model does not internally distinguish between those.

Dan Klein, co-founder and CTO of Scaled Cognition

I’d stop treating confidence in the UI as a proxy for correctness. Treat every fluent sentence as a claim you still owe evidence for.

Why does fluency hide the errors you care about?

Obvious mistakes get caught in QA. Klein’s audit story from enterprise work is worse: when his team audits enterprise systems, he says, the real hallucination rate is often about five times what the customer has noticed, because the rest are fluent enough to pass. His examples are a refund policy that is real and plausible but not yours, or yours but for a customer with a different status.

Looks right. It looks right and is right are not the same. It is very hard to detect hallucinations when they are packaged up sufficiently fluently.

Dan Klein, co-founder and CTO of Scaled Cognition

Chip Huyen made the data angle plain on Chain of Thought ep 5: missing or wrong context pushes the model toward invention.

Models are more likely to hallucinate when it does not have access to the necessary and correct information to answer questions.

Chip Huyen, author of AI Engineering

So the design job is to give the model the facts at answer time, allow silence when facts are missing, and verify what still matters. A smarter model does not remove that work.

What breaks when you only punish wrong guesses?

Pre-training shows the model fluent text, not labeled false statements. Rare facts (like a specific birthday) do not repeat as patterns, so the model still produces something that sounds like an answer. Benchmarks make the incentive worse: “I don’t know” often scores like a wrong answer, while a random guess sometimes hits. OpenAI’s 2025 write-up on why language models hallucinate argues for penalizing confident errors more than honest uncertainty. Anthropic’s interpretability work on Claude describes a default to decline on unrecognized entities, with a “known entity” signal that can flip that off and invite a guess anyway.

For your app, that means instructions that forbid uncertainty fight the same scoreboards that trained the model to guess. If your eval sheet marks abstain as a miss, you are training your team to ship guessers.

How do you ground answers and still leave an out?

Grounding means building the reply from material you put in context at question time, ideally with citations per claim. That cuts invented detail because the model has less room to freestyle. It does not guarantee truth: sources can be stale, retrieval can miss the doc, and the model can still stretch past what the sources say. Many production issues that look like “model bugs” are retrieval or document problems first.

General models also carry a knowledge cutoff: training frozen at a date, then released later. Ask about anything newer and you get “I don’t know” or a plausible guess. The standard fix is the same as grounding: fetch current or private text into the context window before generation (RAG, search, or both).

My default pattern for factual features:

1. Retrieve candidate passages for the question.
2. If nothing passes a relevance threshold, return abstain (or escalate to a human). Do not ask the model to “try anyway.”
3. If you have passages, instruct answer-only-from-these, cite span or doc id per claim.
4. Run a faithfulness check on answers that can move money, policy, or compliance.

Step 4 is where AI evaluation and reliability work pays off: separate counts for answered, abstained, answered_supported and answered_unsupported.

What does an abstain-and-ground sketch look like?

Label this as illustration, not production code from any guest or vendor.

# Sketch: abstain if retrieval is weak; answer only from chunks

MIN_SCORE = 0.42  # tune on your corpus

def answer(user_question: str, retrieve, llm, faithfulness_check):
    chunks = retrieve(user_question, top_k=8)
    best = max((c.score for c in chunks), default=0.0)
    if best < MIN_SCORE:
        return {"status": "abstain", "message": "No trusted source found."}

    prompt = {
        "instruction": (
            "Answer using ONLY the sources. "
            "If they do not support an answer, reply ABSTAIN."
        ),
        "sources": [c.text for c in chunks],
        "question": user_question,
    }
    draft = llm.complete(prompt)
    if "ABSTAIN" in draft:
        return {"status": "abstain", "message": "Model declined."}

    if not faithfulness_check(draft, chunks):
        return {"status": "abstain", "message": "Failed source check."}

    return {"status": "ok", "text": draft, "sources": chunks}

The abstain path is a first-class HTTP response, not a thrown exception buried in logs. Users should see “we don’t know” when you mean it.

Checklist

• Decide where a wrong fluent answer is costly; tighten grounding and checks there first.
• Retrieve before generate; abstain when retrieval is empty or weak instead of prompting harder.
• Tell the model abstaining is allowed; score abstain better than a confident wrong answer in evals.
• Check high-stakes outputs against supplied sources (faithfulness), not vibes.
• Remember fluent output is plausible text until your sources and checks say otherwise.

The longer explainer, with the episode clips, is on Chain of Thought. It draws on this episode.

Subscribe to the Chain of Thought newsletter for new episodes and write-ups like this one.

Drafted with AI assistance from the episode transcripts.