DEV Community

Manohar Papasani
Manohar Papasani

Posted on

How do you reduce hallucination?

#aiinterview #genai #aiarchitect #llm

Asked in an AI engineer interview.

What they are really testing: whether you treat hallucination as a system-level problem with layered controls (grounding, retrieval quality, prompting, verification, and evaluation), rather than one trick like "lower the temperature."

A strong answer:

  • Ground the model: use RAG so answers come from retrieved passages, and instruct the model to answer only from context and to say "I don't know" when the context is insufficient.
  • Fix retrieval first: most RAG hallucinations are retrieval failures. Tune chunking, use hybrid search (BM25 plus dense vectors), add a reranker, and tune top-k so the context is relevant without flooding the window.
  • Constrain generation: lower temperature for factual tasks, require citations with source IDs, and use structured or JSON output with schema validation where the answer has a fixed shape.
  • Verify after generation: check that each claim is entailed by the retrieved sources (NLI model or an LLM judge), and run a self-consistency or multi-sample check where disagreement flags low confidence.
  • Route and abstain: when confidence or verification fails, return a refusal, a cited partial answer, or escalate to a human instead of a guessed answer.
  • Measure continuously: build a golden dataset with faithfulness, answer relevance and context precision metrics (for example with RAGAS or TruLens), track them in CI and in production, and alert on regressions.

Common mistakes:

  • Treating temperature 0 as a fix. It makes output deterministic, not correct, and a confidently wrong answer stays wrong.
  • Blaming the model before inspecting retrieval. If the right passage was never fetched, no prompt will make the answer grounded.
  • Stopping at prompt engineering with no evaluation set, so there is no way to tell whether the fix worked.

How would you order these controls for a production system with a tight latency budget? I'd be interested in where you would draw the line between verification and abstention.

Originally shared on LinkedIn.

An AI interview question every day, leadership for tech people and hands-on AI learning:

Top comments (0)