The hardest LLM bugs are contract failures, not hallucinations

rishabh jain — Sat, 20 Jun 2026 03:43:11 +0000

When people talk about LLM failures, the default word is usually "hallucination."

But after building and testing LLM apps, I think many production bugs are better described as contract failures.

A hallucination is when the model makes something up. That matters, but it is not the only failure mode.

The subtler bugs happen when the model had enough context, but violated the surrounding system contract.

Examples:

It skipped a required tool call.
It returned JSON that looked valid but broke the downstream schema.
It made a claim without the evidence artifact the workflow expected.
It ignored a tool result.
It answered before satisfying a validation step.
It followed one instruction while violating another.

Retrieval failures are usually easier to notice. You can compare the answer against the retrieved chunks and see that the grounding is weak.

Contract failures are harder because the answer may look plausible. The text might even be correct in isolation. But the system still failed because the model did not do the thing the application required.

For example, in a support agent, the problem may not be that the model gave a bad refund answer. The problem may be that it answered without first calling the refund eligibility tool.

In a data extraction app, the problem may not be that the model misunderstood the document. The problem may be that it returned almost-correct JSON that fails validation in the next service.

In a RAG workflow, the problem may not be missing context. The problem may be that the answer made a claim without attaching the citation or artifact that proves it.

This changed how I think about LLM debugging.

Prompt and response logs are useful, but they are not enough. A debugger should also inspect the contracts around the model call:

Was the required tool called?
Were the tool arguments valid?
Was the tool result used?
Did the output match the schema?
Was each claim supported by evidence?
Did the model satisfy the approval or safety preconditions?
Is this failure reproducible as a regression case?

This is the direction I am taking with DebugAI, a Python SDK I have been building.

The goal is to take a bad LLM response and return a structured debug artifact:

failure type
evidence
likely root cause
suggested fix
regression-testable case

For example, instead of only saying "bad answer," the diagnosis should say something like:

{
"failure": "tool_call_failure",
"evidence": [
"Expected refund_order tool before answering",
"No tool call was made"
],
"fix": "Require tool execution before final answer and reject responses without tool evidence."
}

I think this framing is more useful than treating every LLM bug as a hallucination.

Some bugs are grounding failures.
Some are retrieval failures.
Some are instruction failures.
Some are output validation failures.
Some are tool contract failures.

The more specific the failure class, the easier it is to fix and test.

Repo is public here if useful: https://github.com/civicRJ/DebugAI

How I debug RAG failures with deterministic signals

rishabh jain — Mon, 15 Jun 2026 18:24:59 +0000

When building LLM apps, one frustrating problem is that a response can be wrong for many different reasons.

The model may have hallucinated. The retriever may have pulled the wrong chunks. The answer may not be grounded in the provided context. A tool call may be missing. A JSON schema may have failed. Or the prompt may simply be brittle.

I started experimenting with a more structured way to debug these failures.

Instead of only asking another LLM "is this answer good?", I wanted to compute signals from the request and response itself.

The inputs I care about are usually:

prompt
model output
retrieved chunks
similarity scores
tool calls
expected tools
response schema
latency
model settings

From these, I try to classify the failure into categories like:

retrieval failure
RAG hallucination
schema violation
missing tool call
citation failure
prompt brittleness
ambiguous prompt
instruction-following issue

For example, if the retriever returns chunks with low similarity and the output answers confidently anyway, that points toward a retrieval or grounding problem.

If the chunks are relevant but the output introduces unsupported claims, that looks more like hallucination.

If a tool was expected but no tool call happened, that is a different failure mode entirely.

A simple example:

from debugai import analyze

result = analyze(
prompt="What is the refund policy?",
output="You can get a full refund after 90 days.",
chunks=["Electronics can be returned within 30 days with receipt."],
similarity_scores=[0.82],
)

print(result["primary"])

I’m building this into a small Python SDK called DebugAI.

Install:

pip install debugerai

The SDK can run locally without logging into a website. I also have a hosted dashboard for saving traces and diagnoses, but local usage is the main thing I’m testing right now.

The goal is not to magically solve LLM reliability. It is to make failures easier to name and inspect.

I’m still early, so I’m mainly looking for feedback from people building LLM/RAG apps:

Are these the right failure categories?
What signals would you want to see?
Do you debug this another way?
Should tools like this be local-first by default?

Link: https://debugai-5lb2.onrender.com

DEV Community: rishabh jain

The hardest LLM bugs are contract failures, not hallucinations

How I debug RAG failures with deterministic signals