DEV Community

Gecko2087
Gecko2087

Posted on

Your LLM''s evidence is not evidence: verify every quote before you trust a score

Last week I shipped a small feature to a QA pipeline I maintain: before a score is stored, the model has to quote the transcript it claims to be scoring.

That sounds trivial. It is not, and the reason is worth a full article, because "make the model cite its evidence" is advice that feels sufficient and is not.

The failure mode nobody measures

The standard advice for LLM-as-judge systems is: require a literal quote alongside each score. The reasoning is sound — if every score comes with a quote, a human can check it in seconds instead of minutes.

But nobody checks whether the quote exists.

I reviewed the code of a pipeline I was about to adopt, and this is what the persistence layer looked like:

score = Score(
    value=float(item["value"]),
    confidence=float(item.get("confidence") or 0),
    evidence=(item.get("evidence") or "")[:4000],
    rationale=(item.get("rationale") or "")[:4000],
)
session.add(score)
Enter fullscreen mode Exit fullscreen mode

The prompt said, in bold, that every score must cite exact evidence. The code stored whatever string came back. If the model paraphrased, if it summarized, if it hallucinated a sentence that was never said, the pipeline would have stored it and displayed it with a confidence of 0.91 next to a number a supervisor was expected to trust.

The prompt is an instruction. The string is data. Instructions are not controls.

What it looks like in practice

Here is the kind of output you get from a good model asked for literal evidence:

{
  "criterion_key": "plazos",
  "value": 2.0,
  "confidence": 0.90,
  "evidence": "AGENTE: Y después te aviso.",
  "rationale": "The closing statement has no concrete deadline."
}
Enter fullscreen mode Exit fullscreen mode

And here is the same model, asked the same way, on a worse run:

{
  "criterion_key": "plazos",
  "value": 2.0,
  "confidence": 0.88,
  "evidence": "The agent did not provide a specific timeframe.",
  "rationale": "The closing promise was vague."
}
Enter fullscreen mode Exit fullscreen mode

Both look reasonable to a reader. The second one is worthless: it is a description of an absence, which is exactly what a model produces when it cannot find a quote. It reads like evidence but contains no words from the transcript.

A human reviewer skimming it might approve it. The whole point of the quote was to make review cheap. A paraphrase puts the review cost back where it started, while looking like it was solved.

The check is cheap

The verification does not need an embedding model, a second LLM call, or a vector store. It is a substring check against the transcript you already have.

The naive version breaks immediately: models quote with different capitalization, smart quotes, and they join two fragments with an ellipsis. So normalize first, then compare every fragment:

import re
import unicodedata

ELISION = re.compile(r"\s*\.\.\.\s*|\s*…\s*")


def _normalizar(texto: str) -> str:
    plano = unicodedata.normalize("NFKD", texto)
    plano = "".join(c for c in plano if not unicodedata.combining(c))
    plano = plano.lower().replace("“", '"').replace("”", '"')
    plano = plano.replace("‘", "'").replace("’", "'")
    return re.sub(r"\s+", " ", plano).strip()


def evidencia_sostenida(evidencia: str | None, transcripcion: str | None) -> bool:
    if not evidencia or not transcripcion:
        return False
    base = _normalizar(transcripcion)
    if not base:
        return False
    fragmentos = [f for f in ELISION.split(_normalizar(evidencia)) if f]
    if not fragmentos:
        return False
    return all(fragmento in base for fragmento in fragmentos)
Enter fullscreen mode Exit fullscreen mode

Note the last line: every fragment has to appear. A model that quotes one real sentence and appends a fabricated one fails the check, which is the correct outcome.

Stripping accents is deliberate. Transcript pipelines produce inconsistent diacritics — ASR output, diarization labels, and imported captions do not agree on spelling, and a strict comparison would flag correct quotes as false. If you strip accents for matching, you also have to strip them from the quote, which is what _normalizar does.

The behavior change matters more than the check

Now the interesting part. A check that only records a warning changes nothing. The check has to change what happens to the score.

Two decisions carried most of the weight:

1. Unsupported evidence drops the confidence to zero. Not to "low" — to zero. Confidence in that system means "how much does the evidence support this score". If the evidence does not exist, the evidence supports nothing, and the correct confidence is zero. This also makes the routing automatic, because a zero-confidence score is below any sane review threshold and goes to a human. No new code path, no new flag.

2. The evidence keeps the failure attached. When the check fails, the stored evidence is the model's string plus a marker:

if not evidencia_sostenida(evidencia, recording.transcript):
    confianza = 0.0
    evidencia = (evidencia + " " + NO_VERIFICADA).strip()
Enter fullscreen mode Exit fullscreen mode

Do not silently discard the model's answer. You want to see what it claimed, because the pattern of the failures is diagnostic: a model that paraphrases on one criterion and quotes correctly on another is telling you something about that criterion's wording.

The metric that makes it visible

A check that only logs is a check nobody reads. The pipeline now reports the rate of unverifiable citations:

def calidad_evidencia(session) -> dict:
    total = session.scalar(select(func.count(Score.id))) or 0
    if not total:
        return {"puntajes": 0, "no_verificadas": 0, "tasa_no_verificada": None}
    no_verificadas = session.scalar(
        select(func.count(Score.id)).where(Score.evidence.like(f"%{NO_VERIFICADA}%"))
    ) or 0
    return {
        "puntajes": int(total),
        "no_verificadas": int(no_verificadas),
        "tasa_no_verificada": round(no_verificadas / total, 3),
    }
Enter fullscreen mode Exit fullscreen mode

That single number tells you whether your evaluation system is auditable. If the rate is near zero, the model is quoting and the review layer is cheap. If it is high, the scores are prose, and the numbers in your dashboard are decoration.

In my experience this rate correlates with how vague the rubric is. A criterion that says "the agent explains the procedure properly" gives the model nothing to quote, so it describes. A criterion that says "the agent states the return window in days and the shipping cost" gives it a quote to find. Fixing the rubric fixes the evidence rate — which is the same lesson as always, arriving from a different direction.

What this does not solve

Two things, so you do not over-engineer around them:

  • A quote can be real and still irrelevant. The model may quote a sentence that exists but does not support the score it assigned. String matching catches fabrication, not misattribution. That needs human review, which is the part of the system you were never trying to remove.
  • Paraphrase-heavy transcripts break matching. If the "transcript" is a summary rather than a verbatim record, nothing will match. Verify at the boundary: assert that the transcript you store is the verbatim source, not a cleaned-up version.

If you build one thing from this

Add the check and the zero-confidence rule. It is about fifteen lines and one metric, and it converts an instruction into an enforcement.

The prompt can tell the model to quote. Only code can tell you whether it did.


This is part of the QA with AI series. The kit I use for this, with the judge prompt, the rubric guide and the deviation reports, is here: https://kit-qa-con-ia.surge.sh/ (MIT license, US$79). If you only want the idea, the check above is the whole thing.

One question for you: if you have an LLM scoring pipeline in production right now, what percentage of the stored evidence strings would survive a literal substring check against your transcripts? I would bet it is lower than you expect, and I would like to hear the real number.

Top comments (2)

Collapse
 
recallrundev profile image
RecallRun •

The all-fragments rule is the part most people skip, and zeroing confidence instead of adding a flag is a nice way to reuse the review routing you already have.

One edge case I'd put a test on: very short fragments. If the model "quotes" something like "sí" or "te aviso", the substring check passes in almost any long transcript, so the score is marked supported while the quote proves nothing. A minimum length per fragment (a few words) and checking that the match lands in the right speaker's turn close most of that.

RAG citations have the same gap. A check that only asks "was the cited chunk retrieved?" passes when the chunk was retrieved but the sentence it is attached to isn't supported by it. Matching the claim against the chunk text, like you do against the transcript, is the version that catches it.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

A string match on the quote is the cheapest check the model can't argue its way past. Run it before the score is stored, and keep the quotes that failed instead of dropping them.