DEV Community

Faizan Ahmad
Faizan Ahmad

Posted on

Three cheap checks that stop an LLM from inventing form data

If you use an LLM to turn messy input into structured fields (an email into a lead, a voice note into an intake form, a PDF into a row), it will eventually hand you a value nobody said. Not often, and usually with high confidence, which is the problem.

Prompting doesn't fix this. "Only use information from the input" helps, but a prompt is a request, not a check. What fixed it for us was three deterministic checks that run after the model answers, on every extraction. None of them uses a second model, and all three together cost a few microseconds.

The setup

Each field the model extracts comes back with three things, not one:

{
  "budget": {
    "value": "90000 to 100000",
    "confidence": 0.85,
    "source_snippet": "around ninety to a hundred"
  }
}
Enter fullscreen mode Exit fullscreen mode

source_snippet is the exact text in the input that the value came from. Asking for it costs a few tokens per field. It's what makes the checks possible.

Check 1: the snippet has to exist

If the model says a value came from some text, that text has to be in the input. Compare after lowercasing and collapsing whitespace, because models re-space things:

function normalize(s: string): string {
  return s.toLowerCase().replace(/\s+/g, " ").trim();
}

function snippetGrounded(snippet: string | null | undefined, input: string): boolean {
  if (!snippet) return false;
  const s = normalize(snippet);
  return s !== "" && normalize(input).includes(s);
}
Enter fullscreen mode Exit fullscreen mode

When this fails, we throw the value away: null, confidence 0. A missing snippet means the model can't point at where the value came from, so none of it can be trusted.

Note what this deliberately does not check: the value itself. "eight people" should become 8, and "12 LPA" should become 1200000. Requiring the value to appear verbatim would break every useful transformation. Only the evidence has to be verbatim.

That's also its blind spot, which is where check 2 comes in.

Check 2: every number has to come from somewhere

A respondent said their budget was "around ninety to a hundred". The model returned the snippet around ninety to a hundred (grounded, passes check 1) and the value 90000 to 95000. Nobody said thousands, and nobody said 95.

So the second check asks: can every number in the value be derived from the input? A number counts if it was written as digits, spelled as a word, or written as a stated number times a stated scale ("20k", "12 LPA", "100 thousand"). Simplified:

const WORDS: Record<string, number> = {
  one: 1, two: 2, three: 3, four: 4, five: 5, six: 6, seven: 7, eight: 8, nine: 9, ten: 10,
  twenty: 20, thirty: 30, forty: 40, fifty: 50, sixty: 60, seventy: 70, eighty: 80, ninety: 90,
  hundred: 100, thousand: 1000,
};
const SCALES: Record<string, number> = { k: 1e3, thousand: 1e3, lakh: 1e5, lpa: 1e5, m: 1e6, million: 1e6 };

function statedNumbers(input: string): Set<number> {
  const out = new Set<number>();
  for (const m of input.matchAll(/(\d[\d,]*(?:\.\d+)?)\s*(k|thousand|lakh|lpa|m|million)?\b/gi)) {
    const n = Number(m[1].replace(/,/g, ""));
    out.add(n);
    if (m[2]) out.add(n * SCALES[m[2].toLowerCase()]);
  }
  for (const word of input.toLowerCase().match(/[a-z]+/g) ?? []) {
    if (word in WORDS) out.add(WORDS[word]);
  }
  return out;
}

function numbersGrounded(value: unknown, input: string): boolean {
  const stated = statedNumbers(input);
  const numbers = String(value).match(/\d[\d,]*(?:\.\d+)?/g) ?? [];
  return numbers.every((n) => stated.has(Number(n.replace(/,/g, ""))));
}
Enter fullscreen mode Exit fullscreen mode

For "around ninety to a hundred", the stated set is {90, 100}. 90000 isn't in it, so the check fails.

The real version also handles compound words ("twenty-five"), "a hundred", month names and decimals, but the idea fits in 25 lines.

When this one fails, we do not null the value. It's usually partly right (the 90 was said), and deleting it hides the mistake from the person who could fix it. Instead we cap the confidence at 0.5, which sends it to a human.

Check 3: a hedged number is an uncertain number

The third failure was honest. The input said "around ninety to a hundred, depending", the model returned exactly that, and reported confidence 0.7. In our tests it did this in six runs out of six, because the prompt said to go "below 0.7" for uncertain answers and the model read that as "0.7".

Our review threshold is confidence < 0.7, so 0.7 sailed through as approved. The prompt asked for the right thing. Only a check could make sure it happened:

const HEDGE =
  /\b(around|about|approx(?:imately)?|roughly|maybe|probably|-?ish|or so|ballpark|depending|not sure|i think)\b/i;

function hedged(snippet: string, value: unknown, input: string): boolean {
  if (!/\d/.test(String(value))) return false; // "probably remote" is a choice, not a measurement
  if (HEDGE.test(snippet)) return true;
  // The model sometimes leaves the hedge just outside its snippet: "around [90 to 100]".
  const at = normalize(input).indexOf(normalize(snippet));
  return at > 0 && HEDGE.test(normalize(input).slice(Math.max(0, at - 24), at));
}
Enter fullscreen mode Exit fullscreen mode

Two details came from real misses:

  • It only applies to numeric values. Flagging every "I think" would turn the review queue into noise, and people learn to ignore noise.
  • It looks a few words before the snippet. When someone dictated without punctuation, the model snipped 90 to 100 and left "around" outside it. Three words of lookbehind catches the hedge on this number without reaching a hedge on something else in the sentence.

A failure caps confidence at 0.55.

What each check does when it fires

Check On failure Why
Snippet not in input value removed, confidence 0 Nothing supports the value
Number never stated value kept, confidence capped at 0.5 Usually partly right, so a human should look
Number hedged value kept, confidence capped at 0.55 Honest, but uncertain

Two smaller rules sit underneath all three:

  • Placeholder answers ("n/a", "unknown", "not provided") are treated as empty. Models love to fill a required field with a polite non-answer.
  • The model's self-reported confidence is capped at 0.98, so 1.0 stays reserved for values a person typed into a structured field.

Anything under the threshold goes to a review queue instead of auto-approving. A false positive costs one person one look. A false negative costs a wrong number in someone's CRM. For a check that runs on every extraction, that's the right trade.

The general rule

Give the model a way to show its work (source_snippet), then check the work with code. Checks that are cheap, deterministic and dumb catch the failures that a smarter prompt keeps letting through, and they fail the same way every time, so you can test them.


These checks run on every extraction in DodoForm, a form builder where people can answer by typing, talking or pasting something messy. You can try the extractor without an account on the homepage.

Top comments (0)