A model asked to write promotional copy will eventually produce a figure nobody gave it. Not always, and not obviously. It rounds something it saw upstream, or it fills a slot because the sentence shape wanted a number there, and the result sits in the paragraph looking exactly as credible as the true figures around it. Fabricated numbers are the one class of error that gets harder to spot as the prose gets better.
"Do not invent numbers" in the system prompt is a preference. A function that runs after generation and refuses to return the draft is a guard. This post is about the guard: how to key numeric tokens so the comparison actually means something, where the normalization breaks, and what the check still cannot see.
The check is a set difference, and the hard part is the key
The shape is simple. You have a fact sheet for the thing being written about. You extract every number from the fact sheet, extract every number from the draft, and fail on anything in the second set that is not in the first.
That only works if both sides are reduced to the same canonical form. 1,000 and 1000 are the same claim. 9 and nine are the same claim. $90 and 90% are emphatically not, even though the digits match. So the key has to carry the unit, and the same normalizer has to run over both the fact sheet and the draft.
import re
from decimal import Decimal, InvalidOperation
WORD_NUMBERS = {w: str(i) for i, w in enumerate(
"zero one two three four five six seven eight nine ten eleven twelve".split()
)}
TOKEN = re.compile(
r"(?P<currency>[$€£])?"
r"(?P<value>\d[\d,]*(?:\.\d+)?)"
r"\s?(?P<unit>%|percent\b)?",
re.IGNORECASE,
)
def canon(currency, value, unit):
try:
number = Decimal(value.replace(",", "")).normalize()
except InvalidOperation:
return None
if unit:
return f"{number}%"
return f"{currency or ''}{number}"
def numbers_in(text):
words = (WORD_NUMBERS.get(w.lower().strip(".,;:"), w) for w in text.split())
found = {}
for match in TOKEN.finditer(" ".join(words)):
key = canon(match.group("currency"), match.group("value"), match.group("unit"))
if key is not None:
found.setdefault(key, match.group().strip())
return found
def unsupported_numbers(draft, fact_sheet):
allowed = set(numbers_in(fact_sheet))
return sorted(raw for key, raw in numbers_in(draft).items() if key not in allowed)
The word-number pass runs before the regex, so spelled-out figures get the same treatment as digits. Running it as a token-level substitution rather than a regex on the raw string keeps it from mangling words that contain a number word.
Here it is against a fact sheet and a draft that mostly behaves:
FACTS = """
9 channel definitions, each with its own voice rules and link discipline.
A (product, channel) pair may auto-publish only after 10+ reviewed drafts at 90%+ approval, only on free-API channels, only with the master switch on and credentials set.
MIT licensed.
"""
DRAFT = (
"It ships nine channel definitions, holds a 90% approval bar "
"before anything posts itself, and returns $90 of value a month."
)
print(unsupported_numbers(DRAFT, FACTS))
# ['$90']
nine passes because it normalizes to the 9 in the fact sheet. 90% passes on an exact key match. $90 fails, and it fails for the right reason: 90 is in the fact sheet, but as a percentage, and the currency-keyed form has no entry. This is the case a naive digit-scan waves through, and it is the most dangerous one, because a number lifted from a true fact into a false unit is the fabrication a human reviewer is least likely to catch.
Fail, do not fix
Once the check works you have a policy question: when a draft trips it, do you repair the draft or throw it away?
Repairing numbers is a trap. Deleting the offending token leaves a sentence making the same claim with a hole in it, and asking the model to fix the figure invites a second invented one. So the number check hard-fails. The draft does not get published, does not get partially edited, and does not get argued with in code.
A small number of violations are safe to fix mechanically, because the fix carries no semantics. Typographic normalization is the clear case: you can strip characters you never want in output and move on, with no retry and no model in the loop, because removing a dash changes no claim. Keep that list short and keep it strictly separate from the fail list. The rule I would hold to is that anything which could change the meaning of a sentence fails; anything that provably cannot gets sanitized.
Ordering matters too. Sanitize first, then validate, so the validator sees exactly the bytes that would ship.
Make the fact sheet the only source of both
The reason this holds up over time is not the regex. It is that the allowlist is derived from the same structured fact sheet used to build the prompt. One file per product, read by the generator and read by the validator. If you maintain the prompt facts in one place and the validator's permitted figures in another, they drift, and the drift shows up as false failures until someone loosens the check to make the noise stop.
Two more things worth wiring in while you are here, both plain engineering rather than anything clever. Log the reason for every rejection in a form you can read back, and feed recent rejection reasons for that product and channel into the next generation attempt, so the same mistake costs you less each time. And keep the validator flags visible on whatever surface you use to review drafts, so a human approving something can see what the check looked at.
What this does not catch
The honest limits, because they decide whether the check is worth building for your case.
It only sees numbers. A draft claiming an integration that does not exist, a customer you do not have, or a capability you never shipped passes cleanly. Those need their own guards, and some of them need a person.
It sees numbers, not relations. Every figure can be present in the fact sheet and the sentence can still be false: a true count attached to the wrong noun, a true percentage attributed to the wrong condition, a real figure placed in a comparison you never measured. The check confirms provenance of tokens, not truth of sentences.
Magnitude suffixes and ranges need explicit handling. 2k, 2 thousand, and 2,000 are one claim in three spellings, and a range like 10 to 20 is two tokens that may both be absent from the fact sheet even when the range is fine. The version above handles none of that. Extend the canonicalizer, do not extend the allowlist.
Word-number expansion buys false positives. "One of the channels" becomes a bare 1 in the scan. You can narrow it by only expanding word numbers directly preceding a noun, or you can accept the noise and let the fact sheet carry the small integers. Either choice has a cost, and the choice with no cost does not exist.
Carve-outs are where fabrications hide. Skipping code blocks, URLs, and version strings makes the check usable, and every skipped region is a place an invented figure can sit. If you exclude something, exclude it narrowly.
And if your copy carries no figures at all, this guard buys you very little. It earns its keep specifically when a generator writes about things with counts, prices, and percentages attached, and when the cost of one wrong figure under your name is higher than the cost of maintaining a fact sheet.
Top comments (0)