My library told every Hindi user their model was broken

I maintain llmclean, a small zero-dependency library for cleaning up raw LLM output. In 0.4.0 I added a function that tells you whether a model's output got cut off mid-thought:

def looks_truncated(text):
    return text.rstrip()[-1] not in ".!?…\"')]}"

If the last character isn't a sentence terminator, the output probably got chopped by a token limit. Simple, obvious, worked great in my tests.

Then I generated 300 outputs from five local models and ran the detector across all of them. Two rows in the results:

multilingual_hi    11/11 flagged truncated  (100%)
multilingual_zh    12/12 flagged truncated  (100%)

Every single Hindi and Chinese generation. Not one of them was actually truncated.

Look at the terminator list again

".!?…\"')]}". That's Latin punctuation. All of it.

Hindi sentences don't end in a full stop. They end with a danda — ।, U+0964. Chinese ends with the ideographic full stop 。, U+3002. Neither is in that string. Neither is in any string I would have written, because I wrote it while thinking in English.

So my library looked at वर्षा तब होती है। — a complete, correctly punctuated Hindi sentence — and reported it as cut off. Then it did the same to every Chinese sentence. If you were running llmclean over Indic-language output, it was telling you your model was broken, constantly, and it was wrong every time.

I'm Indian. I build things that process Hindi. I still shipped a Latin-only terminator set without noticing.

The part that actually bothers me

I have 194 tests. None of them caught this, and adding more of the same kind wouldn't have.

Think about how I'd have written that test:

def test_truncation():
    assert looks_truncated("The encoder maps input") is True
    assert looks_truncated("The encoder maps input.") is False

I'd have written it in English, because I wrote the function in English. The test inherits the blind spot from the code, because the same person wrote both, in the same sitting, holding the same assumption. A test can only check the cases you thought of. This was a case I didn't think of — which is exactly the category unit tests are worst at.

The fix took two minutes:

_TERMINAL_PUNCT = (
    ".!?…\"')]}"      # latin
    "।॥"              # devanagari danda, double danda
    "。！？．"         # cjk
    "」』）】〉》"     # cjk closing quotes and brackets
    "۔؟"              # urdu, arabic
    "។" "።፧"          # khmer, ethiopic
)

Finding it took building something else entirely.

What found it

Unit tests check the cases you imagined. To find the ones you didn't, you need input you didn't write. So I built a harness that generates a production-shaped corpus:

Five models — Llama 3.1, Gemma 4, Qwen 2.5, DeepSeek-R1, Mistral.

Fifteen task types — flat JSON, nested JSON, JSON from a long context, chat answers, summaries, markdown docs, code, classification, Hindi, Chinese, Hinglish code-switching, tables. The multilingual tasks are the ones that mattered, and I only included them because llmclean's users skew Indian.

Four decode conditions, including a deliberate 24-token cap so that real truncation actually happens and I can tell true positives from false ones.

Then — and this is the part I think people skip — I replayed every generation through eight transport mutations. Because production traffic isn't model output. It's model output after it's been through a network, a gateway, and somebody's encoding bug:

• CRLF line endings from Windows clients (llmclean had a real bug from this in 0.2.0)
• A BOM prepended by a gateway
• The stream cut off mid-word because a connection dropped
• HTML-escaped & and <
• UTF-8 bytes decoded as latin-1 somewhere, producing mojibake
• Duplicated chunk boundaries from streaming reassembly

300 generations × 8 mutations × 14 functions = 30,128 calls. Zero crashes. And one very clear bug that no amount of me writing tests in English was going to surface.

After the fix: false truncation flags dropped from 66% to 27% of normal generations, while the deliberately-capped ones stayed at 85%. I checked the remaining 27% by hand — they're real, output that genuinely ends mid-word or mid-code-fence. Chinese went from 12/12 to 0/12.

The other thing I found in my own code

While I was at it, someone asked me a fair question: does the library log anything when it fails?

llmclean has a hard rule — no public function ever raises. If something goes wrong, you get your input back unchanged. That's thirteen except Exception blocks. And every one of them looked like this:

except Exception:
    return original

No log. No warning. Nothing. I injected a bug into strip_markdown and ran it with logging.basicConfig(level=DEBUG):

returned: '# Title'

That's it. The function silently did nothing and told nobody.

This is funny in a bad way, because degeneracy.py — which I wrote the same week — has this in its docstring: "a cleaner that silently tidies the overflow hides model damage." I wrote that warning about model output and then committed exactly the same sin against my own errors.

Now every fallback logs to logging.getLogger("llmclean") with a NullHandler, so it's silent unless you ask. Same injected bug:

WARNING llmclean: llmclean.strip_markdown returned its input unchanged
  after AttributeError: 'NoneType' object has no attribute 'sub'
Traceback (most recent call last): ...

Still returns '# Title'. Contract intact. But now you can see it.

Writing the test for that turned up one more hole: if the log handler raised, the exception escaped and broke the never-raise guarantee. So the log helpers swallow their own errors. Defensive code for the defensive code.

The rest of 0.4.0

Degeneration detection. Repetition happens at three levels, and my repetition trimmer only handled one:

level	example	before
phrase	"So this is 8 infinity. So this is 8 infinity."	handled
token	"parameter parameter parameter"	missed
subword	"thresholdinginginging"	missed

That last one is nasty. It scores a perfect 1.0 on distinct-word ratio and 0.0 on adjacent duplicates, because it's one unique word. Every word-level metric says it's fine. Only a check inside the word finds it.

degeneracy_score() reports all of it and deliberately doesn't fix anything. The philosophy is report → retry → repair, in that order: if your model is degenerating, a retry usually costs less than the quality loss of repaired text, and quietly cleaning it up means you ship a broken model with a bandage on it.

Three front doors. Someone pointed out that llmclean had fifteen functions and no obvious place to start — pd.read_csv() is one call, why isn't this? Fair. The commonest task used to be four lines and a try/except:

result = enforce_json(strip_fences(raw))
try:
    data = json.loads(result)
except json.JSONDecodeError:
    data = None

Now:

data = llmclean.load_json(raw)

Plus clean_text(raw) for prose and check(raw) for the degeneration report. The other twelve functions are still there when you want to drive the pipeline yourself.

What I'd tell you to take from this

If you build anything that processes text, go find the assumption you encoded without noticing. Mine was that sentences end in a full stop. Yours is something else.

The general version: your tests are written by the same brain that wrote the bug, so they inherit its blind spots. The only reliable way out is input you didn't author — real traffic if you have it, generated traffic if you don't. An afternoon of building a generator found something 194 hand-written tests structurally could not.

llmclean 0.4.0 is on PyPI and GitHub. Zero dependencies, still pure standard library.