DEV Community

Cover image for "No wait, I meant...": when my self-correction feature ate people's sentences
Mohsen Seyedkazemi Ardebili
Mohsen Seyedkazemi Ardebili

Posted on AI-assisted

"No wait, I meant...": when my self-correction feature ate people's sentences

When you talk, you correct yourself. "Send it on Tuesday, no wait, Wednesday."

A good dictation tool should type "Send it on Wednesday." So YazSes, the offline dictation app I work on, has a filter for that. It listens for six correction phrases:

no wait · delete that · scratch that · never mind · forget that · strike that

When it hears one, it throws away what came before it and keeps what came after. It's on by default.

Now try dictating this sentence:

"Delete that file when you are done."

The filter heard "delete that", rolled back, and typed:

"file when you are done."

Worse than a crash

That's not a garbled transcript you'd notice at a glance. It's fluent text you never said. If you're dictating an email quickly and not re-reading every line, it goes out like that.

And it wasn't rare. These phrases are ordinary English. A few real examples from the test suite, with what got typed:

You said It typed
the no wait policy applies to walk-ins policy applies to walk-ins
she said no wait for the second batch for the second batch
the delete that button is confusing button is confusing
they never mind the noise from the street the noise from the street

That last one isn't just shorter. The meaning is gone. Every one of the six triggers destroyed normal prose in some sentence shape, and since the filter is on by default, this was every user.

The fix: look at what's in front of the trigger

The words before a trigger tell you whether it's a correction or just a sentence. The fix came in three steps over a few releases, each one adding a kind of context that means "this is grammar, not a correction":

  1. Negations and verbs that govern the trigger ("do not delete that file", "you should never mind the warning"). Plus: a trigger at the very start of a sentence only counts if Whisper wrote pause punctuation after it.
  2. Articles, possessives, reporting verbs and copulas, which make the trigger a noun or a quote rather than an instruction ("the no wait policy", "your delete that habit", "she said no wait", "there is no wait time").
  3. Subject pronouns, which make the trigger the sentence's own verb ("they never mind the noise").

The words before the trigger decide: noun, quote, or real correction

The start-of-sentence rule is my favourite, because the comment explains the reasoning better than I can:

if text[:start].strip():
    return True
# At the very start there is nothing to roll back ...
# so the punctuation Whisper writes for the pause is the only signal, and the
# safe default is to require it. Without one the words are typed as spoken.
return text[end:end + 1] in tuple(".,;:!?…" + _CJK_CLAUSE_PUNCTUATION + _CJK_SENTENCE_END)
Enter fullscreen mode Exit fullscreen mode

"Scratch that, Wednesday" has a comma where you paused. "Scratch that itch" doesn't.

Measure it in both directions

A fix like this can easily overcorrect: make the filter so cautious it never fires, and you've "fixed" prose by breaking the feature. So I measured both:

  • Normal sentences destroyed: 15 of 15 → 2 of 15 (after step 2)
  • Real corrections missed: 0 of 10 → 0 of 10

Both corpora ship as tests. I also deliberately didn't treat "this" and "that" as signs of prose, because doing so broke a real correction already in the test set ("please do this strike that please do that").

What remains is a known gap: a plural noun as the subject, like "children never mind the cold at all", still loses its first half. Those cases are pinned in a test that fails if the list changes in either direction, so it's a visible limitation rather than a quiet one.

The test that had frozen the bug

Here's the part that surprised me.

YazSes has a cross-platform contract: a set of input/output vectors that every implementation (the Python desktop app and an Android port in Kotlin) must agree on. One of the vectors covered this exact case:

{
  "id": "self-correction-trigger-inside-quote",
  "input": "he said never mind the cost and left",
  "expected": "the cost and left"
}
Enter fullscreen mode Exit fullscreen mode

Its description said it documented "whatever the shipped behaviour is so both platforms agree."

So the bug wasn't just untested. It was the specification. A test someone wrote to pin current behaviour had pinned the defect, and any port that did the right thing would have failed the contract. The fix changed the expected value to the whole sentence.

A few releases later, those same vectors earned their keep: they caught the Kotlin port missing one of the new word lists. But only through a single word ("said") out of 33. One vector covering one word of a 33-word list is luck, not coverage. So now a dedicated test compares the Python and Kotlin word lists as sets.

What I took away

  • A text transform that can fail invisibly is the dangerous kind. Fluent wrong text is worse than obvious garbage.
  • Fuzz with realistic prose that contains your trigger words. "Never mind" and "delete that" are everywhere in normal English.
  • Measure a fix in both directions: what it stops breaking, and what it stops doing.
  • A "characterisation test" freezes behaviour, bugs included. If a test's description says "whatever it does today", read the expected value as a question, not an answer.
  • When two implementations share a list, compare the lists. Don't hope an example happens to hit the missing entry.

YazSes is offline, open source (Apache-2.0) dictation: github.com/MSKazemi/yazses. The filter lives in stt/filters/disfluency.py and the contract in contract/.

I wrote this with help from an AI assistant for drafting and editing; the bug, the fix and the lessons are mine, and I checked every fact against the code, tests and changelog.

Top comments (0)