The transcript was perfect and the agent still answered the wrong question

The word error rate was near zero, and the agent kept confidently answering something the caller never asked. The bug was hiding in the punctuation nobody was looking at.

The escalated call was a billing question. The caller said, and I am quoting the transcript exactly, "so my card was charged twice can you refund the second one." Every word correct. The speech-to-text got all of it. And the agent replied by cheerfully confirming a charge, as if the caller had made a statement of fact and asked for nothing.

I stared at that transcript for a while because on the surface there was nothing wrong with it. The words were right. The caller was clearly asking a question. The agent still whiffed.

Then I looked at what the intent step actually received, and the problem was sitting there in plain sight, which is to say it was invisible. The transcript had no punctuation. No question mark. No period. No sentence boundaries at all. Just a flat run of correct words. The ASR was tuned to minimize word error rate, and it did that beautifully, but it did not restore punctuation or casing, and my downstream logic had been quietly assuming clean, punctuated English this whole time.

Word error rate is not the metric that decides whether you understood

Here is the part that took me a day to accept. Our word error rate on this class of call was near zero. By the number I had been reporting to everyone, ASR was solved. And the agent was still answering the wrong question, because word error rate measures whether you got the words right, not whether the words arrived in a shape the next stage could parse.

Two different failure modes were hiding under that clean number.

A question read as a statement. With no question mark, the intent classifier saw "you charged me twice" as a declaration and routed it to an acknowledgement flow instead of a refund flow. The words were identical to the caller's. The grammar of intent was gone.

One utterance split into two. Without sentence boundaries, a single request like "cancel my appointment and rebook it for Thursday" would sometimes get chunked into two intents, "cancel my appointment" and "rebook it for Thursday," and the agent would execute the cancel, lose the second half, and hang up satisfied.

I could reproduce both on demand. Here is the minimal version of the first one, the intent flip, using a tiny illustrative classifier so the mechanism is visible.

def classify(text: str) -> str:
    """Toy intent router. Real systems use a model, but they inherit the
    same fragility: the decision leans on punctuation and casing that
    raw ASR does not provide."""
    stripped = text.strip()
    is_question = stripped.endswith("?") or stripped.lower().startswith(
        ("can ", "could ", "would ", "will ", "do ", "does ", "is ", "are ")
    )
    if "charged twice" in stripped.lower() or "charged me twice" in stripped.lower():
        return "refund_request" if is_question else "acknowledge_charge"
    return "fallback"

clean = "My card was charged twice, can you refund the second one?"
raw   = "my card was charged twice can you refund the second one"

print(classify(clean))   # refund_request   (correct)
print(classify(raw))     # acknowledge_charge   (WRONG, same words)

Same words. Opposite outcome. The only difference is the punctuation and casing that the ASR threw away and my code assumed would be there.

Stop trusting raw ASR text as if it were clean input

The fix has two parts, and I want to be honest that the first part is a patch and the second part is the actual lesson.

The patch: restore punctuation and casing before the intent step ever sees the text. There are small, fast models that do exactly this, and I ran one as a stage between ASR and NLU. A restoration step turns "my card was charged twice can you refund the second one" back into "My card was charged twice. Can you refund the second one?" and the intent router recovers.

def restore(raw_text: str) -> str:
    """Stand-in for a punctuation/casing restoration model
    (e.g. a small seq2seq or token-classification model run inline).
    Shown as a rule here only to make the pipeline stage explicit."""
    # A real model predicts boundaries and casing from token context.
    restored = punctuation_model.predict(raw_text)   # returns cased, punctuated text
    return restored

routed = classify(restore(raw))
print(routed)   # refund_request   (recovered)

The lesson underneath the patch: do not let the boundary of an utterance be decided by punctuation that may not exist. The ASR already knows where the caller paused. It emits word-level timing and, for the final result, an endpointing signal that says "the caller stopped talking here." That signal is far more reliable than a guessed period. So I stopped inferring sentence boundaries from text and started segmenting on the ASR's own timing and endpointing, then fed the LLM both the raw words and the timing, and let it reason over the actual acoustics of the turn rather than a hallucinated grammar.

def segment_by_endpointing(words, gap_threshold_ms=700):
    """Group ASR word-timings into utterances using pauses, not punctuation.

    words: list of {"word": str, "start_ms": int, "end_ms": int}
    A gap longer than gap_threshold_ms starts a new segment.
    """
    segments, current = [], []
    for i, w in enumerate(words):
        if i > 0:
            gap = w["start_ms"] - words[i - 1]["end_ms"]
            if gap >= gap_threshold_ms:
                segments.append(current)
                current = []
        current.append(w["word"])
    if current:
        segments.append(current)
    return [" ".join(seg) for seg in segments]

With that, "cancel my appointment and rebook it for Thursday" stays one segment, because there was no 700 ms pause in the middle of it, and the agent handles the whole request instead of half of it.

Evaluate on the transcripts you actually get

The reason this shipped broken is the reason a lot of voice bugs ship broken. Every test transcript in my intent suite was hand-typed, and I typed like a literate human. Perfect punctuation. Proper casing. Clean sentence boundaries. My evaluation set was a fantasy version of the input the model would never see in production.

I rebuilt the intent evaluation set from real ASR output: lowercase, unpunctuated, occasionally chunked at the wrong pause. Intent accuracy on that realistic set was about 14 points lower than on my clean set on the first run, which was a miserable number to look at and the single most useful number I got that month. It was finally measuring the thing the caller experiences. I tuned the restoration and endpointing against that set, not the pretty one.

What shipped, and what I'd tell past me

What went to production: a punctuation and casing restoration stage between ASR and intent, utterance segmentation driven by word-timing and endpointing instead of guessed punctuation, the raw transcript plus timing handed to the LLM rather than a cleaned-up string with invented sentence boundaries, and an intent evaluation set rebuilt from real un-punctuated ASR output. The wrong-question failures on billing calls dropped to near zero, and the split-utterance hang-ups went away entirely.

If I could send one note back to the version of me who built the first NLU stage: a clean word error rate is a trap, because it tells you the words are right and lets you believe the meaning is too. Meaning lives in the boundaries and the punctuation and the casing, and cheap ASR gives you none of that. A word-perfect transcript still is not something a downstream model should reason over directly. It is raw material, and it needs a stage of restoration and segmentation first.

The second note is about the evaluation set. Whatever you feed it becomes your assumption about what the real input looks like, and if you type that set by hand you are quietly assuming clean punctuation the microphone will never deliver. So I rebuilt mine from real ASR output, lowercase and unpunctuated and occasionally chunked wrong, and tested against that. The ugly transcripts are the ones your callers actually produce.