orca_forge

Posted on Sep 5 Originally published at forge.workstyle.tech

Chasing Unmeasured Metrics with Thresholds Always Leads to Failure — A Tale of 10 Mistakes in One Day

#debugging #softwaredevelopment #softwareengineering

📝 Originally published (in Japanese) at forge.workstyle.tech.

In a single day, I encountered the same type of failure ten times. I made four of these mistakes myself, while my colleague, working in parallel, made seven (with one overlapping case). The issues spanned various domains, including speech synthesis, search, audit logs, text preprocessing, and frame processing, with no overlap. Yet, the underlying structure of the failures was identical.

The metric returned "0 items" or "no match," which I interpreted as "the target does not exist."
I continued to adjust thresholds and narrow down lists in response.

The target was outside the observable system from the beginning. No matter how I adjusted the thresholds, it was impossible to reach.

The First Six Cases

① Additional sounds at the end of speech model outputs

I assumed this was due to hallucinations in the training corpus and tightened the script alignment insertion tolerance from 3→1→0. After three re-trainings, the additional sounds went from 3/6 → 2/6 → 3/6, which was within the margin of error.

Scanning 198 audio clips, I found that the four clips with additional sounds all had insertion 0. Since the speech-to-text (STT) system didn’t recognize these sounds as text, they weren’t included in the threshold denominator.

② Dialogue act detection never triggered in real conversations

I used a vocabulary list to determine if a statement was a question. Since it never triggered, I added more words, narrowed the list, and adjusted conditions. In reality, it failed to detect compound utterances (e.g., "What is this, and also the previous one") and only passed on simple test sentences.

③ "Evidence present" metric always returned True

I measured whether answers included evidence as a binary value. Since the results were too good, I tightened the threshold. In reality, when automatic page navigation was enabled, something was always attached as evidence, so the metric itself carried no meaningful information.

④ Audit log showed 0 navigate events

When a user reported "the page transitioned automatically," I checked the audit table and found 0 entries, concluding it hadn’t occurred. The implementation didn’t log audit events at issuance, so it couldn’t have been recorded. I mistook the absence of records as evidence that the event didn’t happen. I misdiagnosed the first report and only questioned the logging system after the second report.

⑤ Unable to answer based on page content

I added four guards: search thresholds, exclusion conditions, and handover judgments. When a user suggested, "Shouldn’t you grasp the entire page text first?" I realized I was applying vector search to a 231-character page. The text was too short for meaningful segmentation, and preprocessing was dropping characters, causing searches to fail.

⑥ Pronunciation breakdown

The phrase "少々お待ちください" (Please wait a moment) was heard as "しょもおまちください." I checked the model, synthesis parameters, cache, and routing in sequence. Finally, I printed the preprocessing output and found this:

_clean_tts_text('少々お待ちください。')  →  '少お待ちください。'

The allowed character whitelist included kanji in the range 一-鿿, but 「々」 (CJK Symbols block) was excluded. Synthesizing "少お待ちください" resulted in "ショーをお待ちください" (Show o omachi kudasai).

Four More Cases (Later That Day)

While writing this article, four more cases arrived from my colleague, all following the same pattern:

Misidentified frame type, causing the gate to never trigger (discovered via 0 real-world logs)
Fake VAD passed, but the real one lacked attributes and did nothing
Instructions placed at the end of a large prompt were ignored four times (the same issue was documented in the same file)
elapsed_ms was misinterpreted, leading to unnecessary fixes

The colleague summarized it as:

The common thread is "tests pass, but real-world logs reveal the issue."

The first case was particularly harsh. I built the gate, wrote tests, passed 8/8, deployed, and it never triggered. Tests are written based on my assumptions about the frame, so if the actual frame type differs, both the test and the implementation are based on the same misunderstanding.

The Common Pattern

There are three layers to these failures:

Layer 1: Interpreting "0 items" as evidence

In ①, "0 STT detections meant no additional sounds," and in ④, "0 audit logs meant no transitions." Both assumed a functioning observation system, even though a broken system would also return 0.

"0 items" is not evidence of absence. It could mean either "the observation system is working, and the target is absent" or "the observation system is broken." The value 0 doesn’t distinguish between these two scenarios.

Layer 2: Continuously adding guards to symptoms

In ②, I added vocabulary; in ③, I tightened thresholds; in ⑤, I added four guards. All adjustments were made near the symptoms.

This seems logical—symptoms are likely close to the cause. But when dealing with unobservable targets, tightening around symptoms is ineffective. In ①, adjusting thresholds three times had no meaningful impact.

Layer 3: Not questioning my own observation systems

This is the deepest layer.

In ①, I wrote the quality gate, so I assumed "if it passed, the material was normal."
In ④, the audit logs were implemented by the other party, so I assumed "if they implemented it, it should be visible."
In ③ and ⑤, the creators didn’t question their own metrics or searches.

We’re less likely to doubt systems we’ve built ourselves, especially if we remember them working. And without doubt, we don’t look beyond them.

The colleague who encountered ②–④ also failed to notice that the pipeline failure visualization panel they built was 404ing the next day. They built it, saw it working, and moved on.

The Only Effective Solution

The cases I resolved myself shared a common approach: I changed the axis of observation and measured the target again.

Case	Original Axis	New Axis	Result
① Additional sounds	STT transcription	Waveform envelope	Discovered 4 additional sounds in insertion 0 clips
⑤ Page content	Search score	Character count	Found 231 characters
⑥ Pronunciation	Listening to synthesized audio	Printing preprocessing output	Found "々" was removed

Each took less than 5 minutes. Scanning waveforms in ① took 30 minutes, still less than a tenth of the time spent adjusting thresholds and retraining.

Self-validation of metrics is inherently difficult. You can’t verify a broken metric using the same metric. Instead of asking, "Is this metric correct?" it’s faster to apply a different measure once.

User Reports as Independent Observations

Another effective approach across all cases:

When metrics and user reports conflict, doubt the metrics first.

In ①, even after a user reported hearing "こちらですって" (It’s over here), I prioritized my measurement: "STT detected 0/6, so it wasn’t passed." Another observation system (human ears) had already provided results.

Similarly, in ④, I dismissed the first user report based on 0 audit logs.

User reports are often treated as weaker evidence than machine metrics because they’re subjective and ambiguous. But as independent observation systems, their value is high when conflicts arise. If two independent measurements of the same target disagree, and one says "anomaly," doubt your own instruments first.

Practical Takeaways

Here’s the distilled procedure—nothing special:

Before adjusting thresholds, ask once if the target is observable with that threshold. For ①, ask, "Can additional sounds exist in insertion 0 clips?" This question leads to checking waveforms.

When seeing "0 items," verify the logging system separately. For ④, intentionally trigger an audit event to confirm it’s recorded. Takes 5 minutes.

Before adding a second guard to symptoms, look upstream once. For ⑤, count page characters. If the first guard didn’t fix it, it likely wasn’t capturing the target.

After building a detector, test it with non-target cases first. The waveform detector in ① initially flagged all 12 models as "anomalous," mistaking pauses for additional sounds. If detection rates are too high, doubt the detector, not the target.

For systems you’ve built, verify them with a different axis at least once. This is the most effective step. Doing it immediately after building would have saved the three retraining attempts in ①.

Naming the Issues

Finally, the most frustrating realization:

Almost all were known, standard problems.

Additional sounds in speech are known as trailing artifacts / babbling. The phenomenon of saying "I’ll move" but not moving is called procedural hallucination, with dedicated benchmarks. The need for a 500–800ms microphone gate after speech ends (not 300ms) to prevent self-echo is well-documented.

We treated all issues as environment-specific and dug internally. If we’d first checked if the symptoms had names, we could have quickly found known properties like "STT can’t detect them."

I wrote "10 failures in one day," but accurately, all 10 had precedents outside our environment.

Series: Mass-Producing Practical Voices from Diffusion TTS

This series documents designing voices from single-line captions, manufacturing training corpora, and mass-producing role-specific practical voices. This article is Part 5: Summary.

← Previous: Deploying Overwrote Each Other’s Work

(End of series)

All 18 Parts

The insights are compiled in the notes on Mass-Producing Practical Voices from Diffusion TTS Manufacturing Pipeline.

DEV Community