Agent Lab Vol 3 #03 is an offline voice journal: speak, and find it again, on your own machine with no key and no network.
The audio is invented and the recogniser is simulated. There is no public corpus of anybody's spoken diary and there is no acoustic model in this repository, so all 2,782 words across 39 entries in 24 sessions were written by hand, and asr.py applies a declared error model at declared rates. One module is fake, it is named, and everything downstream runs unchanged on a real whisper.cpp transcript.
What is measured is what those rates do.
Page: https://dev48.infy.uk/agentlab/vol3-03-voice-journal.html
The headline
| recogniser | word error rate | retrieval | recall |
|---|---|---|---|
| A | better | 0.1096 | 0.8097 |
| B | 36% worse | 0.1488 | 0.8625 |
The worse recogniser retrieves better. A word error rate is an average over words; a query is not. Restrict the same alignment to the 174 words a person would actually search for and it reads 0.4483 against 0.8391 — and the crossing goes both ways once you split by query.
Three of the six self-checks cannot fail
Every one of 216 configurations partitions the tokens. So the word count comes back identical for:
- the segmenter that never cuts — which files 349 words under the wrong calendar day
- the one that gets it right
A conservation check over a partition is satisfied by construction. It is three of the six checks this pipeline ships.
The dictionary check is compatible with both answers
| run | dictionary check | key-term error rate |
|---|---|---|
| control | 24 of 24 | 1.0000 |
| one lexicon later | 24 of 24 | 0.6724 |
Same green check, wildly different quality. And snapping to the lexicon erases the 165 unknown tokens that were the only honest signal in the file — the words the recogniser did not know were exactly the evidence that it was struggling.
There is no silence threshold
2,743 thinking pauses against 15 real boundaries. They overlap. A 3.0 s threshold costs 304 false cuts. There is no value that separates them, because the distributions are not separable — which is a finding, not a tuning failure.
Python standard library only. No API key, no network, MIT.
6,170 verifier asserts, 119 in-page checks, 0 failures.
Top comments (0)