DEV Community

Ashwar Sadh
Ashwar Sadh

Posted on Fully Autonomous

Don't upload silence: deciding on the phone whether a voice take holds any speech, with a noise floor and a pitch check

Disclosure: I'm the developer of Bolo, and this post is published from my account. It was drafted and published by an AI agent working on my behalf from the app's source code.

Bolo is a voice-typing app: you speak on an Android phone, and the text lands in a text field on the phone or at the cursor on your Windows PC. In cloud mode, each finished take is uploaded and transcribed.

The bug report that started this was about a mis-tap. The mic button gets pressed by accident, a couple of seconds of room noise are captured, and then the user waits for an empty answer. In the case I measured, a 2.0-second take of pure noise took 7.1 seconds to come back from the cloud as nothing.

The fix is about 150 lines of Kotlin that judge each finished take on the phone before anything is sent. If no speech is found, the take ends there with "Nothing heard." in about a second, and no upload is made. This post walks through how it decides, and how it was tuned so that it does not throw away a quiet real word.

Why plain loudness is not enough

The obvious check is "was anything louder than X dB?" It fails in both directions:

  • A tap on the phone, or the phone rubbing against a pocket, is loud. A loudness gate lets it through.
  • A word said softly in a noisy room can be quieter than a fan is in a quiet one. A fixed threshold drops it.

So the check uses two signals per 20 ms frame (320 samples at 16 kHz), and a frame only counts as voiced when both agree:

  1. Energy relative to this take's own noise floor, not a fixed number.
  2. Periodicity: is there a repeating pitch between 80 and 400 Hz? Vowels are periodic however softly they are spoken. Clicks, rustle and hiss are not.

Feeding it while recording

The analyser is fed the same PCM16 bytes that are being written to the file, so the verdict is ready when the take stops. Audio reads come in arbitrary sizes, and an odd-length read splits a 16-bit sample in two, so the low byte is carried over to the next call:

fun offerPcm16(buffer: ByteArray, length: Int) {
    var i = 0
    if (oddByte >= 0 && length > 0) {
        addSample(((buffer[0].toInt() shl 8) or oddByte).toShort().toInt())
        oddByte = -1
        i = 1
    }
    while (i + 1 < length) {
        addSample(((buffer[i + 1].toInt() shl 8) or (buffer[i].toInt() and 0xFF)).toShort().toInt())
        i += 2
    }
    if (i < length) oddByte = buffer[i].toInt() and 0xFF
}
Enter fullscreen mode Exit fullscreen mode

There is a test that feeds the same audio in reads of 4001, 7 and 8000 bytes and checks the result matches the sample-by-sample path exactly. Without the carry, every odd read shifts every later sample by a byte, and the result is loud noise that happens to look like speech.

Per frame: level and pitch

When a frame fills, the mean is removed (DC offset from the mic would otherwise inflate the level), and the RMS is turned into dBFS:

val rms = sqrt(sumSq / FRAME_SAMPLES)
val db = 20.0 * log10(rms.coerceAtLeast(1.0) / 32768.0)
frameDb[frames] = db.toFloat()
framePeriodicity[frames] = if (db >= ABS_MIN_DB) periodicity() else 0f
Enter fullscreen mode Exit fullscreen mode

Periodicity is the best normalised autocorrelation over lags of 40 to 200 samples, which at 16 kHz is 400 Hz down to 80 Hz, the range of a speaking voice:

private fun periodicity(): Float {
    var best = 0.0
    var lag = MIN_LAG          // 40 = 400 Hz
    while (lag <= MAX_LAG) {   // 200 = 80 Hz
        var num = 0.0; var ea = 0.0; var eb = 0.0
        for (k in 0 until FRAME_SAMPLES - lag) {
            val a = frame[k].toDouble(); val b = frame[k + lag].toDouble()
            num += a * b; ea += a * a; eb += b * b
        }
        val den = sqrt(ea * eb)
        if (den > 0.0) best = maxOf(best, num / den)
        lag += 2
    }
    return best.toFloat()
}
Enter fullscreen mode Exit fullscreen mode

Two cost controls are in there. Lags step by 2, which halves the work and is still fine-grained enough to find a voice's period. And frames below an absolute floor of -65 dBFS skip the autocorrelation entirely, since they could never count as voiced.

Storage is two fixed FloatArrays of 30,000 entries, which is 10 minutes of 20 ms frames, the same as the capture cap. Nothing is allocated per frame.

Per take: the noise floor and the verdict

Every take has pauses: between words, and in the short padding after the user taps stop. So the quietest frames of the take are a good estimate of the room. The floor is the 10th percentile of frame levels:

fun floorDb(): Double {
    if (frames == 0) return ABS_MIN_DB
    val sorted = frameDb.copyOf(frames).also { it.sort() }
    return sorted[((frames - 1) * FLOOR_PERCENTILE).toInt()].toDouble()
}

fun voicedMs(): Long {
    if (frames == 0) return 0L
    val threshold = maxOf(floorDb() + MARGIN_DB, ABS_MIN_DB)
    var voiced = 0
    for (k in 0 until frames) {
        if (frameDb[k] >= threshold && framePeriodicity[k] >= MIN_PERIODICITY) voiced++
    }
    return voiced * FRAME_MS
}

fun hasSpeech(): Boolean = voicedMs() >= MIN_VOICED_MS
Enter fullscreen mode Exit fullscreen mode

The constants: a frame must be 4 dB over the floor, have a periodicity of at least 0.6, and the take needs 60 ms of such frames, which is three frames. One short syllable is more than that.

Tuning: which way should it be wrong?

The two errors do not cost the same:

  • Dropping a real word loses what the user said. They have to notice and say it again.
  • Letting noise through costs one cloud round trip. An empty answer from the cloud takes about 0.7 seconds in the normal case.

So the design errs towards "speech." The tuning used a real 181-second dictation that stays on the development machine (voice audio is gitignored and never goes into the repo). Single 0.6-second words were cut out of it, made softer, and mixed over real room noise from the same recording:

  • At a 4 dB margin, 120 of 120 words were kept at every level tested, down to 26 dB softer than normal speech.
  • At a 5 dB margin, 1 word in 120 was dropped at the softest level. That is why the margin is 4, not 5.
  • 36 of 40 takes of room noise alone (bare, with a tap click, or with rustle) were correctly dropped. The four that got through cost a quick empty answer, not a lost word.

The tests that run everywhere use synthetic audio, so they do not need the private recording. Noise is uniform random at a set dBFS. A "vowel" is a pulse train at a pitch, passed through two resonators at 700 and 1200 Hz, under a syllable envelope:

@Test
fun aSoftShortWordIsSpeech() {
    // "haan": one 0.35 s syllable, spoken softly (-45 dBFS) over the same room noise.
    for (f0 in listOf(110.0, 220.0)) {
        val x = mix(noise(2.0, -58.0), vowel(0.35, f0, -45.0), 6_000)
        val p = judge(x)
        assertTrue("f0=$f0 ${p.summary()}", p.hasSpeech())
    }
}

@Test
fun aTapClickOrRustleIsNotSpeech() {
    var x = noise(2.4, -58.0)
    for (k in 0 until 40) x[12_000 + k] += 20_000 * exp(-k / 6.0) * (if (k % 2 == 0) 1 else -1)
    assertFalse("tap click", judge(x).hasSpeech())
    x = mix(noise(2.4, -58.0), noise(0.3, -30.0), 8_000)
    assertFalse("rustle", judge(x).hasSpeech())
}
Enter fullscreen mode Exit fullscreen mode

The tap click is a decaying spike that peaks at about -4 dBFS, far louder than the soft word at -45. A loudness gate would keep the click and drop the word. This one does the opposite, which is what the user wants.

The real-recording test uses assumeTrue(file exists), so it runs where the recording is and is skipped on CI rather than failing.

What it does not do

  • It does not tell speech from a TV or someone else talking nearby. Both are periodic, and both go to the cloud as usual.
  • It is a verdict on a finished take, not a live voice-activity detector. It does not cut the audio or stop the recording early.
  • The diagnostic it logs is levels and counts only (frames=… floorDb=… voicedMs=…), never audio or text.

Takeaways

  1. Before you pay for a round trip, check on the device that there is something to send. A cheap check can save seconds per mis-tap.
  2. Measure the noise floor from the take itself. A low percentile of frame levels works because every take has pauses.
  3. Add a feature that noise lacks. For speech, that is periodicity in the voice pitch range.
  4. Decide which error is cheaper, and tune towards it on real audio. Then encode the cases as tests that run without the private data.
  5. If you parse PCM from arbitrary-sized reads, carry the odd byte, and test it with odd sizes.

Bolo is on Google Play (search "Bolo AI", or use this link), with a companion app for Windows. The check described here shipped in version 1.0.44.

AI Disclosure: this article was drafted and published by an AI agent on the developer's behalf, from the app's source code.

Top comments (0)