DEV Community

StareBrain
StareBrain

Posted on

Building StareBrain, Part 19: Voice Input While Another App Is in the Foreground

#android #kotlin #a11y #mobile

Building StareBrain, Part 19: Voice Input While Another App Is in the Foreground

Parts 16–18 covered the overlay UI: how it appears, how users reposition it, and how it collapses when they're actively using another app. Today we add voice input.

The core constraint: StareBrain should accept a spoken instruction while the user is in a completely different app — without requiring them to switch back to StareBrain first.

The Trigger: Floating Mic Button

The overlay already floats above every app. We add a small mic button to its collapsed state. Tapping it starts a voice session without leaving whatever app the user is in.

class OverlayView(context: Context) : FrameLayout(context) {

    var onMicTapped: (() -> Unit)? = null

    private val micButton: ImageButton = ImageButton(context).apply {
        setImageResource(R.drawable.ic_mic)
        setOnClickListener { onMicTapped?.invoke() }
    }

    init {
        addView(micButton)
    }
}
Enter fullscreen mode Exit fullscreen mode

In OverlayManager, we wire the callback:

overlayView = OverlayView(context).also { view ->
    view.onMicTapped = { voiceInputManager.startListening() }
    windowManager.addView(view, params)
}
Enter fullscreen mode Exit fullscreen mode

VoiceInputManager

Wraps Android's SpeechRecognizer. Handles start, result, and error:

class VoiceInputManager(
    private val context: Context,
    private val onResult: (String) -> Unit,
    private val onError: (Int) -> Unit
) {

    private var recognizer: SpeechRecognizer? = null

    fun startListening() {
        recognizer?.destroy()
        recognizer = SpeechRecognizer.createSpeechRecognizer(context).apply {
            setRecognitionListener(object : RecognitionListener {
                override fun onResults(results: Bundle?) {
                    val matches = results
                        ?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
                    val best = matches?.firstOrNull() ?: return
                    onResult(best)
                }
                override fun onError(error: Int) {
                    onError(error)
                }
                // Other RecognitionListener methods left as no-ops
                override fun onReadyForSpeech(params: Bundle?) {}
                override fun onBeginningOfSpeech() {}
                override fun onRmsChanged(rmsdB: Float) {}
                override fun onBufferReceived(buffer: ByteArray?) {}
                override fun onEndOfSpeech() {}
                override fun onPartialResults(partialResults: Bundle?) {}
                override fun onEvent(eventType: Int, params: Bundle?) {}
            })
            startListening(Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
                putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                    RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
                putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, 1)
            })
        }
    }

    fun destroy() {
        recognizer?.destroy()
        recognizer = null
    }
}
Enter fullscreen mode Exit fullscreen mode

SpeechRecognizer requires RECORD_AUDIO permission — classified as AppRiskTier.HIGH in our permission model, so it goes through ConfirmationGuard on first use.

Connecting to the Pipeline

Voice input is just another instruction source. The result goes through the same path as a typed instruction:

val voiceInputManager = VoiceInputManager(
    context = this,
    onResult = { instruction ->
        lifecycleScope.launch {
            val context = contextBuilder.build(
                pendingTask = taskManager.currentTask,
                lastUserIntent = sessionMemory.lastIntent
            )
            sessionMemory.updateIntent(instruction)
            val prompt = buildInitialPrompt(context, instruction)
            launchPipeline(prompt, context)
        }
    },
    onError = { errorCode ->
        overlayManager.showError(mapSpeechError(errorCode))
    }
)
Enter fullscreen mode Exit fullscreen mode

No special handling for voice vs text — the pipeline doesn't know or care which input method was used.

Overlay Feedback During Listening

The overlay shows a listening state while SpeechRecognizer is active:

sealed class OverlayState {
    object Idle : OverlayState()
    object Listening : OverlayState()  // new
    data class Running(...) : OverlayState()
    data class AwaitingConfirmation(...) : OverlayState()
}
Enter fullscreen mode Exit fullscreen mode

In VoiceInputManager.startListening(), before starting the recognizer:

overlayManager.setState(OverlayState.Listening)
Enter fullscreen mode Exit fullscreen mode

On result or error, the state transitions to Running or back to Idle. The user can see StareBrain received their instruction without switching apps.

Error Mapping

SpeechRecognizer error codes are integers. We map them to readable messages:

private fun mapSpeechError(errorCode: Int): String {
    return when (errorCode) {
        SpeechRecognizer.ERROR_NO_MATCH -> "Didn't catch that — try again"
        SpeechRecognizer.ERROR_SPEECH_TIMEOUT -> "No speech detected"
        SpeechRecognizer.ERROR_AUDIO -> "Microphone error"
        SpeechRecognizer.ERROR_NETWORK -> "Network unavailable"
        else -> "Voice input failed"
    }
}
Enter fullscreen mode Exit fullscreen mode

What's Next

Part 20 will cover the full session handoff: what StareBrain preserves when the user closes and reopens the app mid-task, and how AgentContext and SessionMemory combine to resume without asking the user to repeat themselves.

Top comments (0)