DEV Community

vmodal_ai
vmodal_ai

Posted on

Building a Voice-Controlled Robot with Kotlin and LLMs

#android #kotlin #llm #robotics

Building a Voice-Controlled Robot with Kotlin and LLMs

Introduction

Natural-language interfaces can make robots easier to operate. Instead of selecting individual buttons, an operator can say commands such as asking a robot to move, inspect an area, or report its status.

In this tutorial, we will design an Android application that captures speech, sends the text through an LLM-based command parser, and converts the result into validated robot actions.

Architecture 

User Voice
    ↓
Android Speech Recognition
    ↓
Command Text
    ↓
LLM Intent Parser
    ↓
Structured Robot Command
    ↓
Safety Validator
    ↓
Robot Gateway
    ↓
ROS 2 / Robot
Enter fullscreen mode Exit fullscreen mode

The LLM should not directly control motors. It should produce structured intent that a deterministic safety layer validates.

Android Voice Input

Android provides speech-recognition APIs that can be used to convert spoken commands into text.

A simplified flow is:

fun onSpeechResult(text: String) {
    viewModel.processCommand(text)
}
Enter fullscreen mode Exit fullscreen mode

The ViewModel can then pass the text to the command-processing layer.

Command Model

Define a strict command structure:

sealed interface RobotCommand {
    data object Stop : RobotCommand

    data class Move(
        val direction: String,
        val distanceMeters: Double
    ) : RobotCommand

    data class Rotate(
        val degrees: Double
    ) : RobotCommand
}
Enter fullscreen mode Exit fullscreen mode

Using a structured representation prevents the robotics layer from receiving arbitrary natural-language instructions.

LLM Intent Parsing

The LLM can transform:

"Move forward two meters"
Enter fullscreen mode Exit fullscreen mode

into structured data such as:

{
  "command": "move",
  "direction": "forward",
  "distance_meters": 2
}
Enter fullscreen mode Exit fullscreen mode

Use structured output or a schema-constrained response where the selected LLM/API supports it.

Validation Layer

Never send the LLM result directly to the robot.

Validate:

LLM Output
    ↓
Schema Validation
    ↓
Range Validation
    ↓
Robot State Check
    ↓
Safety Policy
    ↓
Execution
Enter fullscreen mode Exit fullscreen mode

For example:

fun validate(command: RobotCommand): Boolean {
    return when (command) {
        RobotCommand.Stop -> true
        is RobotCommand.Move ->
            command.distanceMeters in 0.0..5.0
        is RobotCommand.Rotate ->
            command.degrees in -180.0..180.0
    }
}
Enter fullscreen mode Exit fullscreen mode

The exact limits should be determined by the robot's capabilities and safety requirements.

Robot Gateway

After validation, the Android app sends a structured command:

{
  "type": "move",
  "direction": "forward",
  "distanceMeters": 2.0
}
Enter fullscreen mode Exit fullscreen mode

The gateway converts this command into the appropriate ROS 2 service, action, or topic.

Robot Feedback

The robot should return status information:

{
  "state": "executing",
  "battery": 84,
  "position": {
    "x": 2.1,
    "y": 4.3
  }
}
Enter fullscreen mode Exit fullscreen mode

The Android application can show this in a Compose dashboard.

Handling Ambiguous Commands

Natural language can be ambiguous.

For example:

"Go over there."
Enter fullscreen mode Exit fullscreen mode

The system should not guess what "there" means.

Instead, the application can request clarification:

"I need a destination before I can move the robot."
Enter fullscreen mode Exit fullscreen mode

This is especially important for physical actions.

Voice + Vision

The system becomes more powerful when voice and vision are combined.

For example:

User:
"Follow the person wearing a red shirt."

Voice → Intent
Vision → Person Detection
       ↓
Robot Navigation
Enter fullscreen mode Exit fullscreen mode

The LLM can coordinate high-level intent while deterministic robotics components handle perception and navigation.

Offline and Cloud Options

Speech recognition and LLM inference can be deployed in different ways:

Android
  |
  +-- Local speech recognition
  |
  +-- Cloud LLM
Enter fullscreen mode Exit fullscreen mode

or:

Android
  |
Local/Edge AI
  |
Robot / Jetson
Enter fullscreen mode Exit fullscreen mode

Choose the architecture based on latency, privacy, connectivity, and hardware constraints.

Safety Architecture

A recommended control hierarchy is:

Natural Language
       ↓
LLM
       ↓
Structured Intent
       ↓
Deterministic Planner
       ↓
Safety Controller
       ↓
Robot
Enter fullscreen mode Exit fullscreen mode

The LLM should remain outside the final safety-critical control loop.

Testing

Test commands using a simulated robot before physical deployment.

Include:

  • Valid commands
  • Invalid commands
  • Ambiguous commands
  • Extreme values
  • Network failures
  • LLM failures
  • Speech-recognition errors
  • Emergency stop

Conclusion

Combining Android, Kotlin, speech recognition, LLMs, and robotics creates a natural interface for Physical AI systems. The key design principle is to use AI for interpretation and high-level planning while deterministic software remains responsible for validation and safe physical execution.

This architecture can be extended toward multimodal robot agents that combine voice, vision, maps, sensors, and autonomous task planning.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Top comments (0)