DEV Community: vmodal_ai

Building an Android SDK for Search Inside Video Content

vmodal_ai — Tue, 18 Aug 2026 14:03:24 +0000

Searching inside video is usually limited to metadata: titles, descriptions, filenames, tags, or manually added captions.

But what if an Android application could search for something that actually appears inside the video itself?

For example:

Search for "car" → find the moments where a car appears in the video.

This concept explores building a small Android SDK that makes this type of video-content search possible.

The Idea

Traditional video search often looks something like this:

User Query
    ↓
Video Metadata
    ↓
Title / Description / Tags
    ↓
Search Results

This works well when the required information exists in the metadata.

However, metadata doesn't tell you everything that happens inside a video.

A more powerful approach is:

Video
  ↓
Analyze Video Frames
  ↓
Understand Visual Content
  ↓
Create Searchable Data
  ↓
User Query
  ↓
Matching Video Moments

Instead of searching about the video, we're searching inside the visual content of the video.

A Simple Example

Imagine a 10-minute video containing:

A person walking
A car driving past
A dog running
A laptop being used
A person drinking coffee

A traditional metadata search might only know:

Title: My Daily Vlog
Description: A day in my life

A visual-content search could potentially return:

"car"

00:02:14 → 00:02:21
Car detected

"dog"

00:05:32 → 00:05:41
Dog detected

"laptop"

00:07:10 → 00:07:45
Laptop detected

This transforms a video from a simple media file into a searchable visual dataset.

Building the Android SDK

The goal of the SDK is to hide the complexity of video processing behind a simple Android API.

The application developer shouldn't need to implement frame extraction, processing, indexing, and searching themselves.

Conceptually, the SDK could provide an API similar to:

val video = VideoSearch.create(videoUri)

video.analyze()

val results = video.search("car")

results.forEach {
    println("Found at ${it.startTime} - ${it.endTime}")
}

The exact implementation can vary depending on the computer-vision and AI models being used.

How Video Analysis Works

A practical implementation doesn't necessarily need to process every single video frame.

Instead, the SDK can sample frames at a configurable interval.

For example:

Video
─────────────────────────────────────>
  0s     1s     2s     3s     4s
   ↓      ↓      ↓      ↓      ↓
 Frame   Frame  Frame  Frame  Frame

Each selected frame can then be analyzed using an object-detection or vision model.

The result might look conceptually like:

{
  "timestamp": 12.4,
  "objects": [
    "person",
    "car",
    "road"
  ]
}

The SDK can store these results in an index that can later be searched.

From Detection to Search

The important part isn't just detecting objects.

The SDK needs to connect detections with their timestamps.

For example:

00:01:10 → person
00:01:12 → person, car
00:01:14 → car
00:01:16 → car
00:01:18 → road

When the user searches for car, the SDK can combine nearby detections into a meaningful video segment:

00:01:12 → 00:01:16

The application can then jump directly to that point in the video.

Android Integration

An Android SDK like this can be integrated into applications such as:

Video management apps
Media players
Surveillance applications
Educational platforms
Sports analysis tools
E-learning applications
Video libraries
AI-powered media applications

A typical architecture could look like:

Android Application
        │
        ▼
 Video Search SDK
        │
        ├── Video Frame Extraction
        │
        ├── Vision / AI Processing
        │
        ├── Content Index
        │
        └── Search Engine
                │
                ▼
        Matching Video Moments

Important Performance Considerations

Video processing can be expensive on mobile devices.

Processing every frame of a long video can consume significant CPU, memory, battery, and storage.

A production SDK therefore needs to consider:

Frame Sampling

Instead of analyzing 30 or 60 frames per second, the SDK can analyze selected frames.

Background Processing

Analysis should run away from the main Android UI thread so that the application remains responsive.

Incremental Processing

Large videos can be processed in chunks instead of loading the entire video into memory.

Local vs Cloud Processing

There are two major approaches:

On-device processing

Video → Android Device → AI Model → Search Index

Advantages include privacy and offline operation.

Cloud processing

Video → Server → AI Model → Search Index
                    ↓
                 Android

This can provide access to larger models but introduces network requirements and potentially higher processing costs.

Why Build This as an SDK?

Building this functionality as an SDK makes it reusable.

Instead of implementing video understanding separately in every application, developers can integrate the SDK and focus on their application's user experience.

For example:

                    Video Search SDK
                           │
          ┌────────────────┼────────────────┐
          ↓                ↓                ↓
     Video Player     Security App     Education App

The same underlying video-search capability can support completely different products.

Future Possibilities

Once video content becomes searchable, many interesting features become possible.

For example:

Search for objects inside videos
Search for people or activities
Find specific scenes
Automatically generate video highlights
Create searchable video libraries
Build AI-powered video navigation
Jump directly to relevant moments
Combine visual search with speech transcription

The bigger idea is to move from:

"Search for videos"

to:

"Search inside videos."

Android SDK

The Android SDK for this concept is available on GitHub:

v-modal Android SDK on GitHub

The project provides a foundation for experimenting with video processing and content-aware functionality on Android.

Conclusion

Video contains far more information than its title, description, and metadata.

By analyzing the actual visual content of video frames and associating detected content with timestamps, we can turn ordinary videos into searchable visual experiences.

An Android SDK can make this capability easier to integrate into mobile applications while hiding much of the underlying video-processing complexity.

The long-term vision is simple:

Don't just search for the video. Search for what happens inside it.

Useful Links

Website: www.v-modal.com

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Hello, Concept: Building a small Android SDK for search inside video content (inside the real image content , not just metadata), Android SDK: https://github.com/v-modal/vmodal_sdk_android

vmodal_ai — Tue, 18 Aug 2026 13:59:29 +0000

GitHub - v-modal/vmodal_sdk_android: Video Search SDK for Android Kotlin. Integrate in any video app · GitHub

Video Search SDK for Android Kotlin. Integrate in any video app - v-modal/vmodal_sdk_android

github.com

Object Detection on Android for Autonomous Robots

vmodal_ai — Mon, 17 Aug 2026 20:36:47 +0000

Object Detection on Android for Autonomous Robots

Introduction

Autonomous robots need to recognize objects in their environment. Object detection can identify people, vehicles, tools, signs, and obstacles from camera frames.

Android can perform edge inference locally, reducing dependency on network connectivity.

Architecture

CameraX
   |
Preprocessing
   |
Object Detection Model
   |
Postprocessing
   |
Detection Results
   |
Robot Perception Gateway

Detection Model

Define a reusable result type:

data class Detection(
    val label: String,
    val confidence: Float,
    val left: Float,
    val top: Float,
    val right: Float,
    val bottom: Float
)

This keeps the rest of the application independent from a particular model runtime.

CameraX Analysis

The camera analyzer should process frames asynchronously:

imageAnalysis.setAnalyzer(executor) { image ->
    detector.detect(image)
    image.close()
}

Use a latest-frame strategy when real-time responsiveness is more important than processing every frame.

Model Runtime

The detector can be implemented behind an interface:

interface ObjectDetector {
    suspend fun detect(frame: ImageFrame): List<Detection>
}

Possible mobile inference approaches include TensorFlow Lite or ONNX Runtime, depending on the model and deployment requirements.

Confidence Filtering

Not every prediction should be passed to the navigation system.

val valid = detections.filter {
    it.confidence >= 0.6f
}

The threshold should be evaluated against the target environment rather than chosen arbitrarily.

Non-Maximum Suppression

Detection models may produce overlapping predictions.

Prediction A  ───────
Prediction B    ───────
        ↓
       NMS
        ↓
Single Detection

Use the postprocessing method expected by your selected model.

Robot Integration

The Android device can send detections to the robot:

{
  "label": "person",
  "confidence": 0.94,
  "bbox": [120, 80, 350, 500]
}

For autonomous navigation, the robot should combine this with physical measurements such as depth or LiDAR when distance matters.

Tracking

Instead of detecting every object from scratch at every stage, an additional tracking layer can maintain object identities between frames.

Detection
   ↓
Tracking
   ↓
Object ID
   ↓
Navigation / Behavior

Tracking can reduce redundant processing and provide temporal context.

Performance Optimization

Important optimization techniques include:

Use an appropriately sized model
Reduce input resolution when acceptable
Reuse buffers
Avoid bitmap copies
Run inference off the main thread
Drop stale frames
Measure end-to-end latency

Autonomous Decision Making

Keep AI perception separate from robot control:

Camera
  ↓
Object Detection
  ↓
Perception State
  ↓
Navigation / Behavior
  ↓
Safety Controller
  ↓
Robot

This separation makes the system easier to test and safer to operate.

Testing

Evaluate the system using representative scenarios:

Static objects
Moving people
Multiple objects
Low light
Bright light
Partial occlusion
Camera vibration
Device thermal throttling

Measure both detection accuracy and real-time performance.

Conclusion

Object detection on Android can provide useful edge perception for autonomous robots. Kotlin, CameraX, and a mobile inference runtime create a flexible foundation that can later be connected to ROS 2, sensor fusion, navigation, and Physical AI agents.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Building a Voice-Controlled Robot with Kotlin and LLMs

vmodal_ai — Mon, 17 Aug 2026 20:36:38 +0000

Building a Voice-Controlled Robot with Kotlin and LLMs

Introduction

Natural-language interfaces can make robots easier to operate. Instead of selecting individual buttons, an operator can say commands such as asking a robot to move, inspect an area, or report its status.

In this tutorial, we will design an Android application that captures speech, sends the text through an LLM-based command parser, and converts the result into validated robot actions.

Architecture

User Voice
    ↓
Android Speech Recognition
    ↓
Command Text
    ↓
LLM Intent Parser
    ↓
Structured Robot Command
    ↓
Safety Validator
    ↓
Robot Gateway
    ↓
ROS 2 / Robot

The LLM should not directly control motors. It should produce structured intent that a deterministic safety layer validates.

Android Voice Input

Android provides speech-recognition APIs that can be used to convert spoken commands into text.

A simplified flow is:

fun onSpeechResult(text: String) {
    viewModel.processCommand(text)
}

The ViewModel can then pass the text to the command-processing layer.

Command Model

Define a strict command structure:

sealed interface RobotCommand {
    data object Stop : RobotCommand

    data class Move(
        val direction: String,
        val distanceMeters: Double
    ) : RobotCommand

    data class Rotate(
        val degrees: Double
    ) : RobotCommand
}

Using a structured representation prevents the robotics layer from receiving arbitrary natural-language instructions.

LLM Intent Parsing

The LLM can transform:

"Move forward two meters"

into structured data such as:

{
  "command": "move",
  "direction": "forward",
  "distance_meters": 2
}

Use structured output or a schema-constrained response where the selected LLM/API supports it.

Validation Layer

Never send the LLM result directly to the robot.

Validate:

LLM Output
    ↓
Schema Validation
    ↓
Range Validation
    ↓
Robot State Check
    ↓
Safety Policy
    ↓
Execution

For example:

fun validate(command: RobotCommand): Boolean {
    return when (command) {
        RobotCommand.Stop -> true
        is RobotCommand.Move ->
            command.distanceMeters in 0.0..5.0
        is RobotCommand.Rotate ->
            command.degrees in -180.0..180.0
    }
}

The exact limits should be determined by the robot's capabilities and safety requirements.

Robot Gateway

After validation, the Android app sends a structured command:

{
  "type": "move",
  "direction": "forward",
  "distanceMeters": 2.0
}

The gateway converts this command into the appropriate ROS 2 service, action, or topic.

Robot Feedback

The robot should return status information:

{
  "state": "executing",
  "battery": 84,
  "position": {
    "x": 2.1,
    "y": 4.3
  }
}

The Android application can show this in a Compose dashboard.

Handling Ambiguous Commands

Natural language can be ambiguous.

For example:

"Go over there."

The system should not guess what "there" means.

Instead, the application can request clarification:

"I need a destination before I can move the robot."

This is especially important for physical actions.

Voice + Vision

The system becomes more powerful when voice and vision are combined.

For example:

User:
"Follow the person wearing a red shirt."

Voice → Intent
Vision → Person Detection
       ↓
Robot Navigation

The LLM can coordinate high-level intent while deterministic robotics components handle perception and navigation.

Offline and Cloud Options

Speech recognition and LLM inference can be deployed in different ways:

Android
  |
  +-- Local speech recognition
  |
  +-- Cloud LLM

or:

Android
  |
Local/Edge AI
  |
Robot / Jetson

Choose the architecture based on latency, privacy, connectivity, and hardware constraints.

Safety Architecture

A recommended control hierarchy is:

Natural Language
       ↓
LLM
       ↓
Structured Intent
       ↓
Deterministic Planner
       ↓
Safety Controller
       ↓
Robot

The LLM should remain outside the final safety-critical control loop.

Testing

Test commands using a simulated robot before physical deployment.

Include:

Valid commands
Invalid commands
Ambiguous commands
Extreme values
Network failures
LLM failures
Speech-recognition errors
Emergency stop

Conclusion

Combining Android, Kotlin, speech recognition, LLMs, and robotics creates a natural interface for Physical AI systems. The key design principle is to use AI for interpretation and high-level planning while deterministic software remains responsible for validation and safe physical execution.

This architecture can be extended toward multimodal robot agents that combine voice, vision, maps, sensors, and autonomous task planning.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Android + NVIDIA Jetson + ROS 2: Building an AI Robot

vmodal_ai — Mon, 17 Aug 2026 20:36:31 +0000

Android + NVIDIA Jetson + ROS 2: Building an AI Robot

Introduction

A practical Physical AI system often separates the user interface, AI compute, robotics middleware, and hardware control.

In this architecture, Android provides the operator interface, an NVIDIA Jetson provides edge AI compute, and ROS 2 coordinates robotics workloads.

Architecture

             Android / Kotlin
                    |
             Secure Gateway
                    |
                  ROS 2
             /      |              Vision     Nav       Control
          |
    NVIDIA Jetson
          |
     AI Inference
          |
    Robot Sensors
          |
     Robot Hardware

This separation makes it possible to upgrade individual components without rebuilding the entire system.

Android Application

The Android application can provide:

Robot connection status
Camera stream
AI detections
Battery information
Navigation controls
Emergency stop
Robot diagnostics

Use Jetpack Compose to build the operator interface.

Jetson AI Computer

The Jetson can run computationally intensive workloads such as:

Object detection
Object tracking
Depth estimation
Visual SLAM
Navigation
Sensor fusion

The Android device does not need to perform every AI operation itself.

ROS 2 Layer

ROS 2 provides communication between robotics components.

A possible topic layout is:

/cmd_vel
/odom
/scan
/camera/image
/detections
/battery_state
/robot_status

Keep the topic structure small and intentional for the mobile interface.

Android-to-ROS Gateway

Rather than making Android responsible for ROS 2 internals, use a gateway:

Android
   |
WebSocket / MQTT / ROS bridge
   |
ROS 2 Gateway
   |
ROS 2 Nodes

The gateway can authenticate clients, validate commands, and expose only approved functionality.

AI Perception Pipeline

The Jetson can process camera frames:

Camera
   ↓
ROS 2 Image Topic
   ↓
Jetson AI Node
   ↓
Detection / Tracking
   ↓
ROS 2 Detection Topic

The Android app can subscribe to summarized results rather than receiving raw sensor data when bandwidth is limited.

Android Dashboard

The dashboard can display:

Robot: ONLINE
Battery: 87%
Mode: AUTONOMOUS
Objects: 4
Position: X 2.3 / Y 4.8

Compose state can be backed by Kotlin StateFlow.

Command Flow

For manual control:

Android
   ↓
Velocity Command
   ↓
Gateway
   ↓
ROS 2
   ↓
Safety Controller
   ↓
Robot Base

The safety controller should remain authoritative over the physical robot.

Autonomous Mode

For autonomous operation:

Sensors
   ↓
Jetson Perception
   ↓
Localization
   ↓
Navigation
   ↓
Safety Controller
   ↓
Robot

Android becomes a monitoring and supervisory interface rather than the primary controller.

Security

A production system should include:

Device authentication
Encrypted communication
Command authorization
Network segmentation
Rate limiting
Robot-side safety limits
Emergency stop

Never assume that a mobile application being inside the same Wi-Fi network makes the robot network trusted.

Testing

Use simulation before deploying to hardware.

Validate:

ROS 2 topic communication
AI inference
Android connectivity
Command timeouts
Network interruptions
Camera streaming
Safety behavior

Scaling to Robot Fleets

The same architecture can support multiple robots:

Android
   |
Fleet Gateway
   |
+--+---------+---------+
|            |         |
Robot 01   Robot 02  Robot 03

Each robot can expose a controlled namespace and telemetry stream.

Conclusion

Android, NVIDIA Jetson, and ROS 2 form a strong architecture for Physical AI applications. Android handles human interaction, Jetson handles demanding edge AI workloads, and ROS 2 coordinates perception, navigation, and control.

This architecture can later be extended with LLM-based planning, voice interaction, computer vision, and autonomous task execution.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Android Computer Vision for Robot Navigation

vmodal_ai — Mon, 17 Aug 2026 20:36:24 +0000

Android Computer Vision for Robot Navigation

Introduction

Navigation requires a robot to understand its surroundings. Cameras can provide useful visual information such as obstacles, landmarks, people, road boundaries, and navigable regions.

Android devices provide cameras and on-device compute that can be used for robotics prototypes.

Architecture

Android Camera
     |
 CameraX
     |
 Image Processing
     |
 Vision Model
     |
 Navigation Features
     |
 Robot Gateway / ROS 2

The Android device should provide perception data while the robot's navigation stack remains responsible for safe movement.

Camera Pipeline

Use CameraX ImageAnalysis:

imageAnalysis.setAnalyzer(executor) { image ->
    // Convert frame
    // Process image
    // Produce navigation features
    image.close()
}

Never perform expensive image processing on the UI thread.

Navigation Features

Computer vision can produce information such as:

Obstacle detected
Free-space region
Person detected
Lane boundary
Visual landmark

These features can be combined with robot sensors.

Obstacle Detection

A simple perception pipeline is:

Camera
  ↓
Object / Obstacle Detection
  ↓
Bounding Boxes
  ↓
Spatial Reasoning
  ↓
Navigation Layer

A 2D detection alone does not provide reliable distance. For physical navigation, combine vision with depth, stereo cameras, LiDAR, calibration, or other sensors where appropriate.

Visual Odometry

Camera motion can also contribute to localization.

Conceptually:

Frame A
  ↓
Feature Extraction
  ↓
Feature Matching
  ↓
Motion Estimation
  ↓
Frame B

Visual odometry is usually part of a larger localization system and should be fused with other sensors when possible.

Sending Vision Data

The Android application can publish compact perception messages:

{
  "obstacles": [
    {
      "type": "person",
      "confidence": 0.91
    }
  ]
}

A robot-side node can combine this information with LiDAR and odometry.

Performance

For real-time operation:

Reduce unnecessary frame conversions
Process only the latest frame
Use hardware acceleration where supported
Choose an appropriate model size
Measure inference latency
Monitor device temperature

Navigation Safety

Vision should not directly command motors.

Use:

Vision
  ↓
Perception
  ↓
Navigation Planner
  ↓
Safety Controller
  ↓
Motor Controller

The safety controller should enforce speed, collision, and emergency-stop constraints.

Testing

Test perception under:

Different lighting
Motion blur
Shadows
Crowded scenes
Indoor and outdoor environments
Camera orientation changes

Test the complete perception-to-action latency rather than only model inference time.

Conclusion

Android computer vision can provide valuable perception capabilities for robot navigation. When combined with ROS 2, sensor fusion, and a dedicated safety controller, it becomes a useful component in a Physical AI robotics architecture.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

YOLO Object Detection on Android for Robotics

vmodal_ai — Mon, 17 Aug 2026 20:36:18 +0000

YOLO Object Detection on Android for Robotics

Introduction

Object detection is an important capability for autonomous robots. A robot can use detections to identify people, vehicles, tools, obstacles, and other objects in its environment.

YOLO-family models are widely used for real-time object detection. In this tutorial, we will design an Android application that captures camera frames and performs YOLO inference locally.

System Architecture

CameraX
   |
Preprocessing
   |
YOLO Model
   |
Postprocessing
   |
Bounding Boxes
   |
Robot Perception Layer

The exact model format depends on the runtime you choose. For Android edge deployment, an exported model may be converted to a mobile-compatible format and executed using an appropriate inference runtime.

Android Project

Organize the project into separate layers:

vision/
├── CameraManager.kt
├── YoloDetector.kt
├── Detection.kt
└── DetectionOverlay.kt

This prevents camera handling, inference, and rendering from becoming tightly coupled.

Detection Model

Create a Kotlin model:

data class Detection(
    val classId: Int,
    val label: String,
    val confidence: Float,
    val boundingBox: RectF
)

Camera Pipeline

Use CameraX ImageAnalysis to obtain frames.

imageAnalysis.setAnalyzer(executor) { image ->
    detector.process(image)
    image.close()
}

For real-time robotics, use a backpressure strategy that drops stale frames rather than allowing an inference queue to grow indefinitely.

Preprocessing

Most object-detection models expect a fixed input size.

The preprocessing stage normally performs:

Rotation correction
Resize
Color conversion
Normalization
Tensor creation

For example:

Camera Frame
   ↓
Resize
   ↓
Normalize
   ↓
Tensor
   ↓
YOLO

The preprocessing code must match the model's training/export requirements.

Running Inference

Create a detector abstraction:

class YoloDetector {
    suspend fun detect(frame: ImageFrame): List<Detection> {
        // preprocess
        // inference
        // postprocess
        return emptyList()
    }
}

Run inference outside the Android main thread.

Postprocessing

Object detectors can return multiple candidate boxes. Postprocessing commonly includes:

Confidence filtering
Class filtering
Bounding-box conversion
Non-Maximum Suppression

For example:

Raw Predictions
      ↓
Confidence Filter
      ↓
NMS
      ↓
Final Detections

Drawing Bounding Boxes

The Android UI can display detection results over the live camera preview.

+--------------------------+
|                          |
|     +------------+       |
|     |   bottle   |       |
|     |    92%     |       |
|     +------------+       |
|                          |
+--------------------------+

Remember to map coordinates correctly when the preview and model input have different aspect ratios.

Using Detection for Robotics

Detection results can be passed to a robot control layer:

{
  "object": "person",
  "confidence": 0.92,
  "bbox": [120, 80, 350, 500]
}

The robotics layer can combine this information with depth, odometry, LiDAR, or other sensors.

Do not treat a 2D bounding box as a physical distance measurement unless the system has additional calibration or depth information.

Performance Optimization

For edge robotics:

Use a lightweight model where possible.
Reduce inference resolution when acceptable.
Reuse buffers.
Avoid unnecessary bitmap allocations.
Run inference off the UI thread.
Process only the latest frame.
Measure latency and thermal behavior.

A smaller model running consistently can be more useful for robotics than a larger model with unstable frame rates.

Testing

Test the detector with:

Indoor scenes
Outdoor scenes
Low light
Moving objects
Multiple objects
Camera rotation
Network disconnected

For robotics, also measure end-to-end latency:

Capture → Inference → Decision → Robot Command

Conclusion

YOLO-style object detection can turn an Android device into a useful edge-vision component for robotics. Kotlin, CameraX, and a mobile inference runtime provide the foundation for building perception prototypes that can later integrate with ROS 2 and autonomous navigation.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Streaming Robot Camera Feeds to Android with WebRTC

vmodal_ai — Mon, 17 Aug 2026 20:36:11 +0000

Streaming Robot Camera Feeds to Android with WebRTC

Introduction

Remote robot operation often requires a live camera feed. Traditional HTTP streaming can introduce latency, while WebRTC is designed for real-time media communication.

In this tutorial, we will design an Android application that receives a robot camera stream using WebRTC.

Architecture

Robot Camera
     |
 WebRTC Sender
     |
 Signaling Server
     |
 Android WebRTC Client
     |
 Compose UI

The signaling server exchanges connection information; the media path can then be established using WebRTC.

Android Components

A typical implementation contains:

WebRtcClient
SignalingClient
VideoRenderer
RobotCameraViewModel
CameraScreen

Keep WebRTC lifecycle management outside the Compose UI.

Peer Connection

The client creates a peer connection and configures the required ICE servers.

Conceptually:

class WebRtcClient {
    fun connect() {
        // Create peer connection
        // Configure ICE
        // Attach remote video track
    }
}

Use a maintained WebRTC Android library and follow its current API for peer-connection setup.

Signaling

WebRTC requires signaling to exchange session information.

A simplified message flow is:

Android             Robot
   |                   |
   | ---- Offer ------>|
   | <--- Answer ------|
   | <--- ICE -------->|
   | ---- ICE -------->|
   |                   |
   | === Media ========|

The signaling transport can use WebSocket or another real-time messaging mechanism.

Rendering Video

The WebRTC video track should be connected to an Android-compatible video renderer.

Keep the renderer lifecycle synchronized with the Activity or Compose screen lifecycle.

Compose Integration

Compose can host the native video rendering surface using Android interoperability APIs.

The Compose layer should mainly manage:

Connection state
Loading indicator
Error messages
Full-screen mode
Camera selection
Robot selection

Low-Latency Design

For robot teleoperation, latency matters.

Consider:

Hardware-accelerated encoding
Appropriate resolution
Reasonable bitrate
UDP connectivity where possible
Avoiding unnecessary transcoding
Measuring end-to-end latency

Do not optimize for maximum resolution at the expense of control responsiveness.

Connection Recovery

Robot camera connections can fail when the robot moves between networks.

Implement:

Disconnected
    ↓
Reconnect
    ↓
Signaling
    ↓
ICE Negotiation
    ↓
Streaming

Display the current state to the operator.

Security

Use authenticated signaling and encrypted WebRTC transport. Do not expose an unauthenticated robot camera endpoint to the public internet.

Camera + Telemetry

A useful operator screen combines video with telemetry:

+------------------------------+
|                              |
|       Robot Camera           |
|                              |
+------------------------------+
| Battery 82% | Speed 0.8 m/s  |
| Mode: AUTO  | Signal: Good   |
+------------------------------+

Conclusion

WebRTC is a strong technology for low-latency robot video. Combining it with Kotlin, Jetpack Compose, and a secure signaling service creates a foundation for remote robot monitoring and teleoperation.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Real-Time Robot Telemetry Visualization on Android

vmodal_ai — Mon, 17 Aug 2026 20:25:12 +0000

Real-Time Robot Telemetry Visualization on Android

Introduction

Robots continuously generate telemetry such as position, velocity, battery level, temperature, sensor readings, and operating state. A mobile application can turn this data into a real-time monitoring interface.

This tutorial demonstrates a clean Android architecture for receiving telemetry and rendering it efficiently.

Architecture

Robot Sensors
     |
 ROS 2 / Gateway
     |
 WebSocket / MQTT
     |
 Kotlin Repository
     |
 StateFlow
     |
 Compose Dashboard

Telemetry Model

Start with a strongly typed model:

data class Telemetry(
    val timestamp: Long,
    val x: Double,
    val y: Double,
    val velocity: Double,
    val battery: Float,
    val temperature: Float
)

Strong typing makes the UI and processing layer easier to maintain.

Streaming Data

A repository can expose telemetry as a Flow:

interface TelemetryRepository {
    fun telemetry(): Flow<Telemetry>
}

The ViewModel collects the stream:

viewModelScope.launch {
    repository.telemetry().collect {
        _telemetry.value = it
    }
}

Compose Visualization

Display the latest values:

@Composable
fun TelemetryPanel(data: Telemetry) {
    Column {
        Text("Position: ${data.x}, ${data.y}")
        Text("Velocity: ${data.velocity} m/s")
        Text("Battery: ${data.battery}%")
        Text("Temperature: ${data.temperature} °C")
    }
}

For historical values, keep a bounded buffer rather than storing unlimited telemetry in memory.

Real-Time Charts

Telemetry charts can visualize:

Velocity
  |
  |       /\ 
  |  /\  /  \__
  |_/  \/       \_
  +---------------- Time

Store only the amount of history required by the dashboard.

For example:

val history = ArrayDeque<Telemetry>()

Remove old samples when the configured buffer size is reached.

Handling High-Frequency Data

Robots may produce data much faster than the UI needs to refresh. Separate ingestion from rendering.

A practical strategy is:

Robot:       100 Hz
Processing:   50 Hz
UI updates:   10-30 Hz

The exact rates depend on the application.

Connection State

The dashboard should distinguish between:

Connected
Connecting
Disconnected
Reconnecting
Error

Do not display stale telemetry as if it were current.

Network Resilience

Implement:

Automatic reconnection
Connection timeout
Heartbeats
Timestamp validation
Backpressure
Offline/error state

Security

Telemetry can reveal sensitive information about robot operations. Use encrypted transport and authenticated robot gateways in production.

Conclusion

Real-time telemetry visualization becomes manageable when streaming, state management, and rendering are separated. Kotlin Flow and StateFlow provide a natural foundation for reactive robot dashboards, while Jetpack Compose efficiently reflects the latest state.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Building an AI-Powered Robot Vision System with Android

vmodal_ai — Mon, 17 Aug 2026 20:24:44 +0000

Building an AI-Powered Robot Vision System with Android

Introduction

Robot perception is one of the core components of Physical AI. A robot needs to understand its environment before it can make useful decisions.

Android devices provide cameras, GPUs, neural-processing capabilities, and modern machine-learning runtimes, making them useful for prototyping mobile robot perception systems.

In this tutorial, we will build the architecture for an Android vision system that captures camera frames, runs an AI model, and exposes detection results to a robotics application.

Architecture

Android Camera
      |
   Frame Pipeline
      |
   AI Inference
      |
 Detection Results
      |
+-----+------+
|            |
Android UI   Robot Gateway
             |
            ROS 2

The Android application performs perception while the robot middleware handles movement and navigation.

Camera Setup

Use CameraX to acquire frames from the device camera.

A typical pipeline contains:

Preview
   |
ImageAnalysis
   |
Frame Conversion
   |
AI Model

Keep image processing away from the main UI thread.

Frame Processing

The image analyzer should process frames efficiently:

imageAnalysis.setAnalyzer(
    executor
) { imageProxy ->
    // Convert frame
    // Run inference
    // Publish detections
    imageProxy.close()
}

Always close ImageProxy after processing. Otherwise, CameraX may stop delivering new frames.

AI Inference

The inference layer should be independent of CameraX:

interface VisionModel {
    suspend fun detect(frame: ImageFrame): List<Detection>
}

This abstraction lets you switch between TensorFlow Lite, ONNX Runtime, or another supported inference engine without rewriting the camera layer.

Detection Model

Define a common result structure:

data class Detection(
    val label: String,
    val confidence: Float,
    val left: Float,
    val top: Float,
    val right: Float,
    val bottom: Float
)

The UI can draw these bounding boxes over the camera preview.

Showing Detections

A Compose overlay can render detection information above the preview.

Camera Preview
+----------------------+
|     +--------+       |
|     | person  |       |
|     | 0.94    |       |
|     +--------+       |
|                      |
+----------------------+

For high frame-rate applications, avoid unnecessary allocations during every frame.

Sending Results to the Robot

Detection results can be forwarded to a robot gateway:

{
  "label": "person",
  "confidence": 0.94,
  "x": 0.42,
  "y": 0.51
}

The robot can combine these results with its own sensors before making navigation decisions.

Edge AI Considerations

Running inference locally provides several advantages:

Lower latency
Reduced network traffic
Operation without cloud connectivity
Better privacy
More predictable response times

However, thermal throttling, battery consumption, and model size must be considered.

Frame Rate Optimization

Do not necessarily run inference on every camera frame.

A useful strategy is:

Camera: 30 FPS
Inference: 10 FPS
Display: 30 FPS

The preview remains smooth while the AI model processes fewer frames.

Robot Safety

Vision predictions should not directly trigger dangerous robot actions without validation.

Use:

Camera
  ↓
AI Detection
  ↓
Decision Layer
  ↓
Safety Constraints
  ↓
Robot Command

The decision and safety layers should be responsible for validating AI output.

Extending the System

The same architecture can support:

Person detection
Object tracking
QR recognition
Lane detection
Obstacle detection
Pose estimation
Semantic segmentation
Depth estimation

Conclusion

Android can act as a capable edge-perception platform for Physical AI prototypes. Combining CameraX, Kotlin coroutines, edge inference, and a robotics gateway creates a flexible foundation for intelligent robots.

The next step is to combine vision with ROS 2 navigation and sensor fusion so that the robot can use perception results for autonomous behavior.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Building a Robot Dashboard with Jetpack Compose

vmodal_ai — Mon, 17 Aug 2026 20:24:40 +0000

Building a Robot Dashboard with Jetpack Compose

Introduction

A robot dashboard gives operators a single interface for monitoring and controlling a Physical AI system. Jetpack Compose makes it possible to build a responsive dashboard using declarative Android UI.

In this tutorial, we will create the architecture for a dashboard that displays robot connectivity, battery, operating mode, telemetry, and control actions.

Dashboard Architecture

Robot / ROS 2
     |
 Robot Gateway
     |
 Kotlin Repository
     |
 ViewModel + StateFlow
     |
 Jetpack Compose

The UI should observe state rather than communicate directly with the robot.

Robot State

Define a single state model:

data class RobotState(
    val connected: Boolean = false,
    val battery: Int = 0,
    val mode: String = "IDLE",
    val speed: Double = 0.0
)

The ViewModel can expose it using StateFlow:

private val _robotState = MutableStateFlow(RobotState())
val robotState = _robotState.asStateFlow()

Compose Dashboard

A simple dashboard can contain cards for each important metric:

@Composable
fun RobotDashboard(state: RobotState) {
    Column {
        Text("Robot Dashboard")
        Text("Status: ${if (state.connected) "ONLINE" else "OFFLINE"}")
        Text("Battery: ${state.battery}%")
        Text("Mode: ${state.mode}")
        Text("Speed: ${state.speed} m/s")
    }
}

For a production application, split the screen into reusable components such as StatusCard, BatteryCard, TelemetryCard, and ControlPanel.

Robot Controls

Controls should send commands through the ViewModel:

Button(
    onClick = { viewModel.stopRobot() }
) {
    Text("Emergency Stop")
}

Keep command execution outside composables so the UI remains testable.

Telemetry

Telemetry can be collected from a repository:

class RobotRepository {
    fun observeState(): Flow<RobotState> {
        // Connect to gateway
        TODO()
    }
}

The ViewModel collects the stream and exposes it to Compose.

Responsive UI

Robot dashboards may run on phones, tablets, rugged devices, or large operator displays. Use Compose adaptive layouts and avoid hard-coded dimensions.

A useful layout is:

+----------------------------------+
| Robot Status        Battery 87%  |
+----------------------------------+
| Telemetry            Controls    |
| X: 2.4               ↑           |
| Y: 1.8            ← STOP →       |
| Speed: 0.8             ↓         |
+----------------------------------+

Safety

Never make the dashboard the only safety mechanism. Emergency-stop and motion limits should be enforced by the robot-side control system.

Useful UI indicators include:

Connection status
Autonomous/manual mode
Battery level
Current velocity
Emergency-stop state
Sensor warnings

Conclusion

Jetpack Compose is a strong choice for modern Android robot dashboards. A state-driven architecture using Kotlin, StateFlow, ViewModel, and a dedicated robot gateway keeps the UI responsive and separates presentation from robotics communication.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Android + ROS 2: Building a Robot Control App with Kotlin

vmodal_ai — Mon, 17 Aug 2026 20:24:21 +0000

Android + ROS 2: Building a Robot Control App with Kotlin

Introduction

Physical AI is bringing together robotics, edge computing, computer vision, and intelligent mobile interfaces. Android is a useful companion platform because modern phones and tablets provide touch interfaces, cameras, sensors, networking, and strong edge-computing capabilities.

In this tutorial, we will design an Android application in Kotlin that communicates with a ROS 2 robot. The app will provide a simple control interface for sending movement commands and receiving robot telemetry.

Architecture

A practical architecture can look like this:

Android App
   |
   | ROS 2 bridge / WebSocket / MQTT
   v
ROS 2 Middleware
   |
   +---- /cmd_vel ----> Robot Base
   |
   +---- /odom -------> Telemetry
   |
   +---- /battery ----> Battery Status

The Android application should not directly control motors. Instead, it communicates with a ROS 2 node or bridge responsible for validating commands and interfacing with the robot.

Project Setup

Create a Kotlin Android project using Android Studio.

A clean package structure is:

com.example.robotcontroller
├── ui
├── ros
├── model
├── network
└── MainActivity.kt

Keep the ROS communication layer separate from the Compose UI so that the application can later switch between a simulator, development robot, or production robot.

Robot Command Model

Create a simple command model:

data class VelocityCommand(
    val linearX: Double,
    val angularZ: Double
)

The UI can map buttons or a virtual joystick to these values.

For example:

fun moveForward() = VelocityCommand(
    linearX = 0.5,
    angularZ = 0.0
)

fun stop() = VelocityCommand(
    linearX = 0.0,
    angularZ = 0.0
)

The communication layer then converts the command into the message format expected by your ROS 2 bridge.

Jetpack Compose Control UI

A simple control panel can be created with Jetpack Compose:

@Composable
fun RobotControls(
    onForward: () -> Unit,
    onBackward: () -> Unit,
    onLeft: () -> Unit,
    onRight: () -> Unit,
    onStop: () -> Unit
) {
    Column {
        Button(onClick = onForward) {
            Text("Forward")
        }

        Row {
            Button(onClick = onLeft) {
                Text("Left")
            }

            Button(onClick = onStop) {
                Text("Stop")
            }

            Button(onClick = onRight) {
                Text("Right")
            }
        }

        Button(onClick = onBackward) {
            Text("Backward")
        }
    }
}

For a real robot, replace these buttons with a joystick or gesture-based controller.

ROS 2 Communication Layer

The Android app needs a communication mechanism between the mobile device and ROS 2. A common architecture is to expose selected ROS 2 topics through a bridge or gateway.

The Android client can then publish commands such as:

/cmd_vel

and subscribe to telemetry topics such as:

/odom
/battery_state
/robot_status

Avoid exposing the entire ROS graph directly to an untrusted mobile client. Expose only the topics and services required by the application.

Receiving Telemetry

Represent telemetry in Kotlin:

data class RobotTelemetry(
    val battery: Float,
    val x: Double,
    val y: Double,
    val connected: Boolean
)

Use Kotlin coroutines and StateFlow to expose updates to Compose:

private val _telemetry = MutableStateFlow(
    RobotTelemetry(0f, 0.0, 0.0, false)
)

val telemetry: StateFlow<RobotTelemetry> = _telemetry

Compose can collect this state and update the dashboard automatically.

Safety Features

A robot-control application should include a reliable stop mechanism.

Implement:

Emergency stop
Automatic command timeout
Connection-loss detection
Maximum velocity limits
Authentication between the app and robot
TLS for network communication where supported

A particularly useful technique is a command watchdog: if the robot does not receive a valid command within a defined interval, the control node should command zero velocity.

Testing with Simulation

Before connecting physical hardware, test the Android application against a simulated ROS 2 robot.

A simulator lets you verify:

Movement commands
Telemetry
Network failures
Emergency stop behavior
UI responsiveness
Command limits

This dramatically reduces the risk of testing incorrect commands on physical hardware.

Production Architecture

For a production system, consider:

Android / Kotlin
       |
   Secure Gateway
       |
      ROS 2
       |
 Navigation / Perception
       |
 Robot Hardware

This separation allows the Android application to remain a user interface while ROS 2 handles robotics workloads.

Conclusion

Android and Kotlin can provide a powerful human interface for Physical AI systems. By combining Jetpack Compose, Kotlin coroutines, secure networking, and ROS 2, you can build mobile applications that monitor and control robots without coupling the Android UI directly to robot hardware.

The same architecture can later be extended with camera streaming, AI perception, voice commands, autonomous navigation, and LLM-based robot control.

Useful Links

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx