Streaming Robot Vision and AI Object Detection Results to Flutter in Real Time

Computer vision is a major component of Physical AI. Robots use cameras to understand environments, detect objects, avoid obstacles, and interact with people.

This tutorial explores an architecture for streaming robot video and AI object detection results to Flutter in real time.

The Complete Pipeline

Robot Camera
     |
     v
Edge AI Computer
     |
     +--> Object Detection Model
     |
     +--> Object Tracking
     |
     v
Video + Detection Metadata
     |
     v
WebRTC / WebSocket
     |
     v
Flutter Application

Why Separate Video and AI Metadata?

Instead of transmitting fully processed frames, send:

Video Stream
+
Detection Coordinates
+
Object Labels
+
Confidence Scores

Flutter can then render the detection overlays locally.

Detection Data Format

{
  "timestamp": 172000001,
  "detections": [
    {
      "label": "person",
      "confidence": 0.95,
      "x": 0.25,
      "y": 0.20,
      "width": 0.18,
      "height": 0.45
    }
  ]
}

Normalized coordinates make the data independent of screen size.

Creating a Detection Model

class Detection {
  final String label;
  final double confidence;
  final double x;
  final double y;
  final double width;
  final double height;

  Detection({
    required this.label,
    required this.confidence,
    required this.x,
    required this.y,
    required this.width,
    required this.height,
  });
}

Building the Video Stack

Stack(
  children: [
    RobotVideoPlayer(),
    DetectionOverlay(
      detections: detections,
    ),
  ],
)

Drawing Bounding Boxes

Positioned(
  left: detection.x * screenWidth,
  top: detection.y * screenHeight,
  width: detection.width * screenWidth,
  height: detection.height * screenHeight,
  child: Container(
    decoration: BoxDecoration(
      border: Border.all(),
    ),
    child: Text(detection.label),
  ),
)

Synchronizing Video and AI Results

Detection metadata should include timestamps:

Video Frame Timestamp
        |
        v
Detection Timestamp
        |
        v
Flutter Synchronization

Without synchronization, bounding boxes may lag behind moving objects.

Reducing Latency

Optimize:

Camera encoding
Network transmission
AI inference
Metadata size
Flutter rendering

Useful strategies include:

WebRTC for low-latency video
Binary messages
Frame skipping
Object tracking between inference frames
Controlled inference rates

Connecting Vision to Robot Actions

Person Detected
      |
      v
Calculate Distance
      |
      v
Reduce Robot Speed
      |
      v
Replan Navigation

Flutter can show these decisions to the operator in real time.

Production Considerations

A production system should include:

Secure video access
Authentication
Reconnection logic
Bandwidth adaptation
Detection timestamping
Backpressure handling
Privacy controls

Conclusion

Streaming robot vision to Flutter creates a powerful interface for Physical AI systems. Separating video transport from AI metadata can reduce bandwidth usage while keeping the interface responsive and interactive.