Google XR + Flutter: Rendering Budgets for Spatial Companion Interfaces

#flutter #android #xr

Google XR + Flutter: Rendering Budgets for Spatial Companion Interfaces

This advanced tutorial treats Flutter as the operator-facing presentation layer while wearable SDKs, Jetson services, ROS 2 nodes, and NVIDIA-accelerated inference handle device-specific or compute-heavy work. The goal is predictable latency, bounded memory, useful observability, and graceful degradation—not simply maximizing message throughput.

Architecture note: Keep safety-critical control loops on the robot. A Flutter app is not a hard real-time controller and must not be the sole safety mechanism.

1. Define measurable service objectives

Track freshness, event-to-display latency, throughput, frame pacing, CPU, memory, network use, battery, thermal state, dropped messages, reconnect time, and command acknowledgements. Do not use a single FPS number to represent the entire system: a smooth screen can still display stale data.

2. Instrument the complete path

sensor capture
  -> native/ROS processing
  -> gateway publish
  -> Flutter receive
  -> state accepted
  -> widget build
  -> frame rendered

Record source and receive timestamps separately. Use monotonic clocks for elapsed durations within a process. When comparing devices, account for clock offset and uncertainty.

Profile on a physical target in profile mode and inspect Flutter DevTools' Performance view. Debug-mode timing is not a reliable production baseline. See https://docs.flutter.dev/tools/devtools/performance

3. Establish a repeatable baseline

Start from a clean launch.
Connect to the robot or replay a fixed trace.
Run a representative scenario for several minutes.
Record frame times, memory, CPU, message rates, stale-data counts, and reconnects.
Change one variable at a time.
Compare p50, p95, and p99 latency, not just averages.

class PipelineMetrics {
  int received = 0;
  int accepted = 0;
  int dropped = 0;
  int stale = 0;
  int uiPublishes = 0;
}

Use bounded metric windows or histograms; do not store every sample forever.

4. Separate acquisition from presentation

High-rate sources should not directly trigger widget rebuilds. Use a latest-value cache for state-like data, bounded queues for events that must be preserved, and a controlled presentation cadence.

class LatestValue<T> {
  T? _value;
  int _version = 0;

  void put(T value) {
    _value = value;
    _version++;
  }

  T? get value => _value;
  int get version => _version;
}

Replacing an older pending value may be acceptable for battery, pose, or temperature. Commands, alarms, and audit events need a separate reliable path and explicit acknowledgement semantics.

5. Make buffering bounded

Every queue needs a maximum capacity, an overflow policy, a drop/replacement counter, and an owner responsible for recovery.

Latest wins: pose, preview frames, current detections.
Drop old items: display-oriented streams where freshness matters.
Persist: events that must survive disconnection.
Reject with feedback: commands that cannot safely be queued.

An unbounded telemetry or log list turns overload into memory growth rather than a visible, measurable condition.

6. Protect the Flutter frame budget

Keep parsing, filtering, and transformations out of build(). Split screens into independently updating sections and avoid rebuilding camera or map widgets for unrelated state changes.

For CPU-heavy pure Dart work, benchmark compute or a dedicated isolate with realistic payload sizes. Isolates have startup and message-transfer costs. For hardware acceleration or platform APIs, prefer a native service designed for that workload.

7. Keep robotics compute near the robot

Sensors -> ROS 2 / accelerated nodes -> Jetson gateway
                                      |
                             compact UI-facing state
                                      |
                         WebSocket / MQTT / gRPC
                                      |
                              Flutter client

NVIDIA Isaac ROS provides accelerated, low-latency ROS 2 packages for Jetson and other NVIDIA platforms. Keep image processing, inference, and robot control on the compute side; send Flutter only the information needed for visualization and operator decisions. See https://nvidia-isaac-ros.github.io/

8. Treat stale data as a first-class state

Carry source timestamp, receive timestamp, sequence number/version, source identity, and optional confidence or quality. The UI must distinguish zero from unknown, disconnected, and stale. Do not show a green health indicator just because the last payload was healthy.

9. Validate graceful degradation

Simulate network delay and packet loss, bursts, robot process restarts, wearable permission loss, camera interruptions, inference slowdown, low memory, and thermal throttling. Under pressure, reduce preview resolution or update frequency and pause nonessential charts—but preserve command and safety status visibility.

10. Release checklist

[ ] Frame-time baseline captured on physical hardware
[ ] p50/p95/p99 event-to-display latency measured
[ ] Queues bounded and overflow observable
[ ] High-frequency state coalesced before UI updates
[ ] Commands and alarms use reliable semantics
[ ] Clock domains and stale-data rules documented
[ ] Network and reconnect behavior tested
[ ] Thermal and memory pressure scenarios tested
[ ] Robot safety remains independent of the mobile app
[ ] Performance budgets enforced in repeatable tests

Advanced focus: Spatial rendering budget

Keep frequently changing overlays small and isolate them from static panels. Avoid recalculating geometry and large collections for every pose update. If the XR experience is rendered by a native runtime, keep frame-critical spatial rendering native and use Flutter for companion controls unless the platform explicitly supports the required Flutter rendering path. Validate device and SDK support first.

Implementation exercise

Add timestamps and sequence IDs at the gateway boundary.
Capture a normal-load trace and a burst-load trace.
Implement a bounded buffer and an explicit overload policy.
Display stale/disconnected/degraded states separately.
Profile on the target device and compare p50, p95, and p99 results.
Document which events may be coalesced and which must be preserved.

Conclusion

The strongest performance gains usually come from controlling the whole data pipeline: acquisition, processing, filtering, transport, state updates, and rendering. Keep compute close to the hardware, bound every queue, measure freshness as well as frame rate, and keep safety-critical behavior independent of the Flutter client.