Open-Vocabulary Object Detection for Robots Using Vision-Language Models

Traditional object detectors are trained on a fixed set of classes.

For example:

person
car
chair
dog

But robots often encounter objects that were not explicitly included in their original training labels.

Open-vocabulary perception allows a robot to query concepts using natural language.

From Fixed Classes to Natural Language

Traditional:

Image --> Detector --> {person, car, chair}

Open vocabulary:

Image + "find a red toolbox"
             |
             v
       Vision-Language Model
             |
             v
       Candidate Regions

Robot Perception Pipeline

Camera
  |
  v
Image Preprocessing
  |
  v
Vision-Language Model
  |
  +--> "red toolbox"
  +--> "safety helmet"
  +--> "door handle"
  |
  v
Detected Regions
  |
  v
3D Localization
  |
  v
Robot Planner

Why This Matters

A robot deployed in the real world may receive commands such as:

Find the nearest orange package.

The system should not require a new fixed detector class for every possible object.

Connecting 2D and 3D

A VLM may identify an object in an image. Depth or LiDAR can then estimate its 3D location.

RGB Image
   |
   v
2D Object Region
   |
   +---- Depth
   |
   +---- LiDAR
   |
   v
3D Object Position

This transforms semantic understanding into spatial information.

Safety and Verification

Open-vocabulary models can produce uncertain or incorrect detections.

For robot control, add verification:

VLM Detection
      |
      v
Confidence Check
      |
      v
Geometric Validation
      |
      v
Temporal Consistency
      |
      v
Planner

Never assume that a language model's output is automatically safe for direct actuation.

ROS 2 Architecture

A modular implementation might use:

/camera/image
      |
      v
/vlm_detector
      |
      v
/detections
      |
      v
/3d_projection
      |
      v
/object_tracker
      |
      v
/planner

This makes it possible to replace the VLM without redesigning the rest of the robot stack.

Latency Management

Large models may be expensive.

Possible strategies include:

• Run perception at a lower frequency.
• Track detected objects between VLM calls.
• Resize images.
• Use hardware acceleration.
• Cache repeated queries.
• Use a smaller model when appropriate.

Example Application

A warehouse robot could receive:

"Find the damaged-looking package near the loading area."

The perception system can combine:

• Language understanding
• Visual appearance
• Spatial localization
• Object tracking

This is a major step toward more flexible physical AI systems.

Useful Links

• Website: https://www.v-modal.com
• SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
• SDK Android: https://github.com/v-modal/vmodal_sdk_android
• Discord: https://discord.gg/K72z28KU