DEV Community

vmodal_ai
vmodal_ai

Posted on

Getting Started with the Unitree Camera SDK: Streaming Depth, Color, and Point Clouds from Your Go1

#robotics #opencv #computervision #cpp

If you own a Unitree Go1-Edu, you already have a small fleet of cameras on board: five binocular fisheye pairs mounted on the head, chin, both sides of the body, and the belly. The UnitreeCameraSDK is the official C++ library for talking to them — pulling raw frames, rectified stereo pairs, depth maps, and full point clouds, plus a simple way to stream images off the robot over the network.

This tutorial walks through what the SDK actually gives you, how to build it, and how to get each of its core examples running — along with the gotchas that trip people up (OpenCV version pinning, camera device locking, and Jetson-only GStreamer elements).

What you're working with

The SDK lives at unitreerobotics/UnitreecameraSDK and is a fairly thin, pragmatic wrapper around V4L2/GStreamer camera capture plus OpenCV's stereo calibration and rectification pipeline. It is not a general-purpose camera library — it assumes the exact hardware layout, device paths, and OpenCV version used on the Go1's onboard Jetson boards.

Repo layout:

UnitreeCameraSDK/
├── doc/                      # Doxygen-generated HTML docs
├── examples/                 # 7 example .cc programs
├── include/                  # Public headers
├── lib/                      # Prebuilt libraries for Jetson/x86
├── CMakeLists.txt
├── stereo_camera_config.yaml # point cloud / depth reading config
├── trans_rect_config.yaml    # image transmission config
└── version.txt
Enter fullscreen mode Exit fullscreen mode

At the center of it is UnitreeCameraSDK.hpp, which declares the UnitreeCamera class. UnitreeCamera inherits from a lower-level StereoCamera class and adds camera firmware update support on top of it. The actual stereo vision work — capture, rectification, disparity computation, point cloud generation — lives in StereoCameraCommon.hpp/.cc.

Calibration is modeled the way you'd expect from an OpenCV-based stereo pipeline: per-eye intrinsic matrices, distortion coefficients, a stereo rectification rotation matrix, and a translation vector for the baseline between the two lenses. Because these are fisheye lenses, the SDK supports both a FISHEYE/LONGLAT model and a standard PERSPECTIVE (pinhole) model, selected via an ImageRect enum depending on which camera you're reading from.

Where the camera devices sit

If you're working directly on the robot's Jetson boards (rather than through ROS), it helps to know the physical/network layout:

Board IP Cameras Dev IDs
Head Nano 192.168.123.13 Front face, chin Front = 1, Chin = 0
Body Nano 192.168.123.14 Left, right side Left = 0, Right = 1
Main Nano/NX 192.168.123.15 Belly 0

Each camera also has a dedicated port ID for network transmission (e.g. the front camera uses port 9201), which matters later when you try the streaming examples.

Prerequisites

  • OpenCV, version 4 or newer, built with GStreamer support
  • CMake 2.8+
  • OpenGL, GLUT, X11 — only needed if you want the point cloud GUI viewer

⚠️ Pin your OpenCV version. The SDK was built and tested against OpenCV 4.1.1. Several users report that newer OpenCV builds cause a segfault the moment you try to open a camera. If you're setting this up fresh and don't need a newer OpenCV for something else, install 4.1.1 first and save yourself the debugging session.

Building the SDK 

git clone https://github.com/unitreerobotics/UnitreecameraSDK.git
cd UnitreeCameraSDK
mkdir build && cd build
cmake ..
make
Enter fullscreen mode Exit fullscreen mode

This produces a set of example binaries you can run directly from the bin/ directory at the repo root.

Before you run anything: free the camera device

This is the single most common failure mode people hit with this SDK, and it's not really a bug — it's device contention. Unitree's own onboard software (point cloud nodes, MQTT control nodes, human pose estimation, ROS nodes) may already have the cameras open. If you try to open them yourself while those processes are running, you'll get errors like:

Invalid deviceNode!
Segmentation fault (core dumped)
Enter fullscreen mode Exit fullscreen mode

or

Cannot detect any unitree camera!
[UnitreeCameraSDK][ERROR] read to tmp file failed, maybe mkstemp file error!
[UnitreeCameraSDK][ERROR] This camera cannot get internal parameters!
Enter fullscreen mode Exit fullscreen mode

Before running any example, kill the competing processes:

ps -A | grep point | awk '{print $1}' | xargs kill -9
ps -aux | grep mqttControlNode | grep -v grep | head -n 1 | awk '{print $2}' | xargs kill -9
ps -aux | grep live_human_pose | grep -v grep | head -n 1 | awk '{print $2}' | xargs kill -9
Enter fullscreen mode Exit fullscreen mode

If you started any autostart scripts yourself, you may also need to run their corresponding kill.sh (e.g. Unitree/autostart/02camerarosnode/kill.sh and Unitree/autostart/04imageai/kill.sh).

Running the examples

1. Raw frames

Pulls the raw binocular RGB stream straight off a camera pair, no rectification applied.

cd UnitreeCameraSDK
./bin/example_getRawFrame
Enter fullscreen mode Exit fullscreen mode

This is the best first test — if this doesn't work, nothing downstream will either. Start here to confirm the camera device is free and accessible before troubleshooting anything more complex.

2. Calibration parameters

Reads back the camera's stored intrinsic and extrinsic calibration data: left/right intrinsic matrices, distortion coefficients, the stereo rectification rotation, and the baseline translation between the lenses.

./bin/example_getCalibParamsFile
Enter fullscreen mode Exit fullscreen mode

Useful if you're planning to do your own downstream computer vision work (e.g. feeding frames into a separate SLAM or depth pipeline) and need the exact calibration your specific unit shipped with, rather than assuming generic values.

3. Rectified frames

Applies stereo rectification to the raw pair so the left and right images are epipolar-aligned — a prerequisite for any disparity/depth computation.

./bin/example_getRectFrame
Enter fullscreen mode Exit fullscreen mode

4. Depth frames

Computes a depth map from the rectified stereo pair.

./bin/example_getDepthFrame
Enter fullscreen mode Exit fullscreen mode

5. Point clouds

Generates a 3D point cloud and renders it in a live OpenGL/GLUT window (this is why X11/OpenGL/GLUT are dependencies).

./bin/example_getPointCloud
Enter fullscreen mode Exit fullscreen mode

Point cloud and depth reading behavior — including things like depth range and output frame — are configured via stereo_camera_config.yaml at the repo root, worth checking if your output looks off.

6/7. Sending and receiving images over the network

This pair is meant for streaming frames from a robot-mounted board to another machine on the network.

On the sender (robot side):

./bin/example_putImagetrans
Enter fullscreen mode Exit fullscreen mode

On the receiver (your dev machine, or another board):

./bin/example_getimagetrans
Enter fullscreen mode Exit fullscreen mode

⚠️ This one is Jetson-specific. The receiving pipeline uses GStreamer elements like h264parse and omxh264dec — omxh264dec in particular is an NVIDIA Jetson hardware decoder element that generally won't exist on a generic desktop Linux GStreamer install. If you're trying to receive a stream on your laptop rather than another Jetson board, expect:

GStreamer warning: Error opening bin: no element "h264parse"
Enter fullscreen mode Exit fullscreen mode

and you'll need a GStreamer install with the relevant plugins, or a modified pipeline that swaps omxh264dec for a software decoder like avdec_h264.

A minimal custom program

Once the examples run, using the SDK directly in your own code follows the same basic shape. Something like:

#include "UnitreeCameraSDK.hpp"
#include <opencv2/opencv.hpp>

int main() {
    // deviceNode corresponds to the dev ID for the camera pair you're targeting
    UnitreeCamera cam(std::string("/dev/videoX"));

    if (!cam.isOpened()) {
        std::cerr << "Failed to open camera" << std::endl;
        return -1;
    }

    cam.startCapture();

    cv::Mat left, right;
    while (true) {
        if (!cam.getRawFrame(left, right)) continue;
        cv::imshow("left", left);
        cv::imshow("right", right);
        if (cv::waitKey(1) == 27) break; // Esc to quit
    }

    cam.stopCapture();
    return 0;
}
Enter fullscreen mode Exit fullscreen mode

The exact method names and constructor signature vary slightly by SDK version — check include/UnitreeCameraSDK.hpp and the matching example .cc file for the calls that ship with your checkout, since this class is a thin wrapper you're expected to read alongside the examples rather than a fully documented public API.

Known rough edges to plan around

  • OpenCV version lock-in. Pin to 4.1.1 unless you enjoy debugging segfaults on camera open. Community forks (e.g. ngmor/unitree_camera, a ROS 2 wrapper) exist partly to work around this.
  • Device contention with Unitree's own services. Always check for and kill competing processes before running an example.
  • Jetson-only transmission pipeline. Don't expect example_getimagetrans to "just work" on a non-Jetson receiver without pipeline changes.
  • Low commit velocity. This is vendor reference code, not an actively iterated library — treat the examples and headers as the primary documentation, and be prepared to read the .cc source directly when the Doxygen docs run thin.

Wrapping up

The Unitree Camera SDK is small enough to read end-to-end in an afternoon, which is actually its main strength: UnitreeCameraSDK.hpp and the seven examples cover raw capture, calibration, rectification, depth, point clouds, and network transmission in a way that's easy to trace through. The friction people hit is almost always environmental — OpenCV version, device locks, Jetson-specific GStreamer elements — rather than the SDK's API itself. Get past those three, and you have direct access to depth and point cloud data from all five camera pairs on the Go1.

If you build something on top of this — a custom SLAM front-end, an obstacle-avoidance layer, whatever — I'd love to hear about it in the comments.

Useful Links

Website: www.v-modal.com

SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter

SDK Android: https://github.com/v-modal/vmodal_sdk_android

Discord: https://discord.gg/K72z28KUx

Top comments (0)