Turing Sim: Camera Ground Truth from One Hydra Render

Camera sensors now render RGB with pixel-aligned depth, normals, object and class masks, and boxes. 相机传感器现在能渲染 RGB，并输出逐像素对齐的深度、法线、物体和类别分割以及检测框。

The overview camera switching between RGB, depth, normals, objects, classes and boxes.

One colour per object: each UR10 link, the table and the cube.

Metric depth along the optical axis, shown with an inverse-depth Turbo ramp.

---

Situation

After IMU and LiDAR, the sensor stage needed cameras: not only pictures, but the ground truth that robot learning uses, such as depth, segmentation and boxes, aligned exactly with the image. Turing Sim renders with Storm (rasterised) or Turing RT (ray traced) through Hydra. Neither exposed its depth or id buffers to Python.

Task

Give camera sensors every output from a single render, so all of them describe the same pixels, with accuracy that can be checked. They also had to run in the editor without degrading the viewport.

Action

1. Cameras are USD cameras. A camera sensor is a standard UsdGeom.Camera with turingSensor:type = "camera". Presets set the lens from a field of view and the image's aspect.
2. Read the AOVs back. A small native module reads Hydra's color, depth, primId and instanceId buffers and decodes ids to scene paths, including instance proxies. It works with Storm and with Turing RT.
3. A separate render process. Integer ids need a render without multisampling, and Hydra reads that setting once per process, so cameras render in their own worker and the editor's viewport keeps its antialiasing. Storm's eye-normal buffer turned out to clamp negative components, so normals are rebuilt from depth.
4. From buffers to ground truth. RGB is averaged over odd supersampling; ground truth uses each pixel's centre sample. Depth is converted to metres along the optical axis. An object is the nearest labelled prim, else its rigid body. Tight and amodal 2D boxes and oriented 3D boxes come with OpenCV intrinsics and the camera's pose.

Result

On a wall 4 m away, depth is within 2 mm wherever the wall shows, and back-projected pixels land on the wall plane within 2 mm. Storm and Turing RT agree pixel for pixel on the masks, with 0.01 mm median depth difference.

A 640×480 frame with every output takes 36 ms; with 3×3 antialiasing, 73 ms. The presets' rates follow these measurements.

The captures show the editor's camera preview switching between outputs on the UR10 workcell. A follow-up commit stopped box names from piling up on the robot's links. Frames are best effort in the editor; exact-rate capture belongs to the synthetic-data pipeline, which is next.

---

背景

做完 IMU 和 LiDAR，传感器阶段还需要相机：不只是画面，还有机器人学习要用的真值，比如深度、分割、检测框，而且要和画面严格对齐。Turing Sim 通过 Hydra 用 Storm（光栅化）或 Turing RT（光线追踪）渲染，两者都没有把深度和 ID 缓冲交给 Python。

目标与难点

让相机传感器从一次渲染里得到全部输出，保证所有输出描述的是同一组像素，而且精度可以检验。同时要能在编辑器里运行，不影响视口画质。

解决思路与行动

1. 相机就是 USD 相机。 相机传感器是一个标准的 UsdGeom.Camera，带 turingSensor:type = "camera"。预设根据视场角和画面宽高比设置镜头。
2. 读回 AOV。 一个小的原生模块读取 Hydra 的颜色、深度、primId 和 instanceId 缓冲，并把 ID 解码成场景路径（包括实例代理）。Storm 和 Turing RT 都适用。
3. 单独的渲染进程。 整数 ID 必须在关闭多重采样的情况下渲染，而 Hydra 每个进程只读一次这个设置，所以相机在自己的 worker 进程里渲染，编辑器视口照样保留抗锯齿。另外发现 Storm 的视空间法线缓冲会把负分量截掉，所以法线改为由深度重建。
4. 从缓冲到真值。 RGB 按奇数倍超采样取平均；真值取每个像素的中心采样。深度换算成沿光轴的米数。物体取最近的带标签 prim，没有标签就取所属刚体。输出紧框和投影框两种 2D 框、有向 3D 框，以及 OpenCV 内参和相机位姿。

结果与下一步

对 4 米外的一面墙，只要墙出现的像素，深度误差都在 2 mm 以内，反投影回去的点也落在墙面 2 mm 以内。Storm 和 Turing RT 的分割掩码逐像素一致，深度中位差 0.01 mm。

带全部输出的 640×480 一帧要 36 ms；开 3×3 抗锯齿要 73 ms。预设的帧率就是按这些实测值定的。

截图展示的是编辑器里的相机预览在 UR10 工作站上切换各种输出。后续一个 commit 修好了机器人连杆上框标签互相重叠的问题。编辑器里的帧是尽力而为；按精确帧率采集属于合成数据流程，那是下一步。