DEV Community: Rerun

Generate Images with Depth Guided Stable Diffusion and Rerun

Rerun — Mon, 13 May 2024 15:33:36 +0000

Depth Guided Stable Diffusion enriches the image generation process by incorporating depth information, providing a unique way to control the spatial composition of generated images. This approach allows for more nuanced and layered creations, making it especially useful for scenes requiring a sense of three-dimensionality.

Logging and visualizing with Rerun

The visualizations in this example were created with the Rerun SDK, demonstrating the integration of depth information in the Stable Diffusion image generation process. Here is the code for generating the visualization in Rerun.

Prompt

Visualizing the prompt and negative prompt

rr.log("prompt/text", rr.TextLog(prompt))
rr.log("prompt/text_negative", rr.TextLog(negative_prompt))

Text

Visualizing the text input ids, the text attention mask and the unconditional input ids

rr.log("prompt/text_input/ids", rr.BarChart(text_input_ids))
rr.log("prompt/text_input/attention_mask", rr.BarChart(text_inputs.attention_mask))
rr.log("prompt/uncond_input/ids", rr.Tensor(uncond_input.input_ids))

Text embeddings

Visualizing the text embeddings. The text embeddings are generated in response to the specific prompts used while the unconditional text embeddings represent a neutral or baseline state without specific input conditions.

rr.log("prompt/text_embeddings", rr.Tensor(text_embeddings))
rr.log("prompt/uncond_embeddings", rr.Tensor(uncond_embeddings))

Depth map

Visualizing the pixel values of the depth estimation, estimated depth image, interpolated depth image and normalized depth image

rr.log("depth/input_preprocessed", rr.Tensor(pixel_values))
rr.log("depth/estimated", rr.DepthImage(depth_map))
rr.log("depth/interpolated", rr.DepthImage(depth_map))
rr.log("depth/normalized", rr.DepthImage(depth_map))

Latents

Log the latents, the representation of the images in the format used by the diffusion model.

rr.log("diffusion/latents", rr.Tensor(latents, dim_names=["b", "c", "h", "w"]))

Denoising loop

For each step in the denoising loop we set a time sequence with step and timestep and log the latent model input, noise predictions, latents and image. This make is possible for us to see all denoising steps in the Rerun viewer.

rr.set_time_sequence("step", i)
rr.set_time_sequence("timestep", t)
rr.log("diffusion/latent_model_input", rr.Tensor(latent_model_input))
rr.log("diffusion/noise_pred", rr.Tensor(noise_pred, dim_names=["b", "c", "h", "w"]))
rr.log("diffusion/latents", rr.Tensor(latents, dim_names=["b", "c", "h", "w"]))
rr.log("image/diffused", rr.Image(image))

Diffused image

Finally we log the diffused image generated by the model.

rr.log("image/diffused", rr.Image(image_8))

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42))

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to Visualize a recording from the NYUD dataset with RGB and Depth channels

Rerun — Thu, 09 May 2024 17:31:49 +0000

Try it in browser Source Code Explore Other Examples

Visualizes an example recording from the NYUD dataset with RGB and Depth channels.

The dataset, known as the NYU Depth V2 dataset, consists of synchronized pairs of RGB and depth frames recorded by the Microsoft Kinect in various indoor scenes. This example visualizes one scene of this dataset, and offers a rich source of data for object recognition, scene understanding, depth estimation, and more.

Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code:

Timelines

All data logged using Rerun in the following sections is connected to a specific time. Rerun assigns a timestamp to each piece of logged data, and these timestamps are associated with a timeline.

rr.set_time_seconds("time", time.timestamp())

Image

The example image is logged as Image to the world/camera/image/rgb entity.

rr.log("world/camera/image/rgb", rr.Image(img_rgb).compress(jpeg_quality=95))

Depth image

Pinhole camera is utilized for achieving a 3D view and camera perspective through the use of the Pinhole.

rr.log(
    "world/camera/image",
    rr.Pinhole(
        resolution=[img_depth.shape[1], img_depth.shape[0]],
        focal_length=0.7 * img_depth.shape[1],
    ),
)

Then, the depth image is logged as an DepthImage to the world/camera/image/depth entity.

rr.log("world/camera/image/depth", rr.DepthImage(img_depth, meter=DEPTH_IMAGE_SCALING))

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to Visualize LiDAR Data

Rerun — Wed, 08 May 2024 17:06:57 +0000

Source Code Explore Other Examples

This example demonstrates the ability to read and visualize LiDAR data from the nuScenes dataset, which is a public large-scale dataset specifically designed for autonomous driving. The scenes in this dataset encompass data collected from a comprehensive suite of sensors on autonomous vehicles, including 6 cameras, 1 LIDAR, 5 RADAR, GPS and IMU sensors.

It's important to note that in this example, only the LiDAR data is visualized. For a more extensive example including other sensors and annotations check out the nuScenes example.

Logging and visualizing with Rerun

The visualization in this example was created with just the following lines.

rr.set_time_seconds("timestamp", sample_data["timestamp"] * 1e-6) # Setting the time
rr.log("world/lidar", rr.Points3D(points, colors=point_colors)) # Log the 3D data

When logging data to Rerun, it's possible to associate it with specific time by using the Rerun's timelines. In the following code, we first establish the desired time frame and then proceed to log the 3D data points.

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42))

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

Use Hugging Face's ControlNet

Rerun — Tue, 07 May 2024 20:32:45 +0000

Source Code Explore Other Examples

Hugging Face's ControlNet allows to condition Stable Diffusion on various modalities. In this example we condition on edges detected by the Canny edge detector to keep our shape intact while generating an image with Stable Diffusion. This generates a new image based on the text prompt, but that still retains the structure of the control image. We visualize the whole generation process with Rerun.

Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code.

Images

rr.log("input/raw", rr.Image(image), timeless=True)
rr.log("input/canny", rr.Image(canny_image), timeless=True)

The input image and control canny_image are marked as timeless and logged in rerun.

Timeless entities belong to all timelines (existing ones, and ones not yet created) and are shown leftmost in the time panel in the viewer. This is useful for entities that aren't part of normal data capture, but set the scene for how they are shown.

This designation ensures their constant availability across all timelines in Rerun, aiding in consistent comparison and documentation.

Prompts

rr.log("positive_prompt", rr.TextDocument(prompt), timeless=True)
rr.log("negative_prompt", rr.TextDocument(negative_prompt), timeless=True)

The positive and negative prompt used for generation is logged to Rerun.

Custom diffusion step callback

We use a custom callback function for ControlNet that logs the output and the latent values at each timestep, which makes it possible for us to view all timesteps of the generation in Rerun.

def controlnet_callback(
    iteration: int, timestep: float, latents: torch.Tensor, pipeline: StableDiffusionXLControlNetPipeline
) -> None:
    rr.set_time_sequence("iteration", iteration)
    rr.set_time_seconds("timestep", timestep)
    rr.log("output", rr.Image(image))
    rr.log("latent", rr.Tensor(latents.squeeze(), dim_names=["channel", "height", "width"]))

Output image

Finally, we log the output image generated by ControlNet.

rr.log("output", rr.Image(images))

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to Visualize Live-Streaming Frames from Intel RealSense Depth Sensor

Rerun — Mon, 06 May 2024 14:25:42 +0000

Source Code Explore Other Examples

This example requires a connected RealSense depth sensor.

The Intel RealSense depth sensor can stream live depth and color data. To visualize this data output, we utilized Rerun.

Logging and visualizing with Rerun

The RealSense sensor captures data in both RGB and depth formats, which are logged using the Image and DepthImage archetypes, respectively. Additionally, to provide a 3D view, the visualization includes a pinhole camera using the Pinhole and Transform3D archetypes.

The visualization in this example were created with the following Rerun code.

RDF

For visualize the data as RDF:

rr.log("realsense", rr.ViewCoordinates.RDF, timeless=True)

Image

First, the pinhole camera is set using the Pinhole and Transform3D archetypes. Then, the images captured by the RealSense sensor are logged as an Image object, and they're associated with the time they were taken.

rgb_from_depth = depth_profile.get_extrinsics_to(rgb_profile)
rr.log(
    "realsense/rgb",
    rr.Transform3D(
        translation=rgb_from_depth.translation,
        mat3x3=np.reshape(rgb_from_depth.rotation, (3, 3)),
        from_parent=True,
    ),
    timeless=True,
)

rr.log(
    "realsense/rgb/image",
    rr.Pinhole(
        resolution=[rgb_intr.width, rgb_intr.height],
        focal_length=[rgb_intr.fx, rgb_intr.fy],
        principal_point=[rgb_intr.ppx, rgb_intr.ppy],
    ),
    timeless=True,
)

rr.set_time_sequence("frame_nr", frame_nr)
rr.log("realsense/rgb/image", rr.Image(color_image))

Depth image

Just like the RGB images, the RealSense sensor also captures depth data. The depth images are logged as DepthImage objects and are linked with the time they were captured.

rr.log(
    "realsense/depth/image",
    rr.Pinhole(
        resolution=[depth_intr.width, depth_intr.height],
        focal_length=[depth_intr.fx, depth_intr.fy],
        principal_point=[depth_intr.ppx, depth_intr.ppy],
    ),
    timeless=True,
)

rr.set_time_sequence("frame_nr", frame_nr)
rr.log("realsense/depth/image", rr.DepthImage(depth_image, meter=1.0 / depth_units))

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to Visualize Autonomous Driving Dataset (nuScenes)

Rerun — Sun, 05 May 2024 14:18:38 +0000

Try it in browser Source Code Explore Other Examples

This example demonstrates the ability to read and visualize scenes from the nuScenes dataset, which is a public large-scale dataset specifically designed for autonomous driving. The scenes in this dataset encompass data collected from a comprehensive suite of sensors on autonomous vehicles. These include 6 cameras, 1 LIDAR, 5 RADAR, GPS and IMU sensors. Consequently, the dataset provides information about the vehicle's pose, the images captured, the recorded sensor data and the results of object detection at any given moment.

Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code:

Sensor calibration

First, pinhole cameras and sensor poses are initialized to offer a 3D view and camera perspective. This is achieved using the Pinhole and Transform3D archetypes.

rr.log(
    f"world/ego_vehicle/{sensor_name}",
    rr.Transform3D(
        translation=calibrated_sensor["translation"],
        rotation=rr.Quaternion(xyzw=rotation_xyzw),
        from_parent=False,
    ),
    timeless=True,
)

rr.log(
    f"world/ego_vehicle/{sensor_name}",
    rr.Pinhole(
        image_from_camera=calibrated_sensor["camera_intrinsic"],
        width=sample_data["width"],
        height=sample_data["height"],
    ),
    timeless=True,
)

Timelines

All data logged using Rerun in the following sections is initially connected to a specific time. Rerun assigns a timestamp to each piece of logged data, and these timestamps are associated with timelines.

rr.set_time_seconds("timestamp", sample_data["timestamp"] * 1e-6)

Vehicle pose

As the vehicle is moving, its pose needs to be updated. Consequently, the positions of pinhole cameras and sensors must also be adjusted using Transform3D.

rr.log(
    "world/ego_vehicle",
    rr.Transform3D(
        translation=ego_pose["translation"],
        rotation=rr.Quaternion(xyzw=rotation_xyzw),
        from_parent=False,
    ),
)

LiDAR data

LiDAR data is logged as Points3D archetype.

rr.log(f"world/ego_vehicle/{sensor_name}", rr.Points3D(points, colors=point_colors))

Camera data

Camera data is logged as encoded images using ImageEncoded.

rr.log(f"world/ego_vehicle/{sensor_name}", rr.ImageEncoded(path=data_file_path))

Radar data

Radar data is logged similar to LiDAR data, as Points3D.

rr.log(f"world/ego_vehicle/{sensor_name}", rr.Points3D(points, colors=point_colors))

Annotations

Annotations are logged as Boxes3D, containing details such as object positions, sizes, and rotation.

rr.log("world/anns", rr.Boxes3D(sizes=sizes, centers=centers, rotations=rotations, class_ids=class_ids))

Setting up the default blueprint

The default blueprint for this example is created by the following code:

sensor_space_views = [
    rrb.Spatial2DView(
        name=sensor_name,
        origin=f"world/ego_vehicle/{sensor_name}",
    )
    for sensor_name in nuscene_sensor_names(nusc, args.scene_name)
]
blueprint = rrb.Vertical(
    rrb.Spatial3DView(name="3D", origin="world"),
    rrb.Grid(*sensor_space_views),
    row_shares=[3, 2],
)

We programmatically create one view per sensor and arrange them in a grid layout, which is convenient when the number of views can significantly vary from dataset to dataset. This code also showcases the row_shares argument for vertical containers: it can be used to assign a relative size to each of the container's children. A similar column_shares argument exists for horizontal containers, while grid containers accept both.

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to Visualize RRT* Algorithm

Rerun — Sat, 04 May 2024 14:07:56 +0000

Try it in browser Source Code Explore Other Examples

This example visualizes the path finding algorithm RRT* in a simple environment.

The algorithm finds a path between two points by randomly expanding a tree from the start point. After it has added a random edge to the tree it looks at nearby nodes to check if it's faster to reach them through this new edge instead, and if so it changes the parent of these nodes. This ensures that the algorithm will converge to the optimal path given enough time.

A detailed explanation can be found in the original paper Karaman, S. Frazzoli, S. 2011. "Sampling-based algorithms for optimal motion planning". or in this medium article.

Logging and visualizing with Rerun

All points are logged using the Points2D archetype, while the lines are logged using the LineStrips2D.

The visualizations in this example were created with the following Rerun code:

Map

Starting point

rr.log("map/start", rr.Points2D([start_point], radii=0.02, colors=[[255, 255, 255, 255]]))

Destination point

rr.log("map/destination", rr.Points2D([end_point], radii=0.02, colors=[[255, 255, 0, 255]]))

Obstacles

rr.log("map/obstacles", rr.LineStrips2D(self.obstacles))

RRT tree

Edges

rr.log("map/tree/edges", rr.LineStrips2D(tree.segments(), radii=0.0005, colors=[0, 0, 255, 128]))

New edges

rr.log("map/new/new_edge", rr.LineStrips2D([(closest_node.pos, new_point)], colors=[color], radii=0.001))

Vertices

rr.log("map/tree/vertices", rr.Points2D([node.pos for node in tree], radii=0.002), rr.AnyValues(cost=[float(node.cost) for node in tree]))

Close nodes

rr.log("map/new/close_nodes", rr.Points2D([node.pos for node in close_nodes]))

Closest node

rr.log("map/new/closest_node", rr.Points2D([closest_node.pos], radii=0.008))

Random points

rr.log("map/new/random_point", rr.Points2D([random_point], radii=0.008))

New points

rr.log("map/new/new_point", rr.Points2D([new_point], radii=0.008))

Path

rr.log("map/path", rr.LineStrips2D(segments, radii=0.002, colors=[0, 255, 255, 255]))

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to Visualize VRS Files

Rerun — Fri, 03 May 2024 15:04:04 +0000

Source Code Explore Other Examples

This C++ example demonstrates how to visualize VRS files with Rerun. VRS is a file format optimized to record & playback streams of sensor data, such as images, audio samples, and any other discrete sensors (IMU, temperature, etc), stored in per-device streams of time-stamped records.

Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code:

3D arrows

void IMUPlayer::log_accelerometer(const std::array<float, 3>& accelMSec2) {
    _rec->log(_entity_path + "/accelerometer", rerun::Arrows3D::from_vectors({accelMSec2}));
    // … existing code for scalars …
}

Scalars

void IMUPlayer::log_accelerometer(const std::array<float, 3>& accelMSec2) {
    // … existing code for Arrows3D …
    _rec->log(_entity_path + "/accelerometer/x", rerun::Scalar(accelMSec2[0]));
    _rec->log(_entity_path + "/accelerometer/y", rerun::Scalar(accelMSec2[1]));
    _rec->log(_entity_path + "/accelerometer/z", rerun::Scalar(accelMSec2[2]));
}

void IMUPlayer::log_gyroscope(const std::array<float, 3>& gyroRadSec) {
    _rec->log(_entity_path + "/gyroscope/x", rerun::Scalar(gyroRadSec[0]));
    _rec->log(_entity_path + "/gyroscope/y", rerun::Scalar(gyroRadSec[1]));
    _rec->log(_entity_path + "/gyroscope/z", rerun::Scalar(gyroRadSec[2]));
}

void IMUPlayer::log_magnetometer(const std::array<float, 3>& magTesla) {
    _rec->log(_entity_path + "/magnetometer/x", rerun::Scalar(magTesla[0]));
    _rec->log(_entity_path + "/magnetometer/y", rerun::Scalar(magTesla[1]));
    _rec->log(_entity_path + "/magnetometer/z", rerun::Scalar(magTesla[2]));
}

Images

_rec->log(
    _entity_path,
    rerun::Image({
    frame->getHeight(),
    frame->getWidth(),
    frame->getSpec().getChannelCountPerPixel()},
    frame->getBuffer()
    )
);

Text document

_rec->log_timeless(_entity_path + "/configuration", rerun::TextDocument(layout_str));

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to Visualize Objectron Dataset

Rerun — Thu, 02 May 2024 15:20:28 +0000

Source Code Explore Other Examples

Visualize the Google Research Objectron dataset including camera poses, sparse point-clouds and surfaces characterization.

This example visualizes the Objectron database, a rich collection of object-centric video clips accompanied by AR session metadata. With high-resolution images, object pose, camera pose, point-cloud, and surface plane information available for each sample, the visualization offers a comprehensive view of the object from various angles. Additionally, the dataset provides manually annotated 3D bounding boxes, enabling precise object localization and orientation.

Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code:

Timelines

For each processed frame, all data sent to Rerun is associated with the two timelines time and frame_idx.

rr.set_time_sequence("frame", sample.index)
rr.set_time_seconds("time", sample.timestamp)

Video

Pinhole camera is utilized for achieving a 3D view and camera perspective through the use of the Pinhole and Transform3D archetypes.

rr.log(
        "world/camera",
        rr.Transform3D(translation=translation, rotation=rr.Quaternion(xyzw=rot.as_quat())),
)

rr.log(
    "world/camera",
    rr.Pinhole(
        resolution=[w, h],
        image_from_camera=intrinsics,
        camera_xyz=rr.ViewCoordinates.RDF,
    ),
)

The input video is logged as a sequence of ImageEncoded objects to the world/camera entity.

rr.log("world/camera", rr.ImageEncoded(path=sample.image_path))

Sparse point clouds

Sparse point clouds from ARFrame are logged as Points3D archetype to the world/points entity.

rr.log("world/points", rr.Points3D(positions, colors=[255, 255, 255, 255]))

Annotated bounding boxes

Bounding boxes annotated from ARFrame are logged as Boxes3D, containing details such as object position, sizes, center and rotation.

rr.log(
    f"world/annotations/box-{bbox.id}",
    rr.Boxes3D(
        half_sizes=0.5 * np.array(bbox.scale),
        centers=bbox.translation,
        rotations=rr.Quaternion(xyzw=rot.as_quat()),
        colors=[160, 230, 130, 255],
        labels=bbox.category,
    ),
    timeless=True,
)

Setting up the default blueprint

The default blueprint is configured with the following code:

blueprint = rrb.Horizontal(
    rrb.Spatial3DView(origin="/world", name="World"),
    rrb.Spatial2DView(origin="/world/camera", name="Camera", contents=["/world/**"]),
)

In particular, we want to reproject the points and the 3D annotation box in the 2D camera view corresponding to the pinhole logged at "/world/camera". This is achieved by setting the view's contents to the entire "/world/**" subtree, which include both the pinhole transform and the image data, as well as the point cloud and the 3D annotation box.

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to Visualize Structure-from-Motion

Rerun — Wed, 01 May 2024 06:57:11 +0000

Try it in browser Source Code Explore Other Examples

COLMAP is a general-purpose Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline. In this example, a short video clip has been processed offline using the COLMAP pipeline. The processed data was then visualized using Rerun, which allowed for the visualization of individual camera frames, estimation of camera poses, and creation of point clouds over time. By using COLMAP in combination with Rerun, a highly-detailed reconstruction of the scene depicted in the video was generated.

Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code:

Timelines

All data logged using Rerun in the following sections is connected to a specific frame. Rerun assigns a frame id to each piece of logged data, and these frame ids are associated with a timeline.



rr.set_time_sequence("frame", frame_idx)

Images

The images are logged through the Image to the camera/image entity.



rr.log("camera/image", rr.Image(rgb).compress(jpeg_quality=75))

Cameras

The images stem from pinhole cameras located in the 3D world. To visualize the images in 3D, the pinhole projection has to be logged and the camera pose (this is often referred to as the intrinsics and extrinsics of the camera, respectively).

The Pinhole is logged to the camera/image entity and defines the intrinsics of the camera. This defines how to go from the 3D camera frame to the 2D image plane. The extrinsics are logged as an Transform3D to the camera entity.



rr.log("camera", rr.Transform3D(translation=image.tvec, rotation=rr.Quaternion(xyzw=quat_xyzw), from_parent=True))

rr.log(
    "camera/image",
    rr.Pinhole(
        resolution=[camera.width, camera.height],
        focal_length=camera.params[:2],
        principal_point=camera.params[2:],
    ),
)

2D points

The 2D image points that are used to triangulate the 3D points are visualized by logging as Points2D to the camera/image/keypoints entity. Note that these keypoints are a child of the camera/image entity, since the points should show in the image plane.



rr.log("camera/image/keypoints", rr.Points2D(visible_xys, colors=[34, 138, 167]))

3D points

The colored 3D points were added to the visualization by logging the Points3D archetype to the points entity.



rr.log("points", rr.Points3D(points, colors=point_colors), rr.AnyValues(error=point_errors))

Reprojection error

For each image a Scalar archetype containing the average reprojection error of the keypoints is logged to the plot/avg_reproj_err entity.



rr.log("plot/avg_reproj_err", rr.Scalar(np.mean(point_errors)))

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to Visualise MediaPipe’s Face and Face Landmark Detection in 2D and 3D with Rerun

Rerun — Mon, 29 Apr 2024 15:57:33 +0000

The face and face landmark detection technology aims to give the ability of the devices to interpret face movements and facial expressions as commands or inputs. At the core of this technology, a pre-trained machine-learning model analyses the visual input, locates face and identifies face landmarks and blendshape scores (coefficients representing facial expression). Human-Computer Interaction, Robotics, Gaming, and Augmented Reality are among the fields where this technology shows significant promise for applications.

In this example, the MediaPipe Face and Face Landmark Detection solutions were utilized to detect human face, detect face landmarks and identify facial expressions. Rerun was employed to visualize the output of the Mediapipe solution over time to make it easy to analyze the behavior.

Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code.

Timelines

For each processed video frame, all data sent to Rerun is associated with the two timelines time and frame_idx.



rr.set_time_seconds("time", bgr_frame.time)
rr.set_time_sequence("frame_idx", bgr_frame.idx)

Video

The input video is logged as a sequence of Image objects to the 'Video' entity.



rr.log(
    "video/image",
    rr.Image(frame).compress(jpeg_quality=75)
)

Face Landmark Points

Logging the face landmarks involves specifying connections between the points, extracting face landmark points and logging them to the Rerun SDK. The 2D points are visualized over the video/image for a better understanding and visualization of the face. The 3D points allows the creation of a 3D model of the face reconstruction for a more comprehensive representation of the face.

The 2D and 3D points are logged through a combination of two archetypes. First, a timeless ClassDescription is logged, that contains the information which maps keypoint ids to labels and how to connect the keypoints. Defining these connections automatically renders lines between them. Second, the actual keypoint positions are logged in 2D and 3D as Points2D and Points3D archetypes, respectively.

Label mapping and keypoint connections

An annotation context is logged with one class ID assigned per facial feature. The class description includes the connections between corresponding keypoints extracted from the MediaPipe face mesh solution. A class ID array is generated to match the class IDs in the annotation context with keypoint indices (to be utilized as the class_ids argument to rr.log).



# Initialize a list of facial feature classes from MediaPipe face mesh solution
classes = [
    mp.solutions.face_mesh.FACEMESH_LIPS,
    mp.solutions.face_mesh.FACEMESH_LEFT_EYE,
    mp.solutions.face_mesh.FACEMESH_LEFT_IRIS,
    mp.solutions.face_mesh.FACEMESH_LEFT_EYEBROW,
    mp.solutions.face_mesh.FACEMESH_RIGHT_EYE,
    mp.solutions.face_mesh.FACEMESH_RIGHT_EYEBROW,
    mp.solutions.face_mesh.FACEMESH_RIGHT_IRIS,
    mp.solutions.face_mesh.FACEMESH_FACE_OVAL,
    mp.solutions.face_mesh.FACEMESH_NOSE,
]

# Initialize class descriptions and class IDs array
self._class_ids = [0] * mp.solutions.face_mesh.FACEMESH_NUM_LANDMARKS_WITH_IRISES
class_descriptions = []

# Loop through each facial feature class
for i, klass in enumerate(classes):
    # MediaPipe only provides connections for class, not actual class per keypoint. So we have to extract the
    # classes from the connections.
    ids = set()
    for connection in klass:
        ids.add(connection[0])
        ids.add(connection[1])

    for id_ in ids:
        self._class_ids[id_] = i

    # Append class description with class ID and keypoint connections
    class_descriptions.append(
        rr.ClassDescription(
            info=rr.AnnotationInfo(id=i),
            keypoint_connections=klass,
        )
    )

# Log annotation context for video/landmarker and reconstruction entities
rr.log("video/landmarker", rr.AnnotationContext(class_descriptions), timeless=True)
rr.log("reconstruction", rr.AnnotationContext(class_descriptions), timeless=True)

rr.log("reconstruction", rr.ViewCoordinates.RDF, timeless=True) # properly align the 3D face in the viewer
With the below annotation, the keypoints will be connected with lines to enhance visibility in the video/detector entity.


rr.log(
    "video/detector",
    rr.ClassDescription(
        info=rr.AnnotationInfo(id=0), keypoint_connections=[(0, 1), (1, 2), (2, 0), (2, 3), (0, 4), (1, 5)]
    ),
    timeless=True,
)

Bounding box



rr.log(
    f"video/detector/faces/{i}/bbox",
    rr.Boxes2D(
        array=[bbox.origin_x, bbox.origin_y, bbox.width, bbox.height], array_format=rr.Box2DFormat.XYWH
    ),
    rr.AnyValues(index=index, score=score),
)

2D points



rr.log(
    f"video/detector/faces/{i}/keypoints",
    rr.Points2D(pts, radii=3, keypoint_ids=list(range(6)))
)

rr.log(
    f"video/landmarker/faces/{i}/landmarks",
    rr.Points2D(pts, radii=3, keypoint_ids=keypoint_ids, class_ids=self._class_ids),
)

3D points



rr.log(
    f"reconstruction/faces/{i}",
    rr.Points3D(
        [(lm.x, lm.y, lm.z) for lm in landmark],
        keypoint_ids=keypoint_ids,
        class_ids=self._class_ids,
    ),
)

Scalar

Blendshapes are essentially predefined facial expressions or configurations that can be detected by the face landmark detection model. Each blendshape typically corresponds to a specific facial movement or expression, such as blinking, squinting, smiling, etc.

The blendshapes are logged along with their corresponding scores.



for blendshape in blendshapes:
    if blendshape.category_name in BLENDSHAPES_CATEGORIES:
        rr.log(f"blendshapes/{i}/{blendshape.category_name}", rr.Scalar(blendshape.score))

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub

How to visualise MediaPipe’s Hand Tracking in 2D and 3D with Rerun

Rerun — Sun, 28 Apr 2024 13:34:38 +0000

Source Code Explore Other Examples

The hand tracking and gesture recognition technology aims to give the ability of the devices to interpret hand movements and gestures as commands or inputs. At the core of this technology, a pre-trained machine-learning model analyses the visual input and identifies hand landmarks and hand gestures. The real applications of such technology vary, as hand movements and gestures can be used to control smart devices. Human-Computer Interaction, Robotics, Gaming, and Augmented Reality are a few of the fields where the potential applications of this technology appear most promising.

In this example, the MediaPipe Gesture and Hand Landmark Detection solutions were utilized to detect and track hand landmarks and recognize gestures. Rerun was employed to visualize the output of the Mediapipe solution over time to make it easy to analyze the behavior.

Logging and visualizing with Rerun

The visualizations in this example were created with the following Rerun code.

Timelines

For each processed video frame, all data sent to Rerun is associated with the two timelines time and frame_idx.



rr.set_time_sequence("frame_nr", frame_idx)
rr.set_time_nanos("frame_time", frame_time_nano)

Video

The input video is logged as a sequence of Image objects to the Media/Video entity.



rr.log(
    "Media/Video",
    rr.Image(frame).compress(jpeg_quality=75)
)

Hand landmark points

Logging the hand landmarks involves specifying connections between the points, extracting pose landmark points and logging them to the Rerun SDK. The 2D points are visualized over the video and at a separate entity. Meanwhile, the 3D points allows the creation of a 3D model of the hand for a more comprehensive representation of the hand landmarks.

The 2D and 3D points are logged through a combination of two archetypes. For the 2D points, the Points2D and LineStrips2D archetypes are utilized. These archetypes help visualize the points and connect them with lines, respectively. As for the 3D points, the logging process involves two steps. First, a timeless ClassDescription is logged, that contains the information which maps keypoint ids to labels and how to connect the keypoints. Defining these connections automatically renders lines between them. Mediapipe provides the HAND_CONNECTIONS variable which contains the list of (from, to) landmark indices that define the connections. Second, the actual keypoint positions are logged in 3D Points3D archetype.

Label mapping and keypoint connections



rr.log(
    "/",
    rr.AnnotationContext(
        rr.ClassDescription(
            info=rr.AnnotationInfo(id=0, label="Hand3D"),
            keypoint_connections=mp.solutions.hands.HAND_CONNECTIONS,
        )
    ),
    timeless=True,
)

rr.log("Hand3D", rr.ViewCoordinates.LEFT_HAND_Y_DOWN, timeless=True)

2D points



# Log points to the image and Hand Entity
for log_key in ["Media/Points", "Hand/Points"]:
    rr.log(
      log_key,
      rr.Points2D(points, radii=10, colors=[255, 0, 0])
    )

# Log connections to the image and Hand Entity [128, 128, 128]
for log_key in ["Media/Connections", "Hand/Connections"]:
    rr.log(
      log_key,
      rr.LineStrips2D(np.stack((points1, points2), axis=1), colors=[255, 165, 0])
    )

3D points



rr.log(
    "Hand3D/Points",
    rr.Points3D(
        landmark_positions_3d,
        radii=20,
        class_ids=0,
        keypoint_ids=[i for i in range(len(landmark_positions_3d))],
    ),
)

Detection

To showcase gesture recognition, an image of the corresponding gesture emoji is displayed within a TextDocument under the Detection entity.



# Log the detection by using the appropriate image
rr.log(
    "Detection",
    rr.TextDocument(f"![Image]({GESTURE_URL + gesture_pic})".strip(), media_type=rr.MediaType.MARKDOWN),
)

Join us on Github

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.

Build time aware visualizations of multimodal data

Use the Rerun SDK (available for C++, Python and Rust) to log data like images, tensors, point clouds, and text. Logs are streamed to the Rerun Viewer for live visualization or to file for later use.

A short taste

import rerun as rr  # pip install rerun-sdk
rr.init("rerun_example_app")

rr.connect()  # Connect to a remote viewer
# rr.spawn()  # Spawn a child process with a viewer and connect
# rr.save("recording.rrd")  # Stream all logs to disk

# Associate subsequent data with 42 on the “frame” timeline
rr.set_time_sequence("frame", 42)

# Log colored 3D points to the entity at `path/to/points`
rr.log("path/to/points", rr.Points3D(positions, colors=colors

…

View on GitHub