DEV Community

Richard Minsoo Go
Richard Minsoo Go

Posted on

Perception and Sensor Fusion — Chapter 07: 3D Scene Understanding [EN]

Chapter 07. 3D Scene Understanding

07.01. 3D Scene Understanding Overview Geometry Semantics

3D scene understanding enables a robot to transform raw spatial measurements into an internal representation of the surrounding world that supports reasoning and action. Rather than treating sensor observations as isolated pixels or points, the system organizes them according to geometry, objects, surfaces, free space, and semantic meaning. This representation provides the bridge between low-level perception and higher-level navigation, manipulation, planning, and autonomous decision making.

The geometric component describes where structures exist and how they are arranged in three-dimensional space. Depth images, stereo vision, LiDAR point clouds, and reconstructed surfaces can provide information about distance, orientation, shape, volume, and spatial connectivity. Depending on the application, geometry may be represented as point clouds, voxels, meshes, occupancy structures, signed distance fields, or learned continuous representations, extending the spatial representations introduced earlier in the perception pipeline.

Geometry alone, however, does not explain what the observed structures mean. A robot may reconstruct a vertical planar surface accurately without knowing whether it corresponds to a wall, door, cabinet, vehicle, or another object. Semantic understanding associates geometric observations with meaningful categories, attributes, identities, and contextual information. Combining geometry and semantics therefore converts a metric reconstruction into a representation that can directly support task-oriented reasoning.

A useful 3D scene model can be viewed as several interconnected levels of abstraction. At the lowest level are measurements such as pixels, depth values, radar returns, and 3D points. These observations can be grouped into surfaces, segments, or spatial regions and subsequently associated with objects and semantic classes. At higher levels, relationships such as inside, on top of, adjacent to, connected to, or reachable from can describe how entities interact within the environment.

Coordinate systems are fundamental to this process because observations from different sensors and different times must refer to a consistent spatial frame. Camera, LiDAR, radar, IMU, robot-body, odometry, and map frames may coexist within one perception system. Accurate calibration and pose estimation allow measurements to be transformed between these frames. Errors in extrinsic calibration or localization can consequently distort reconstructed geometry and produce inconsistent semantic associations even when individual sensors perform correctly.

Three-dimensional understanding becomes particularly powerful when observations from multiple viewpoints are accumulated over time. A single camera image or LiDAR scan contains only a partial view because objects occlude one another and sensors have limited fields of view. Robot motion provides additional observations that can gradually reveal previously hidden regions. Temporal integration can therefore produce a more complete representation, while simultaneously requiring mechanisms for handling localization uncertainty, moving objects, and changes in the environment.

Semantic segmentation assigns category information to individual spatial elements, while instance reasoning separates different objects belonging to the same category. Panoptic representations combine these perspectives so that the robot can distinguish both background structures and countable objects. These concepts form the basis for subsequent 3D semantic and panoptic scene segmentation, followed by representations that explicitly encode relationships among entities in the environment.

Scene graphs provide a higher-level representation in which objects, regions, or places become nodes and their spatial or semantic relationships become edges. Instead of reasoning directly over millions of points or voxels, a robot can reason about statements such as a tool being located on a workbench or a container being beside a manipulator. This abstraction is especially valuable when perception must communicate with task planning, language-based reasoning, manipulation systems, or other symbolic and learned decision modules.

A complete scene representation must also distinguish static structure from dynamic content. Floors, walls, pillars, and fixed machinery may define persistent geometry, whereas humans, vehicles, doors, movable containers, and other robots can change position or state. Treating every observation as permanent produces corrupted maps and unreliable planning information. Modern scene understanding therefore combines spatial reconstruction with object tracking, temporal consistency, change detection, and mechanisms for updating previously stored knowledge.

Uncertainty is unavoidable because 3D scene understanding depends on noisy sensors, incomplete observations, imperfect calibration, occlusion, and learned models that can make incorrect predictions. A robust system should preserve confidence information instead of presenting every estimate as equally reliable. Geometric uncertainty may describe uncertain depth or pose, while semantic uncertainty can represent ambiguous class predictions. Planning modules can then account for uncertain regions rather than assuming that the perceived scene is perfectly known.

The representation selected for a robot depends strongly on the downstream task. Navigation often emphasizes free space, obstacles, traversability, and connectivity, whereas manipulation requires accurate local surfaces, object boundaries, poses, and contact-relevant geometry. Inspection may require dense reconstruction and change detection, while semantic navigation requires associations between spatial locations and meaningful entities. Consequently, no single representation is universally optimal; practical systems commonly maintain several complementary representations simultaneously.

Learned representations increasingly complement explicit geometric models. Neural networks can encode point clouds, sparse voxels, images, and multimodal sensor features into latent representations that capture both spatial structure and semantic context. Neural implicit representations and neural rendering techniques can model continuous scene properties, while 3D Gaussian representations can provide efficient reconstruction and rendering. These approaches extend conventional geometric mapping rather than eliminating the need for coordinate consistency and physical spatial reasoning.

Real-time robotics introduces an additional constraint: the scene model must remain computationally manageable while the robot moves. Dense 3D processing can consume substantial memory, bandwidth, and GPU resources, especially when large environments are continuously reconstructed. Sparse representations, hierarchical maps, region-of-interest processing, incremental updates, and hardware-accelerated inference are therefore important design strategies. The chapter structure consequently progresses from semantic understanding and scene graphs toward reconstruction, change detection, manipulation use, and edge deployment.

For autonomous robots, the ultimate purpose of 3D scene understanding is not reconstruction for its own sake but actionable understanding of the physical environment. A robot must determine where it can move, which entities matter, how objects relate to one another, what has changed, and which interactions are physically possible. Geometry establishes the spatial structure of the world, semantics gives that structure meaning, and temporal reasoning maintains consistency as both the robot and environment evolve.

This combination forms an important intermediate layer between multi-sensor fusion and higher-level spatial maps within the overall perception architecture. The preceding sensor-fusion stage integrates complementary measurements, while subsequent occupancy and semantic mapping organizes environmental knowledge for navigation and long-term operation. 3D scene understanding connects these stages by converting fused observations into structured geometric and semantic knowledge that autonomous systems can use for planning, interaction, and intelligent physical behavior.

07.02. 3D Semantic Segmentation Cylinder3D Mask3D [w/Code]

3D semantic segmentation assigns a semantic category to individual elements of a three-dimensional scene, allowing a robot to distinguish surfaces and structures such as floors, walls, vegetation, vehicles, furniture, and other meaningful regions. Unlike 3D object detection, which primarily predicts bounding volumes around objects, semantic segmentation produces dense spatial labels and therefore provides detailed information about the composition of the environment.

The input to a 3D semantic segmentation system commonly consists of LiDAR point clouds, RGB-D measurements, reconstructed point clouds, or voxelized scene representations. Each point may contain spatial coordinates together with attributes such as intensity, color, surface normal, or learned features. The segmentation network transforms these irregular observations into semantic predictions while preserving sufficient spatial detail for navigation, mapping, manipulation, and scene reasoning.

A fundamental difficulty is that raw point clouds are sparse, irregular, and nonuniformly distributed. LiDAR measurements become increasingly sparse with distance, while indoor RGB-D scans contain occlusions and missing surfaces. Conventional dense 3D convolution over a uniform voxel grid can waste substantial computation on empty space. Practical architectures therefore employ point-based processing, sparse convolutions, projection-based representations, or combinations of these methods to obtain computationally efficient spatial features.

Cylinder3D addresses large-scale LiDAR semantic segmentation by organizing points using cylindrical spatial partitioning rather than relying only on conventional Cartesian voxelization. This representation is well matched to rotating LiDAR geometry because angular sampling and radial distance strongly influence point density. Cylindrical partitioning can allocate spatial cells more effectively across near and distant regions, reducing some of the representation imbalance produced by uniform Cartesian grids.

In a cylindrical representation, the Cartesian coordinates of a point are transformed into radial, angular, and vertical components. Points are then assigned to cylindrical cells that provide a structured representation suitable for efficient feature processing. This transformation does not remove the original geometry; instead, it changes how space is discretized so that the network can better accommodate the characteristic distribution of outdoor LiDAR measurements while maintaining three-dimensional spatial relationships.

Cylinder3D combines this representation with three-dimensional feature extraction to learn contextual information across neighboring regions. Local geometric patterns can indicate surfaces such as roads, sidewalks, walls, poles, vegetation, or vehicles, while deeper network features capture broader spatial context. Semantic predictions are ultimately associated with the original points, producing a densely labeled point cloud that can be integrated with downstream mapping and autonomous navigation modules.

One important principle illustrated by Cylinder3D is that the representation should reflect the physical characteristics of the sensing system. A theoretically simple spatial discretization is not necessarily optimal when sensor density varies strongly across the scene. Sensor-aware representations can improve the effective use of memory and computation while retaining information in regions that are important for recognition. This principle extends beyond LiDAR segmentation to many robotic perception pipelines.

Mask3D approaches 3D scene segmentation from a different perspective by representing semantic entities through learned masks. Instead of treating segmentation purely as independent classification of every point or voxel, mask-based architectures predict groups of spatial elements associated with semantic or instance-level concepts. This formulation is particularly useful for complex indoor scenes where multiple objects of the same category must be distinguished while maintaining accurate object boundaries.

The mask-based approach typically extracts features from a sparse three-dimensional representation and uses learned queries to identify meaningful regions within the scene. Each query can generate a mask describing which spatial elements belong together and a corresponding semantic prediction describing what that region represents. Attention-based processing allows the model to associate global scene context with local geometry, helping separate nearby objects that may have similar appearance or geometric structure.

Cylinder3D and Mask3D therefore illustrate two complementary directions in modern 3D segmentation. Cylinder3D emphasizes an efficient sensor-aware spatial representation for point-wise semantic understanding, particularly for large LiDAR scenes. Mask3D emphasizes object- and region-oriented reasoning using mask prediction and learned queries. The choice between these approaches depends on the sensing configuration, environment scale, required semantic granularity, computational budget, and downstream robotic task.

Semantic segmentation should also be distinguished from instance and panoptic segmentation. Semantic segmentation assigns categories without necessarily distinguishing separate objects of the same class. Instance segmentation identifies individual object instances, while panoptic segmentation combines semantic labeling of background regions with instance-level separation of countable objects. These representations form a natural progression toward richer 3D scene understanding and scene graph construction within the perception architecture.

Training a 3D segmentation model requires accurately labeled spatial data, but producing point-level or voxel-level annotations is expensive. Labels must remain consistent despite occlusion, sparse measurements, ambiguous boundaries, and differences between sensor viewpoints. Data augmentation techniques such as rotation, scaling, point perturbation, spatial cropping, and scene mixing can improve robustness. Synthetic data and transferred representations can further reduce dependence on manually annotated real-world scenes.

Evaluation commonly measures how accurately predicted semantic regions overlap with ground-truth labels. Intersection over Union, often summarized as mean IoU across classes, is widely used because it penalizes both missing regions and incorrect assignments. Overall accuracy alone can be misleading when dominant categories such as road or floor contain far more points than smaller objects. Per-class evaluation is therefore important for identifying failures affecting safety-critical or task-relevant categories.

Temporal consistency becomes important when segmentation is deployed on a moving robot. Independent predictions from consecutive frames may fluctuate because of sensor noise, changing viewpoints, occlusion, or model uncertainty. Associating observations through robot motion, object tracking, or temporal feature fusion can stabilize semantic estimates. Persistent semantic information can then be accumulated into a map instead of repeatedly interpreting every observation as an unrelated scene.

For robotic operation, segmentation uncertainty should propagate into downstream decisions. A region classified confidently as traversable floor may support navigation, whereas an ambiguous region near an obstacle boundary may require conservative treatment. Similarly, uncertain object masks should not automatically become permanent semantic landmarks. Confidence thresholds, unknown classes, temporal confirmation, and sensor redundancy provide mechanisms for preventing uncertain predictions from becoming unsafe assumptions.

Real-time deployment introduces significant memory and latency constraints because 3D networks process large spatial datasets. Sparse tensors, reduced-resolution voxel structures, mixed-precision inference, region filtering, model pruning, and GPU acceleration can reduce computational cost. The required operating point depends on the robot: an indoor manipulator may prioritize fine spatial detail, whereas a moving AMR may favor predictable latency and sufficient segmentation range for safe motion.

The output of 3D semantic segmentation becomes more valuable when combined with geometry, localization, and temporal mapping. Semantic labels can transform an unlabeled point cloud into an interpretable spatial model containing traversable surfaces, structural elements, movable objects, and task-relevant regions. These results can feed occupancy maps, semantic maps, manipulation planning, change detection, and higher-level scene graphs, linking dense perception with autonomous reasoning and action.

Within the broader 3D scene-understanding pipeline, Cylinder3D and Mask3D demonstrate that semantic interpretation is inseparable from representation design. Effective systems must jointly consider sensor geometry, spatial discretization, learned features, semantic categories, computational constraints, and downstream requirements. The resulting segmentation is not merely a visualization layer; it is structured environmental knowledge that allows a physical robot to reason about what occupies three-dimensional space and how that information should influence its behavior.

07.03. Panoptic 3D Scene Segmentation [w/Code]

Panoptic 3D scene segmentation provides a unified representation in which every observable element of a three-dimensional environment receives both semantic meaning and, when appropriate, an individual instance identity. It combines semantic segmentation, which determines what category a spatial element belongs to, with instance segmentation, which separates distinct objects belonging to the same category. The result is a structured interpretation of the complete scene rather than a collection of independently detected objects.

The central distinction in panoptic segmentation is between "stuff" and "things." Stuff classes describe regions that do not normally require individual identities, such as road, floor, wall, terrain, or vegetation. Thing classes represent countable entities such as vehicles, pedestrians, chairs, containers, or robots. A panoptic system assigns semantic labels to both groups while additionally assigning unique instance identifiers to individual thing objects.

In a 3D perception pipeline, panoptic predictions can be generated from LiDAR point clouds, RGB-D measurements, multi-view images projected into 3D, or reconstructed scene geometry. Each point, voxel, or other spatial primitive ultimately receives a panoptic label. This label usually combines a semantic class identifier with an instance identifier, allowing downstream modules to determine both that a region represents a vehicle and that it represents a particular vehicle within the observed scene.

Panoptic 3D segmentation extends the semantic segmentation methods introduced by architectures such as Cylinder3D and Mask3D. Semantic feature extraction provides category-level understanding, while instance-oriented mechanisms identify coherent objects within those semantic regions. Mask-based approaches are especially suitable because learned queries can generate object masks and class predictions jointly, creating a natural path from semantic and instance reasoning toward a unified panoptic representation.

A typical architecture begins by encoding the input geometry into learned 3D features. Sparse voxel convolution, point-based networks, projection-based encoders, or transformer architectures can be used depending on the sensor and computational constraints. These features contain local geometric information and broader contextual information. Prediction heads or learned queries then estimate semantic classes, instance masks, object confidence, and other information required to construct the final panoptic scene.

Separating instances in three dimensions is challenging when objects are close together, partially occluded, sparsely measured, or geometrically similar. Two adjacent chairs, parked vehicles, storage boxes, or pedestrians may produce spatial observations that appear connected or overlap from particular viewpoints. Effective models therefore combine geometric discontinuities, semantic features, contextual relationships, and learned object-level representations to determine whether observations belong to one object or several different instances.

Instance association becomes even more difficult for sparse outdoor LiDAR data. Nearby objects may contain many measurements, whereas distant objects may be represented by only a small number of points. Small objects such as poles, signs, or pedestrians can therefore be difficult to separate reliably. Sensor-aware representations, multi-scale features, temporal accumulation, and camera-LiDAR fusion can provide additional evidence when a single 3D observation does not contain enough information for stable instance separation.

The panoptic output must also resolve conflicts between semantic and instance predictions. Multiple masks may overlap, different queries may predict the same object, or a semantic classifier may disagree with an instance-level prediction. Post-processing can rank predictions using confidence, suppress duplicates, resolve overlapping assignments, and ensure that each spatial element receives a consistent final label. Some architectures instead learn more of this assignment process directly to reduce dependence on handcrafted post-processing.

Temporal panoptic understanding extends segmentation from individual frames to persistent objects across time. If a vehicle is assigned one instance identifier in the current scan and another identifier in the next scan, higher-level reasoning becomes unnecessarily difficult. Temporal association can use motion estimation, geometric overlap, appearance features, object tracking, or learned embeddings to preserve identity. This transforms frame-level panoptic segmentation toward panoptic tracking and persistent scene understanding.

Persistent identity is particularly important for mobile robots operating in dynamic environments. The robot may need to distinguish a newly observed person from one already being tracked, recognize that a container has moved, or determine that another robot remains the same physical entity after temporary occlusion. Stable instance identities allow semantic mapping, trajectory prediction, interaction reasoning, and change detection to operate on entities rather than isolated measurements.

Panoptic segmentation also provides a useful interface between dense geometric perception and scene graphs. Dense points or voxels describe the detailed spatial extent of an object, while the instance identifier provides a natural entity from which a scene-graph node can be created. Semantic labels define object types, and geometric relationships can generate edges such as near, inside, on, connected to, or obstructing. Panoptic perception therefore supports the transition from dense sensing toward relational scene reasoning.

Evaluation must consider both recognition quality and segmentation quality. A system should correctly identify semantic categories while also separating individual instances and preserving their spatial boundaries. Panoptic Quality is commonly formulated to reflect both segmentation overlap and recognition performance. For robotics, additional evaluation may consider temporal identity stability, small-object performance, distance-dependent accuracy, inference latency, and the effect of segmentation errors on downstream planning or safety.

Class imbalance remains a significant challenge because large structural surfaces can contain millions of points while small task-relevant objects may contain very few. A model can achieve apparently strong aggregate accuracy while performing poorly on pedestrians, tools, handles, signs, or other important entities. Balanced sampling, class-aware losses, multi-scale feature extraction, and targeted augmentation can improve performance on rare or geometrically small classes without discarding the context provided by large scene structures.

Panoptic predictions should retain uncertainty because incorrect instance boundaries or semantic assignments can propagate into planning. Merging two nearby people into one instance, dividing one obstacle into several objects, or labeling an unknown structure as traversable space may produce unsafe decisions. Confidence estimates, unknown-region handling, temporal confirmation, multi-sensor consistency checks, and conservative map updates can prevent uncertain panoptic predictions from immediately becoming trusted environmental knowledge.

For navigation, panoptic understanding distinguishes persistent surfaces from individual dynamic obstacles and provides richer information than occupancy alone. For manipulation, it separates neighboring objects and preserves their semantic identity, supporting object selection and grasp planning. For inspection and industrial robotics, instance-level comparison can reveal whether specific components have moved, disappeared, or changed. The same representation can therefore support multiple robot behaviors without requiring completely separate scene interpretations.

Real-time deployment requires careful management of memory, computation, and latency. Large point clouds and dense indoor reconstructions can generate millions of spatial elements, while query-based mask processing may add substantial computational cost. Sparse feature representations, spatial cropping, hierarchical processing, mixed-precision inference, optimized attention, and GPU acceleration help maintain practical update rates. The appropriate configuration depends on scene scale, sensor frequency, robot speed, and required segmentation detail.

Within 3D scene understanding, panoptic segmentation represents an important transition from asking only "what exists here?" to simultaneously asking "what is it?" and "which individual entity is it?" By combining geometric structure, semantic categories, and instance identities, the robot obtains a more complete description of its surroundings. This structured representation provides the foundation for subsequent scene graph construction, persistent mapping, change detection, manipulation planning, and higher-level physical reasoning.

07.04. 3D Scene Graph Construction [w/Code]

A 3D scene graph represents an environment as a structured network of entities and their relationships, transforming dense geometric and semantic perception into a higher-level representation that a robot can reason about. Instead of treating a scene as an unorganized collection of points, voxels, or segmented regions, the graph represents meaningful entities as nodes and spatial, semantic, or functional relationships as edges. This provides an intermediate representation between 3D perception and higher-level reasoning, planning, and interaction.

The construction of a 3D scene graph generally begins with geometric and semantic observations obtained from cameras, LiDAR, RGB-D sensors, or fused perception systems. Objects, surfaces, rooms, structures, and other meaningful regions are extracted from the observed scene and converted into candidate graph entities. Semantic segmentation, instance segmentation, and panoptic segmentation provide useful information for determining object categories and individual identities. The resulting entities become the basic nodes from which the scene graph is constructed.

A scene graph node should contain more information than a simple object label. It may include an object category, instance identity, geometric extent, position, orientation, confidence, temporal state, and associated visual or geometric features. For example, a node representing a table can include its 3D bounding region, estimated pose, semantic class, and confidence score. Maintaining these attributes allows downstream modules to use the graph not merely as a symbolic description but as a connection to the underlying spatial evidence.

Edges describe relationships between nodes and are essential to the meaning of the graph. Common spatial relationships include near, far, left of, right of, above, below, inside, and adjacent to. Other relationships can describe physical or functional structure, such as on, attached to, connected to, supported by, or obstructing. The choice of relationships depends on the robot's application, because a navigation system may emphasize spatial connectivity while a manipulation system may require support, containment, and contact relationships.

Geometric relationships can be derived directly from estimated object positions, orientations, bounding volumes, surfaces, or point-cloud geometry. For example, the distance between two object centers can support a near or far relationship, while relative height can indicate that one object is above another. More complex relations may require geometric reasoning over surfaces and volumes. Because these relationships are derived from uncertain perception, their confidence should be retained rather than treating every graph edge as an exact fact.

Semantic relationships provide information that cannot always be obtained from geometry alone. An object may be recognized as a chair, table, vehicle, door, or machine component, while contextual information determines how that object relates to surrounding entities. A chair may be associated with a table, a vehicle may occupy a parking region, or a tool may belong to a particular workstation. Combining semantic labels with geometric relationships allows the graph to represent both what entities are and how they are situated within the environment.

Hierarchical scene graphs can represent multiple spatial levels simultaneously. Individual objects can belong to larger regions, rooms can belong to buildings, and buildings can belong to sites or facilities. This hierarchy allows a robot to reason at different scales without processing every low-level point at every stage. A navigation task may operate at the room or building level, while manipulation may descend to individual objects and their local relationships. Hierarchical structure therefore connects local perception with large-scale environmental reasoning.

Instance identity is particularly important when multiple objects share the same semantic category. Several chairs in a room may all have the semantic class "chair," but each must remain a separate graph entity if the robot needs to interact with a particular chair. Panoptic 3D segmentation provides a useful source of such identities by combining semantic categories with instance labels. The graph can then maintain separate nodes while associating them with common class-level information.

Temporal information extends a scene graph from a static description into a representation of a changing environment. Objects may move, disappear, appear, or change state as the robot operates. A robust system can associate observations from consecutive frames with existing graph nodes instead of continuously creating new entities. Temporal tracking, pose updates, confidence changes, and state transitions allow the graph to represent persistent entities while adapting to environmental changes.

Scene graph construction must handle uncertainty and incomplete observation because a robot rarely observes the entire environment simultaneously. Occlusion may hide part of an object, sensor range may limit visibility, and semantic classifiers may produce ambiguous predictions. A graph should therefore distinguish observed facts from uncertain or inferred relationships. Confidence values, unknown states, and provisional edges can prevent incomplete observations from being incorrectly treated as permanent knowledge.

Graph construction can also incorporate multimodal information. Camera images provide appearance and semantic cues, LiDAR contributes accurate spatial geometry, and IMU or localization systems provide the robot's pose and temporal reference. Multi-sensor fusion allows graph entities and relationships to be estimated from complementary evidence. This is consistent with the broader perception architecture in which camera, LiDAR, radar, IMU, and other sensing modalities contribute different information before higher-level scene understanding is performed.

The resulting graph can serve as an interface between perception and reasoning. Navigation systems can query which regions are connected, which objects obstruct a route, or where a particular semantic entity is located. Manipulation systems can determine which object is on a surface, which container contains another object, or which objects are spatially adjacent. Inspection systems can compare graph states over time to identify missing, moved, or newly introduced components.

For semantic navigation, scene graphs provide a mechanism for connecting language or task descriptions with physical entities. A command referring to a particular object or location can be grounded against semantic labels, spatial relationships, and graph hierarchy. Instead of searching an undifferentiated map, the robot can reason over entities and relations such as a target being inside a room, near a particular landmark, or beside another object. This makes the spatial representation more compatible with higher-level task reasoning.

Scene graphs can also support manipulation planning by exposing object relationships that are difficult to represent in a conventional occupancy map. A planner may need to know which object is on a table, which container is closed, or which object is blocking access to another object. These relationships can be converted into constraints or goals for motion planning and manipulation. The graph therefore acts as a semantic bridge between perception and physical interaction rather than replacing the geometric representations required for precise control.

Maintaining a useful scene graph requires continuous updating rather than one-time construction. As new sensor observations arrive, object poses may change, previously unknown regions may become visible, and environmental states may be revised. Incremental graph updates should therefore add new entities, refine existing attributes, modify relationships, remove obsolete information, and preserve persistent identities when appropriate. This approach prevents the graph from becoming a static snapshot that no longer corresponds to the physical environment.

In the broader 3D scene-understanding architecture, scene graph construction follows geometric and semantic interpretation and provides a higher-level representation for subsequent reasoning. The chapter structure places 3D semantic segmentation, panoptic segmentation, scene graph construction, neural implicit representations, 3D Gaussian Splatting, online reconstruction, change detection, manipulation planning, and edge deployment within the same progression. This organization positions the scene graph as a key abstraction that connects structured perception with downstream robotic intelligence.

The practical value of a 3D scene graph is therefore determined by how accurately and consistently it connects perception to actionable relationships. A useful graph preserves spatial structure, semantic identity, instance separation, temporal continuity, and uncertainty while remaining computationally manageable. When these properties are maintained, the scene graph becomes a compact representation through which a robot can reason about objects, places, relationships, changes, and task context, supporting navigation, manipulation, inspection, planning, and broader physical interaction.

07.05. Neural Implicit Representations NeRF 3DGS [w/Code]

Neural implicit representations provide a way to describe a three-dimensional scene as a continuous function rather than storing the environment only as discrete points, voxels, or meshes. Instead of explicitly recording every surface element, a neural network learns a function that maps spatial coordinates and viewing information to scene properties such as density, occupancy, color, or other learned features. This approach enables detailed scene reconstruction and view synthesis from a relatively limited collection of observations.

Neural Radiance Fields, commonly called NeRF, are a representative neural implicit representation for three-dimensional scenes. A NeRF models a scene as a continuous volumetric field that predicts density and view-dependent color at sampled locations. During rendering, rays are traced through the learned field and accumulated according to the predicted density and color. By optimizing the network so that rendered images reproduce observed camera images, the model learns a representation capable of synthesizing new viewpoints.

The basic NeRF formulation relies on camera poses and multiple images captured from different viewpoints. For each training image, camera rays are sampled through the scene, and points along each ray are evaluated by the neural field. Volumetric rendering integrates these samples to produce a predicted pixel color, which is compared with the corresponding observed pixel. Repeated optimization across many rays and views allows the neural representation to encode both scene geometry and appearance within the learned continuous field.

One important characteristic of NeRF is that geometry is represented implicitly rather than as an explicit mesh or point cloud. Density values provide information about where surfaces are likely to exist, while view-dependent radiance captures appearance changes caused by viewing direction, illumination, or material properties. This enables highly realistic novel-view synthesis, but extracting precise geometric information can require additional processing because the learned representation is fundamentally designed around differentiable image rendering.

NeRF-based reconstruction can provide visual information that is difficult to obtain from a single viewpoint. Previously hidden surfaces can become represented when they are observed from additional cameras, while the continuous field can produce visually consistent intermediate viewpoints. However, reconstruction quality depends strongly on camera coverage, pose accuracy, image quality, scene motion, and the ability of the model to generalize beyond the training observations.

Traditional NeRF optimization can be computationally expensive because training requires repeated ray sampling, neural-network evaluation, and volumetric rendering over many images. This characteristic limits its direct use in applications that require rapid reconstruction or frequent scene updates. Research therefore introduced improved architectures, sampling strategies, acceleration structures, and alternative representations designed to reduce training and rendering costs while retaining the advantages of continuous scene modeling.

Three-dimensional Gaussian Splatting, commonly called 3DGS, provides a different approach to neural scene representation and rendering. Instead of representing the entire scene primarily through a multilayer neural field, 3DGS represents the scene using a collection of three-dimensional Gaussian primitives. Each Gaussian can encode a position, scale, orientation, opacity, and appearance-related parameters. These primitives are rendered by projecting them into the image plane and efficiently accumulating their contributions.

The Gaussian representation provides an explicit set of spatial primitives while retaining a differentiable optimization process. During training, the positions, shapes, orientations, opacity, and appearance of the Gaussian primitives are optimized so that rendered images match the observed views. Adaptive refinement can add or modify Gaussians in regions where additional spatial detail is required. The result is a representation that can preserve fine visual structures while supporting efficient rendering.

Compared with conventional NeRF, 3DGS can provide substantially faster rendering because it avoids repeatedly evaluating a deep neural network at large numbers of sampled points along every camera ray. The scene is instead represented by spatial primitives that can be projected and rasterized efficiently. This characteristic makes Gaussian Splatting attractive for interactive visualization, virtual environments, digital twins, robotics simulation, and other applications where rapid novel-view rendering is important.

NeRF and 3DGS represent two related but distinct design philosophies. NeRF emphasizes a continuous learned function that implicitly describes scene radiance and density, whereas 3DGS uses an explicit collection of optimized spatial primitives to represent the scene. Both methods learn from multi-view observations and can synthesize viewpoints that were not directly captured. Their differences in representation, optimization, memory organization, and rendering behavior lead to different trade-offs for robotics applications.

For robotic perception, neural implicit representations should not be considered replacements for every geometric representation. Point clouds, meshes, occupancy grids, signed distance fields, and semantic maps remain useful because navigation and manipulation often require explicit spatial reasoning. Neural representations can instead complement these structures by providing dense appearance information, detailed reconstruction, view synthesis, or learned features that can be converted into task-specific geometric or semantic representations.

Integration with camera pose estimation is particularly important. Neural reconstruction assumes that observations correspond to known or consistently estimated viewpoints. Errors in camera calibration or pose can cause geometric distortions, blurry structures, duplicated surfaces, or inconsistent appearance. For robots, this means that neural reconstruction should be connected carefully to localization, visual odometry, SLAM, and multi-sensor calibration rather than being treated as an independent perception component.

Temporal reconstruction introduces another challenge because real environments are not always static. A conventional NeRF or 3DGS model generally assumes that the observed scene can be explained by a relatively stable representation. Moving people, vehicles, doors, robots, or changing objects can violate this assumption. Practical robotic systems therefore need mechanisms for separating static and dynamic content, updating local regions, tracking changes, or maintaining multiple representations when the environment evolves.

The relationship between neural reconstruction and semantic understanding is also important. A visually detailed reconstruction does not automatically provide knowledge about what each region represents. Semantic segmentation, instance segmentation, panoptic segmentation, and scene graphs can be associated with reconstructed geometry to produce richer representations. In this way, neural rendering can provide detailed spatial and appearance information while semantic perception supplies object identity, category information, and relationships.

Neural implicit representations also have implications for edge deployment. Large neural fields or high-resolution Gaussian representations can require substantial memory and computation, while robotic systems often operate under strict GPU, power, latency, and thermal constraints. Practical deployment may therefore require scene cropping, level-of-detail control, representation compression, reduced precision, selective reconstruction, or region-of-interest processing. The goal is not necessarily to reconstruct everything at maximum quality but to maintain sufficient information for the robot's current task.

Within the broader 3D scene-understanding architecture, neural implicit representations follow semantic and panoptic scene interpretation and precede techniques for Gaussian-based reconstruction, online 3D reconstruction, scene change detection, manipulation planning, and edge deployment. The chapter structure therefore treats NeRF and 3DGS as part of a progression from understanding scene content toward richer and more efficient spatial reconstruction.

The practical value of NeRF and 3DGS for robotics ultimately depends on how their reconstructed information is connected to physical action. A robot does not require visual realism alone; it needs reliable spatial information that can support localization, navigation, inspection, manipulation, and planning. Neural implicit representations are therefore most useful when their strengths in continuous reconstruction, appearance modeling, and efficient rendering are integrated with explicit geometry, semantic understanding, temporal reasoning, and task-oriented robotic decision making.

07.06. 3D Gaussian Splatting for Scene Reconstruction [w/Code]

3D Gaussian Splatting, commonly called 3DGS, represents a three-dimensional scene using a collection of explicit Gaussian primitives that can be optimized from multi-view images. Each Gaussian describes a localized region of space through parameters such as position, scale, orientation, opacity, and appearance. Unlike conventional mesh reconstruction, which explicitly models connected surfaces, 3DGS maintains a distributed set of spatial primitives that can reproduce the visual structure of a scene through efficient projection and rendering.

The reconstruction process begins with images captured from multiple viewpoints together with estimated camera poses. The images provide appearance information, while camera poses establish the spatial relationship between observations. An initial set of three-dimensional points can be obtained from structure-from-motion or another geometric reconstruction method. These points provide the starting locations for Gaussian primitives, which are subsequently optimized so that their rendered projections reproduce the observed images.

Each Gaussian can be understood as an anisotropic three-dimensional probability-like volume centered at a particular position. Its covariance determines the spatial shape, scale, and orientation, while opacity controls how strongly it contributes to the rendered image. Appearance parameters determine the visible color or radiance of the Gaussian. A collection of these primitives therefore forms an explicit volumetric representation in which different spatial regions can have different sizes, orientations, densities, and visual properties.

Rendering is performed by projecting three-dimensional Gaussians into the camera view and accumulating their contributions according to depth, opacity, and appearance. This process, often called splatting, avoids constructing a conventional polygon mesh before visualization. Because the primitives can be projected and rasterized efficiently, 3DGS can achieve high-quality novel-view rendering at interactive or near-real-time rates on suitable hardware. This rendering efficiency is one of the major practical characteristics that distinguishes 3DGS from computationally intensive neural volumetric rendering.

Optimization adjusts the Gaussian parameters so that rendered images match the original multi-view observations. The training objective can include image reconstruction losses and additional regularization mechanisms that encourage stable and visually consistent representations. During optimization, poorly represented regions can be refined by modifying existing Gaussians or introducing additional primitives. Regions with insufficient detail can therefore receive more spatial primitives, while redundant regions can remain comparatively compact.

Adaptive density control is an important component of practical 3DGS reconstruction. A uniform distribution of primitives is inefficient because different parts of a scene require different levels of representation detail. Fine structures such as thin objects, edges, vegetation, and textured surfaces may require many small Gaussians, whereas large smooth surfaces can often be represented using fewer primitives. Adaptive splitting, cloning, pruning, and parameter optimization allow the representation to allocate capacity according to the observed visual complexity.

The explicit nature of Gaussian primitives provides an important distinction from implicit neural fields such as NeRF. NeRF represents a scene through a learned continuous function that must be evaluated during ray-based rendering, whereas 3DGS stores optimized spatial primitives that can be directly projected into the image. This difference can reduce rendering computation and provide more direct access to spatial elements. At the same time, 3DGS does not automatically provide the physically meaningful surfaces, semantic labels, or navigation structures required by every robotic application.

Camera calibration and pose accuracy strongly influence reconstruction quality. If camera positions or orientations are incorrect, the same physical structure may be projected into inconsistent locations across different images. Optimization can then produce blurred surfaces, duplicated structures, distorted geometry, or incorrectly positioned Gaussians. For robotics, reliable camera calibration, visual odometry, SLAM, and multi-sensor calibration are therefore important foundations for producing a stable Gaussian representation.

The geometric quality of a Gaussian scene depends not only on image appearance but also on viewpoint coverage. Areas observed from many directions generally provide stronger constraints on spatial structure, while regions visible from only one or two viewpoints may remain ambiguous. Occlusion creates a similar limitation because hidden surfaces cannot be reconstructed reliably without additional observations. A robot operating in a physical environment must therefore consider sensor trajectory and viewpoint coverage when collecting data for Gaussian-based reconstruction.

Although 3DGS provides an explicit spatial representation, the Gaussian primitives are not equivalent to conventional object surfaces. A Gaussian may overlap multiple physical structures or represent appearance without defining a precise physical boundary. Consequently, direct geometric queries such as collision checking, precise object dimensions, or manipulation contact surfaces may require additional processing. Gaussian representations are particularly strong for visual reconstruction and rendering, while explicit meshes, point clouds, occupancy structures, or signed distance representations may remain necessary for physical interaction.

Semantic integration can extend Gaussian Splatting from visual reconstruction toward structured scene understanding. Semantic labels can be associated with individual Gaussians or groups of primitives, allowing reconstructed regions to be classified as walls, floors, furniture, vehicles, vegetation, or other meaningful entities. Instance information can further separate different objects belonging to the same category. Combining Gaussian geometry and appearance with semantic and instance information produces a richer representation that can support scene graphs, semantic maps, inspection, and task-oriented reasoning.

Temporal reconstruction introduces additional challenges because Gaussian Splatting commonly assumes that the scene remains sufficiently stable during reconstruction. Moving people, vehicles, robots, doors, or other objects can produce inconsistent observations and cause their appearance to be distributed incorrectly across space. Practical systems may separate static and dynamic components, reconstruct local regions independently, update selected Gaussian subsets, or maintain temporal associations between scene elements. Such mechanisms are important when 3DGS is applied to continuously changing robotic environments.

Scene changes can also be detected by comparing Gaussian-based representations collected at different times. Differences in the spatial distribution, appearance, opacity, or semantic attributes of Gaussian primitives may indicate that an object has moved, disappeared, or been introduced. However, changes caused by lighting, viewpoint, sensor noise, or reconstruction uncertainty must be distinguished from genuine physical changes. Reliable change detection therefore benefits from temporal consistency, confidence estimation, geometric verification, and complementary sensor information.

For robotics, the representation can be organized at different spatial scales. A global Gaussian model can provide a detailed visual representation of a large environment, while local Gaussian subsets can be loaded or refined around the robot's current position. Level-of-detail management can reduce memory and rendering costs by using fewer primitives for distant or visually unimportant regions. This hierarchical strategy is particularly relevant for mobile robots that must operate continuously rather than reconstruct an entire environment at maximum resolution.

Edge deployment requires additional consideration because large Gaussian scenes can contain millions of primitives. Although rendering may be efficient, storing, transferring, updating, and processing large representations can still consume significant memory and bandwidth. Compression, pruning, spatial partitioning, streaming, level-of-detail selection, and region-of-interest processing can reduce these costs. A practical robotic system should therefore optimize not only image quality but also memory footprint, update latency, GPU utilization, and the amount of scene information actually required by downstream tasks.

3DGS can support digital twins and simulation by providing visually detailed reconstructions of physical environments. A reconstructed facility, laboratory, warehouse, or outdoor site can be rendered from viewpoints that were not directly captured. When combined with semantic information, object identities, spatial maps, and dynamic state information, the Gaussian representation can become one layer of a richer digital-twin system. It should nevertheless be distinguished from a complete physical simulation because visual reconstruction alone does not automatically encode physical dynamics, collision properties, or control interfaces.

For navigation and inspection, Gaussian Splatting can provide a detailed visual context around the robot and support viewpoint synthesis for areas that are difficult to observe directly. For manipulation, reconstructed appearance can help identify objects and surfaces, but precise interaction still requires reliable geometry, object pose, collision information, and contact constraints. The most practical architecture is therefore often hybrid: 3DGS provides high-quality visual reconstruction and appearance, while explicit geometric and semantic representations provide the information required for physical reasoning and control.

Within the broader 3D scene-understanding architecture, 3D Gaussian Splatting follows neural implicit representations and precedes online reconstruction, scene change detection, manipulation planning, and edge deployment. The chapter structure explicitly places neural implicit representations, Gaussian Splatting, online 3D reconstruction, scene change detection, manipulation planning, and edge deployment as successive components of the same 3D scene-understanding progression. This positioning emphasizes that 3DGS is not an isolated rendering technique but part of a larger pipeline connecting perception, reconstruction, spatial understanding, and robotic action.

The practical significance of 3D Gaussian Splatting for robotics ultimately depends on how efficiently its visual and spatial information can be connected to physical tasks. Its strengths include explicit spatial primitives, high-quality novel-view synthesis, efficient rendering, adaptive representation density, and compatibility with multi-view reconstruction. Its limitations include dependence on viewpoint coverage, sensitivity to camera pose quality, challenges with dynamic scenes, potentially large memory requirements, and the need for additional geometric and semantic processing. When integrated with localization, mapping, semantic perception, temporal reasoning, and task-specific planning, 3DGS can become an effective reconstruction layer within a broader physical AI perception system.

07.07. Online 3D Reconstruction TSDF KinectFusion [w/Code]

Online 3D reconstruction continuously builds and updates a three-dimensional representation while new sensor observations arrive. Unlike offline reconstruction, which can process a complete dataset after acquisition, an online system must estimate the sensor pose, integrate each new depth observation, update the scene representation, and maintain sufficient accuracy while the robot is moving. This makes online reconstruction an important connection between perception, localization, mapping, and real-time robotic operation.

A representative classical approach is KinectFusion, which combines depth sensing, camera tracking, and volumetric reconstruction in a continuous processing loop. RGB-D or depth cameras provide dense depth measurements, while the system estimates the camera pose relative to the existing model. Each new depth frame is then aligned with the current reconstruction and integrated into a volumetric representation. This iterative process allows a robot or handheld sensor to progressively build a 3D model while moving through an environment.

The central representation in KinectFusion is the truncated signed distance function, commonly called TSDF. A TSDF volume stores an estimated signed distance for each voxel relative to nearby surfaces. Values indicate whether a voxel is in front of, near, or behind an observed surface, while truncation limits the stored distance to a narrow band around the surface. This makes the representation useful for integrating noisy depth measurements while maintaining a smooth volumetric description of scene geometry.

Depth integration updates the TSDF volume whenever a new depth frame is received. The depth image is projected into the reconstruction volume using the estimated camera pose and calibrated camera model. For each relevant voxel, the system compares its predicted position with the measured surface depth and computes a truncated signed distance. Repeated observations are fused using weighted updates, allowing measurements from different viewpoints to reinforce consistent surfaces and reduce the influence of individual noisy observations.

Camera tracking is essential because every depth observation must be associated with the correct position and orientation in the global reconstruction. KinectFusion estimates the camera pose by aligning the current depth observation with the existing model. The alignment can use geometric information such as surface points and normals, commonly through an iterative optimization process. Accurate tracking allows consecutive observations to accumulate into a coherent model rather than producing disconnected or duplicated surfaces.

The tracking and reconstruction processes therefore form a closed loop. The current model provides a reference for estimating the new camera pose, while the newly estimated pose determines how the next depth frame is integrated into the model. If tracking becomes inaccurate, the resulting depth integration can introduce geometric distortion, double surfaces, or misalignment. Maintaining reliable localization is consequently as important as the volumetric integration method itself.

A major advantage of TSDF-based reconstruction is that it produces a volumetric representation from which surfaces can be extracted explicitly. Once the TSDF volume has been updated, an isosurface can be generated using methods such as Marching Cubes. The resulting mesh provides a continuous surface representation that can be visualized, analyzed, or passed to downstream geometric processing. This creates a useful relationship between volumetric mapping and explicit surface reconstruction.

KinectFusion was particularly influential because it demonstrated that dense 3D reconstruction and camera tracking could be performed incrementally from depth observations. However, the original approach has practical limitations. A dense voxel volume can consume substantial memory, and the reconstructed workspace is often constrained by the selected volume size and voxel resolution. Large environments therefore require alternative memory structures, spatial partitioning, or hierarchical representations to maintain scalability.

The computational requirements are also significant because each incoming frame may require depth preprocessing, pose estimation, TSDF integration, and surface processing. Real-time operation therefore depends on efficient algorithms and hardware acceleration. Parallel processing on GPUs can accelerate voxel integration, geometric correspondence, and other stages. Modern systems can further improve efficiency through sparse representations, hierarchical volumes, reduced-resolution processing, or selective updates of regions that are relevant to the robot.

Online reconstruction is strongly dependent on sensor calibration. The intrinsic parameters of the depth camera determine how image coordinates correspond to rays in three-dimensional space, while extrinsic parameters determine the relationship between the camera and other sensors or the robot body. Calibration errors can produce systematic surface distortions that accumulate over time. Accurate calibration and synchronization are therefore essential when depth information is combined with RGB cameras, IMUs, LiDAR, or other sensing modalities.

Sensor noise and missing depth measurements also influence TSDF quality. Depth cameras can produce unreliable measurements on reflective, transparent, dark, or distant surfaces, and occlusion can leave parts of the scene unobserved. Weighted integration helps reduce the influence of individual noisy measurements, but persistent sensing errors cannot be eliminated by averaging alone. Robust preprocessing, confidence estimation, outlier rejection, sensor fusion, and appropriate observation weighting can improve reconstruction stability.

A further challenge is the assumption of a mostly static environment. Classical TSDF integration naturally accumulates observations into a persistent model, which can cause moving objects to leave ghost surfaces or motion trails. People, vehicles, doors, or movable equipment can therefore corrupt the reconstruction if they are treated as static geometry. Practical robotic systems may need dynamic-object detection, temporal filtering, selective integration, or separate static and dynamic representations to maintain a useful map.

The relationship between online reconstruction and SLAM is particularly important. Reconstruction provides detailed geometric information, while SLAM provides localization and global consistency over the robot trajectory. Local tracking can align consecutive frames effectively, but accumulated drift may cause large-scale deformation unless loop closure or global optimization is introduced. A practical system can therefore combine local dense tracking with pose-graph optimization, loop closure, or external localization sources to maintain consistency over larger environments.

TSDF reconstruction can also be combined with semantic perception. Semantic segmentation can assign categories to reconstructed surfaces, while instance or panoptic segmentation can associate spatial regions with individual objects. The resulting representation can distinguish walls, floors, furniture, machines, vehicles, or other entities within the reconstructed geometry. This combination moves online reconstruction from purely geometric mapping toward semantic scene understanding.

Multi-resolution and spatially adaptive representations are useful when the environment contains both large-scale structure and fine geometric details. A robot may require high resolution near its current operating region while only coarse geometry is necessary for distant areas. Hierarchical TSDF volumes, voxel hashing, submap structures, and local reconstruction windows can allocate computational and memory resources according to spatial importance. These strategies allow online reconstruction to extend beyond the limitations of a single dense fixed-size volume.

For navigation, reconstructed geometry can provide obstacle surfaces, free space, terrain structure, and spatial constraints for planning. For manipulation, local high-resolution reconstruction can provide surfaces and object geometry needed for grasping and interaction. For inspection, repeated reconstructions can be compared to identify geometric changes. The same underlying reconstruction framework can therefore support different robotic tasks when its geometric output is connected to appropriate semantic, planning, and decision-making modules.

Online reconstruction also provides a foundation for change detection. A current TSDF or surface model can be compared with a previous map to identify regions where geometry has changed. However, apparent changes can result from localization error, sensor noise, incomplete observation, or different viewpoints rather than actual environmental modifications. Reliable change detection therefore requires geometric registration, uncertainty handling, temporal consistency, and, when possible, semantic or multimodal confirmation.

The progression from TSDF and KinectFusion toward modern neural and Gaussian representations does not make classical volumetric reconstruction obsolete. TSDF remains valuable because it provides an explicit spatial representation, supports geometric queries, integrates depth measurements naturally, and can produce surfaces suitable for physical reasoning. Neural implicit representations and 3D Gaussian Splatting provide different advantages in appearance modeling, view synthesis, and efficient rendering, but robotic systems may still require explicit geometry for collision checking, navigation, manipulation, and metric reasoning.

Within the broader 3D scene-understanding architecture, online 3D reconstruction follows neural implicit and Gaussian-based scene representations and connects directly to scene change detection, manipulation planning, and edge deployment. The progression from semantic understanding and scene graphs through neural representations, Gaussian Splatting, online reconstruction, and subsequent robotic applications reflects an increasing emphasis on continuously updating spatial knowledge rather than constructing a static model.

The practical objective of online 3D reconstruction is therefore not simply to create a visually complete model but to maintain a geometrically useful representation while the robot operates. KinectFusion and TSDF provide a clear foundation for understanding this problem: calibrated depth observations are tracked in space, integrated into a volumetric model, and continuously converted into usable geometric information. When combined with robust localization, sensor fusion, semantic understanding, dynamic-scene handling, and efficient computation, online reconstruction becomes a core capability for persistent spatial awareness in autonomous robots.

07.08. Scene Change Detection and Map Update [w/Code]

Scene change detection identifies meaningful differences between a previously constructed 3D scene representation and newly acquired observations. In a robotic system, the objective is not simply to detect visual differences but to determine whether the physical environment has actually changed. Objects may move, disappear, appear, or change state, while lighting, sensor noise, viewpoint variation, and localization errors can also produce apparent differences. Reliable change detection therefore requires geometric, semantic, and temporal reasoning.

A practical change-detection pipeline begins with a reference map and newly acquired sensor observations. The current observation may originate from LiDAR, RGB-D cameras, stereo cameras, or a fused 3D perception system. Before comparison, the new data must be transformed into a common coordinate frame using localization, SLAM, or map registration. Accurate spatial alignment is essential because even a small pose error can create large apparent differences, particularly around object boundaries and fine structures.

Geometric change detection compares spatial representations such as point clouds, occupancy maps, meshes, TSDF volumes, or other 3D reconstruction outputs. Corresponding regions are evaluated according to distance, occupancy, surface position, or geometric overlap. A region may be considered changed when the observed surface consistently differs from the reference geometry beyond an appropriate tolerance. The threshold must account for sensor accuracy, reconstruction uncertainty, and the spatial resolution of the map.

Point-cloud comparison is a direct method for identifying geometric differences. Newly observed points can be compared with nearby points in the reference map using distance-based correspondence or local surface models. Excess points may indicate newly introduced objects, while missing reference points may indicate removed or occluded structures. However, point density changes caused by distance, viewing angle, or sensor characteristics can produce false detections, so geometric comparison should normally incorporate spatial tolerance and confidence information.

Occupancy-based comparison provides another practical approach. A reference occupancy map represents regions as occupied, free, or unknown, while a newly generated observation updates these states. A transition from free to occupied may indicate a newly introduced obstacle, while an occupied-to-free transition may indicate object removal or movement. Unknown regions should not automatically be classified as changed because insufficient observation can produce the same pattern as physical disappearance.

Semantic change detection adds information about what has changed rather than only where geometry differs. A region that remains geometrically occupied may nevertheless change from one semantic category to another, or an object may retain its category while changing its instance identity or state. Semantic labels from 3D segmentation, panoptic segmentation, and object recognition can therefore be compared across time to identify meaningful environmental changes.

Instance-level change detection is particularly useful for mobile robots operating around movable objects. A chair may move from one location to another while remaining the same semantic class, or a vehicle may leave a parking area and another vehicle may occupy the same region. Instance identities, object poses, geometric extents, and temporal associations allow the system to distinguish object motion from simple appearance or disappearance. This creates a connection between scene change detection and persistent object tracking.

Temporal consistency is necessary because a single observation should rarely be sufficient to declare a permanent environmental change. A person temporarily blocking a wall does not mean that the wall has disappeared, and a temporary sensor failure does not mean that an object has been removed. A robust system can require repeated observations, temporal confirmation, or confidence accumulation before modifying the persistent map. This prevents short-lived events from unnecessarily corrupting long-term spatial knowledge.

Dynamic objects should therefore be handled differently from static environmental structures. People, vehicles, doors, movable equipment, and other objects can change position frequently. If their observations are directly integrated into a static map, the resulting representation may contain ghost objects, duplicated surfaces, or obsolete geometry. Dynamic-object detection and semantic classification can identify regions that should be tracked separately, temporarily excluded from static-map updates, or maintained in a dynamic layer.

Map updating should be treated as a controlled process rather than a simple replacement of old information with new observations. Stable regions can be retained, confirmed changes can update the reference representation, and uncertain regions can remain unchanged until additional evidence becomes available. This approach allows the map to preserve previously reliable information while gradually incorporating verified environmental changes. Confidence, observation history, and update timestamps can be stored with map elements to support this process.

Registration quality directly affects map updating because change detection cannot reliably distinguish environmental change from localization error when the two are spatially similar. Loop closure, global pose optimization, scan matching, and multi-sensor localization can reduce accumulated drift. In larger environments, submaps or local map regions can be independently registered and then connected through a global pose graph. This hierarchical approach helps maintain spatial consistency while allowing continuous local updates.

Scene graphs provide a higher-level representation for change management. Instead of comparing only raw geometry, the system can compare graph nodes and relationships over time. A node representing a machine may change position, an object may disappear from a workspace, or a new object may become associated with a particular zone. Relationships such as inside, near, on, connected to, or obstructing can also change. These graph-level changes can provide more meaningful descriptions than raw point differences.

Change detection can support semantic map maintenance by updating information at different spatial and semantic levels. Static structures such as walls and floors may be updated slowly, while movable objects may require frequent updates. Functional zones, workspaces, traffic areas, and restricted regions can also change as their associated objects or operating conditions change. A layered map architecture can therefore separate long-term structural information from short-term environmental states.

For navigation, reliable map updates prevent planners from relying on obsolete obstacles or incorrectly assuming that previously free areas remain available. If a pallet is placed in a warehouse aisle, the navigation system should incorporate the new obstacle into the relevant representation. Conversely, when the pallet is removed and the change is confirmed, the corresponding free-space representation can be restored. Change-aware mapping therefore provides an important connection between perception and continuously updated navigation.

For manipulation and inspection, change detection provides more detailed information about object state. A robot can determine whether a component has moved, whether a tool is missing from a workstation, or whether an inspected structure differs from a previous reference state. Comparing object geometry, pose, semantic identity, and local scene relationships can support maintenance, inventory monitoring, process verification, and task execution. The same mechanism can operate at different spatial scales depending on the application.

Uncertainty must remain part of the change representation. A detected difference should not automatically be treated as a confirmed physical change because the cause may be sensor noise, incomplete visibility, calibration error, localization drift, or reconstruction artifacts. Change confidence can be estimated from observation frequency, geometric consistency, semantic agreement, sensor agreement, and temporal persistence. Maintaining confirmed, probable, and uncertain states allows downstream systems to respond according to the reliability of the evidence.

Multi-sensor fusion can improve change detection by combining complementary observations. LiDAR provides accurate spatial structure, cameras contribute appearance and semantic information, RGB-D sensors provide dense local geometry, and IMU or localization systems provide motion and pose information. When multiple sensors independently support the same change, confidence can increase. Conversely, disagreement between modalities can trigger additional observation rather than immediately modifying the persistent map.

Online 3D reconstruction methods such as TSDF-based mapping provide a natural geometric foundation for continuous scene comparison. Neural implicit representations and 3D Gaussian Splatting can additionally provide detailed appearance and spatial information, although their suitability for direct geometric change measurement depends on the representation and application. A practical system can therefore combine explicit geometric maps with richer neural or Gaussian representations, using each according to its strengths.

Scene change detection also supports long-term autonomy by allowing a robot to maintain environmental knowledge over extended periods. A map created during an initial deployment should not remain permanently fixed when the physical environment evolves. Instead, the robot can accumulate observations, identify persistent changes, update affected regions, preserve stable structures, and retain historical information when required. This transforms mapping from a one-time reconstruction process into continuous environmental state management.

Within the broader 3D scene-understanding architecture, scene change detection follows semantic and panoptic interpretation, scene graph construction, neural scene representation, Gaussian Splatting, and online 3D reconstruction. It connects these perception and reconstruction layers to later functions such as manipulation planning and edge deployment. The resulting progression moves from recognizing scene content, through reconstructing and organizing spatial information, toward maintaining an evolving representation that remains useful for autonomous robotic operation.

The practical objective of scene change detection and map update is therefore to maintain a trustworthy representation of the current physical environment rather than simply producing a difference image. A robust system aligns observations accurately, distinguishes real changes from sensing and localization errors, separates static and dynamic information, associates changes with semantic and instance identities, and updates the map only when sufficient evidence is available. When integrated with localization, 3D reconstruction, semantic perception, scene graphs, and task planning, change-aware mapping becomes a foundation for persistent spatial awareness and long-term autonomous operation.

07.09. 3D Scene Understanding for Manipulation Planning [w/Code]

3D scene understanding for manipulation planning transforms raw visual and geometric observations into a representation that allows a robot to determine what objects exist, where they are located, how they are related, and how they can be physically manipulated. Manipulation requires substantially richer understanding than simple object detection because the robot must reason about object pose, surfaces, accessibility, contact possibilities, obstacles, and task context. The resulting representation connects perception directly with grasping, motion planning, collision checking, and action execution.

The process begins with multimodal sensing. Cameras provide visual appearance and semantic information, while depth sensors, RGB-D cameras, LiDAR, tactile sensors, force sensors, and proprioceptive measurements provide complementary geometric and physical information. These observations can be converted into depth maps, point clouds, surface normals, occupancy or signed-distance representations, semantic regions, object poses, and other spatial features. Combining these representations allows the robot to construct a richer description of the manipulation workspace than any single sensor can provide.

Object understanding is a fundamental component because manipulation requires more than identifying an object category. The robot may need to estimate a six-degree-of-freedom pose, identify graspable surfaces, determine object orientation, infer approximate stability, recognize articulated components, and estimate whether surrounding geometry restricts access. Modern vision models can provide generalized semantic understanding of objects beyond fixed training categories, but the resulting predictions must still be connected to accurate spatial information before they can support reliable physical interaction.

Affordance understanding connects scene perception with possible actions. An affordance describes an interaction opportunity offered by an object or surface, such as a handle that can be pulled, a button that can be pressed, a drawer that can be opened, or a cup that can be grasped. Affordances depend on both object properties and spatial configuration. Two different objects may provide similar manipulation opportunities, while the same object may provide different opportunities depending on its orientation, surrounding obstacles, and accessibility.

Scene understanding extends this reasoning from individual objects to relationships among multiple entities. A target object may be inside a container, behind another object, partially occluded, resting on a surface, or surrounded by obstacles. A robot therefore needs relationships such as inside, on, near, connected to, supported by, and obstructing. Scene graphs, semantic maps, occupancy representations, and world models provide structured mechanisms for representing these relationships and connecting them with task-level reasoning.

Occlusion reasoning is particularly important because manipulation often occurs in cluttered environments. A target object may be partially hidden by another object or visible only from a restricted viewpoint. The robot must distinguish between what is directly observed and what is inferred from surrounding geometry and previous observations. Active perception can improve this process by changing the camera viewpoint or robot position to obtain additional information when uncertainty about an important object remains high.

Spatial feature representation provides the bridge between perception and planning. Useful features include point clouds, depth maps, surface normals, object poses, semantic segmentation, occupancy or signed-distance fields, dense correspondences, and affordance maps. These representations describe not only the appearance of objects but also their geometry and possible interaction regions. A manipulation planner can then use these features to evaluate grasp candidates, approach directions, collision constraints, and feasible trajectories.

Multi-modal fusion combines these heterogeneous observations into a unified spatial representation. Visual features, depth and geometry, point-cloud features, surface normals, tactile or force information, proprioception, and language instructions can be processed through cross-modal attention and feature fusion. This is especially important when one sensing modality is insufficient. Vision may identify an object, depth may estimate its geometry, tactile sensing may confirm contact, and language may specify which object should be manipulated.

Once the scene representation has been constructed, manipulation planning can evaluate candidate actions. Grasp detection identifies potential contact locations and evaluates their suitability, while collision checking verifies whether the robot, gripper, or manipulated object can move without intersecting surrounding geometry. Pose and trajectory planning then determines how the robot should approach the object and execute the manipulation while respecting kinematic, geometric, and environmental constraints. This creates a continuous connection between perception and physical action.

Manipulation planning operates at several levels. Task-level reasoning determines what must be accomplished, such as picking up an object and placing it on a shelf. Behavioral planning selects an appropriate interaction strategy, such as grasping, pushing, pulling, sliding, insertion, or dual-arm coordination. Motion planning generates collision-free trajectories, while grasp planning determines contact locations, gripper configuration, approach direction, and force distribution. These levels interact continuously with perception because changes in object pose or scene state can invalidate an existing plan.

The perception-planning loop should therefore remain closed rather than treating perception as a one-time preprocessing step. After an action begins, new observations can reveal object motion, unexpected contact, occlusion, grasp failure, or environmental changes. The robot can update its spatial representation, reassess affordances and constraints, and replan when necessary. This closed-loop structure is especially important for uncertain environments and aligns with embodied perception, where observation and action continuously influence one another.

Memory and world-model components can preserve 3D maps, object states, previous observations, and task-relevant relationships. Persistent information allows the robot to distinguish newly observed objects from previously known entities and to maintain useful context when an object temporarily disappears from view. World models can also connect spatial state with task state, allowing planning systems to reason about what has already been accomplished, what remains uncertain, and what observations should be obtained next.

The resulting perception and planning architecture supports a wide range of manipulation applications, including grasping, assembly, insertion, sorting, tool use, deformable-object manipulation, inspection, pick-and-place, door opening and closing, and screw driving. These tasks require different combinations of geometric precision, semantic understanding, force interaction, and planning complexity. A common 3D scene representation can therefore serve as a foundation while task-specific modules determine the appropriate action policy.

Vision-Language-Action systems extend this architecture by connecting natural-language instructions with spatial perception and physical actions. A language instruction can identify an intended object, destination, or task sequence, while spatial feature extraction determines the corresponding physical entities and relationships. Fusion and reasoning combine language, perception, memory, and world state before an action policy or planner generates robot behavior. This provides a pathway from language-grounded task understanding to physically executable manipulation.

Training such systems requires diverse real and simulated data because manipulation must generalize across objects, environments, viewpoints, and interaction conditions. Large-scale datasets, photorealistic simulation, domain randomization, supervised and self-supervised learning, affordance learning, continual learning, and transfer learning can contribute to robustness. Simulation is particularly useful for generating controlled variations in lighting, textures, camera configuration, physics, object placement, and task conditions before deployment on physical robots.

Within the broader 3D scene-understanding architecture, manipulation planning represents the transition from understanding the environment to acting upon it. Semantic and panoptic perception identify objects and instances, scene graphs organize their relationships, neural and Gaussian representations enrich spatial and visual information, and online reconstruction maintains the evolving geometry. Manipulation planning then uses these representations to determine where, how, and in what sequence the robot should interact with the physical environment.

The practical objective is therefore not simply to create an accurate 3D model, but to create a task-relevant spatial understanding that supports reliable action. A manipulation-capable robot must connect geometry, semantics, object identity, affordances, relationships, uncertainty, and temporal state with grasp and motion planning. When these elements operate in a closed perception-reasoning-planning-action loop, 3D scene understanding becomes a foundation for physical interaction rather than merely a scene description.

07.10. 3D Scene Understanding Deployment on Edge [w/Code]

3D scene understanding on the edge converts complex spatial perception into a form that can operate directly on a robot with limited computing, memory, power, and thermal resources. The objective is not simply to reproduce the complete capability of a high-end workstation, but to preserve the scene information required for navigation, manipulation, inspection, safety, and local autonomy. Edge deployment therefore requires careful coordination between perception models, spatial representations, inference acceleration, memory usage, latency, and the robot's real-time software architecture.

The edge perception pipeline begins with sensor acquisition from cameras, depth sensors, LiDAR, IMU, GNSS, and other sources. These measurements are processed locally to generate object detection, semantic segmentation, localization, mapping, tracking, and scene-understanding results. A unified robotics architecture places these functions within a broader perception and understanding layer together with sensor fusion and SLAM, while connecting them to a world model and digital twin when required. This structure allows scene understanding to remain closely connected to the robot's physical operating state.

Computational efficiency becomes critical because 3D scene understanding can involve point-cloud processing, volumetric reconstruction, neural inference, semantic interpretation, and temporal reasoning simultaneously. Edge systems therefore benefit from heterogeneous computing using CPU, GPU, NPU, and specialized accelerators according to workload characteristics. Neural networks can be optimized through quantization, operator fusion, memory-aware scheduling, reduced precision, and hardware-specific compilation. The objective is to maintain sufficient perception accuracy while keeping inference latency and power consumption within the robot's operational limits.

Memory management is particularly important for large 3D representations. Point clouds, voxel maps, TSDF volumes, Gaussian primitives, semantic maps, and scene graphs can consume substantial memory when maintained at high resolution. An edge system can reduce this burden through spatial partitioning, local reconstruction, level-of-detail control, compression, pruning, and region-of-interest processing. Rather than maintaining every part of the environment at maximum resolution, computational resources can be concentrated around the robot, active objects, navigation corridors, or regions directly relevant to the current task.

Real-time behavior requires more than high average inference throughput. A robotic system must maintain predictable execution timing across sensing, perception, world-model updates, planning, and control. The broader platform architecture therefore separates different processing rates, allowing perception and scene understanding to operate at appropriate frequencies while lower-level control operates at substantially higher rates. This multi-rate structure prevents computationally intensive 3D understanding from unnecessarily interfering with safety-critical control and actuator loops.

Model optimization is another major requirement for edge deployment. Training can be performed using standard machine-learning frameworks, while deployment requires conversion into hardware-compatible representations and execution through an accelerator-specific software stack. Quantization is especially important because integer arithmetic can substantially improve throughput and energy efficiency. Calibration or quantization-aware training should be performed carefully so that reductions in numerical precision do not produce unacceptable degradation in object detection, segmentation, depth estimation, or scene-understanding accuracy.

3D scene understanding often requires several neural models rather than a single network. Object detection may identify relevant entities, semantic or panoptic segmentation may describe their spatial extent, depth estimation may provide geometry, and tracking may maintain temporal identity. Running every model continuously at maximum resolution can exceed edge resources. A practical architecture therefore uses task-aware scheduling, shared feature extraction, selective inference, model cascading, and region-of-interest processing so that computationally expensive operations are activated when their information is actually needed.

The deployment architecture should also maintain a clear boundary between high-level AI inference and deterministic robotic execution. Scene understanding may generate semantic objects, spatial relationships, occupancy information, or task-relevant hypotheses, while navigation, manipulation, motion control, safety supervision, and hardware interfaces remain responsible for converting these results into physical actions. The robotics software architecture places perception and understanding above control and execution, with middleware such as ROS 2 and DDS providing standardized communication between distributed components.

Edge deployment does not imply that every computation must remain permanently on the robot. Cloud-edge collaboration can separate latency-sensitive functions from computationally intensive or globally coordinated functions. Real-time perception, sensor processing, local autonomy, and safety-related functions can remain on the edge, while model training, fleet management, digital twins, data analytics, knowledge management, and software deployment can be coordinated through cloud or on-premise infrastructure. This division allows local operation to continue even when communication with remote infrastructure is delayed or temporarily unavailable.

For 3D scene understanding, world-model integration provides an additional layer above individual perception outputs. The edge system can maintain locally relevant objects, spatial relationships, occupancy information, scene state, and temporal changes while synchronizing selected information with a larger world model or digital twin. This approach avoids transferring every raw sensor stream to a remote server. Instead, the robot can transmit structured information, selected observations, events, or compressed representations that are useful for fleet-level reasoning and long-term environmental knowledge.

Reliability and observability are essential because edge perception operates continuously in physical environments. Sensor health, GPU or NPU utilization, memory consumption, inference latency, localization confidence, communication latency, thermal conditions, and model stability should be monitored during operation. ROS 2 provides mechanisms for distributed communication, diagnostics, recording, and runtime inspection that can support this monitoring architecture. Recorded sensor and system data can also be replayed for failure analysis, model improvement, and simulation-based validation.

Safety requires that scene-understanding outputs never be treated as infallible commands. Object detection, segmentation, depth estimation, and scene interpretation can fail because of occlusion, lighting, sensor degradation, unusual objects, dynamic environments, or model uncertainty. Edge systems should therefore expose confidence and validity information and provide defined fallback behaviors. Safety supervisors, runtime monitors, collision checking, and lower-level control constraints can prevent uncertain high-level perception results from directly producing unsafe physical actions.

Deployment validation should evaluate both AI accuracy and robotic system performance. Conventional metrics such as detection accuracy, segmentation quality, depth error, tracking consistency, and scene-understanding accuracy should be evaluated together with latency, throughput, memory consumption, thermal behavior, power usage, and recovery behavior. Field testing is particularly important because a model that performs well on a benchmark may behave differently under actual sensor noise, vibration, illumination changes, occlusion, network interruptions, and computational contention.

Within the broader robotics software architecture, edge-based 3D scene understanding belongs to the perception and understanding layer and interacts with the world model, mission and orchestration, control and execution, software infrastructure, and hardware abstraction layers. The architecture is intended to remain unified across different robot platforms while allowing the edge compute configuration to vary according to embodiment and workload. This supports indoor AMRs, outdoor vehicles, inspection robots, manipulators, quadrupeds, humanoids, and other platforms through standardized interfaces rather than completely independent software stacks.

The practical objective of edge deployment is therefore to transform 3D scene understanding from a computationally expensive perception capability into a persistent real-time robotic function. The system should select appropriate spatial representations, optimize AI inference, manage memory and thermal resources, maintain predictable latency, expose uncertainty, and connect perception results to navigation, manipulation, safety, and mission execution. When combined with cloud-edge collaboration, world models, standardized middleware, diagnostics, and continuous model improvement, edge 3D scene understanding becomes a foundation for sustained local autonomy across heterogeneous physical AI robots.


Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go

Original GitHub manuscript: https://github.com/Statera-Library/08_Vol_14_Perception_and_Sensor_Fusion/blob/main/Chapter_07_3D_Scene_Understanding/Chapter_07_3D_Scene_Understanding_EN.md

Top comments (0)