DEV Community

Richard Minsoo Go
Richard Minsoo Go

Posted on

World Models for Physical AI — Ch.05: BEV Representation for World Models [EN]

STATERA Library · Series 09 · Volume 07 · Chapter 05 · EN
Source: https://github.com/Statera-Library/09_Vol_07_World_Models_for_Physical_AI/blob/main/Chapter_05_BEV_Representation_for_World_Models/Chapter_05_BEV_Representation_for_World_Models_EN.md

Volume 07. World Models for Physical AI

Chapter 05. BEV Representation for World Models

05.01. Why BEV for Physical AI

Physical AI must reason about a world that exists in three-dimensional space while making decisions that are usually expressed relative to the ground, such as moving forward, turning, avoiding obstacles, approaching objects, or selecting traversable regions. Bird's-Eye View (BEV) representation provides a common spatial frame in which observations from different sensors can be transformed into an environment-centered or agent-centered geometric representation suitable for prediction, planning, and action.

Raw camera images are perspective projections whose geometry changes with viewpoint, depth, camera orientation, and object distance. The same physical object may occupy very different image regions as the robot moves, making direct spatial reasoning difficult. BEV reduces this viewpoint dependency by projecting relevant scene information onto a ground-aligned coordinate system, allowing positions, distances, directions, free space, and motion to be represented more consistently across time.

This property is particularly important for Physical AI because intelligent behavior depends not only on recognizing what objects are, but also on understanding where they are relative to the embodied agent. A robot must distinguish whether a pedestrian is three meters ahead or ten meters away, whether an obstacle blocks its planned trajectory, and whether sufficient free space exists for maneuvering. BEV organizes such relationships in a metric spatial structure that naturally supports physical reasoning.

BEV also creates a convenient intermediate representation between perception and world modeling. Camera, LiDAR, radar, depth sensors, maps, and other spatial observations may have very different measurement characteristics, yet their information can be transformed into a shared top-down coordinate frame. Once aligned, complementary evidence can be fused into a unified representation describing geometry, semantics, occupancy, motion, traversability, and other properties of the surrounding environment.

For a world model, this shared coordinate system is valuable because prediction requires consistent state representation over time. If every observation remains tied to the instantaneous viewpoint of individual sensors, ego motion can create large apparent changes even when the environment itself is static. BEV can compensate for much of this viewpoint variation through geometric transformation and ego-motion alignment, making temporal changes more closely correspond to meaningful changes in the physical world.

A temporally aligned BEV therefore provides a natural substrate for memory. Past observations can be transformed into the current reference frame and accumulated to preserve information that may no longer be directly visible. An object temporarily hidden behind another object, a wall observed several seconds earlier, or a traversable region outside the current camera field of view can remain represented within a persistent spatial memory rather than disappearing immediately from the internal state.

The representation is equally useful for separating static structure from dynamic behavior. Roads, walls, shelves, terrain boundaries, and fixed obstacles may form relatively stable spatial layers, while pedestrians, vehicles, robots, machinery, and other moving entities can be represented with position, velocity, orientation, and predicted motion. A world model can consequently learn not only what occupies space now, but also how different components of that spatial state are likely to evolve.

This makes BEV especially compatible with future prediction. Instead of generating future camera pixels containing large amounts of appearance detail, a predictive world model can estimate future BEV states that emphasize variables directly relevant to physical interaction. Future occupancy, object trajectories, free-space evolution, semantic regions, motion fields, and traversability can be predicted over several horizons, producing compact hypotheses about how the actionable environment may change.

BEV representation also provides an important bridge from prediction to planning. Navigation and motion-planning algorithms operate naturally on spatial quantities such as paths, distances, collision regions, costs, velocities, and reachable areas. When the world model predicts these quantities in a ground-aligned frame, candidate robot trajectories can be evaluated directly against predicted environmental states without repeatedly translating between unrelated sensor coordinate systems and planning representations.

For autonomous vehicles, AMRs, quadrupeds, mobile manipulators, and other mobile Physical AI systems, this connection can simplify the perception--prediction--planning interface. The perception system constructs a spatial state, the world model estimates how that state may evolve, and the planner evaluates possible actions within the same or closely related coordinates. BEV thus acts as a computational meeting point where sensing, learned representation, dynamics prediction, and control-oriented reasoning can interact.

The advantages extend beyond purely geometric mapping. Individual BEV cells or learned BEV features can encode semantic classes, object identities, occupancy probability, elevation, surface type, motion, uncertainty, or learned latent features. Consequently, BEV should not be understood merely as a conventional two-dimensional map. Modern neural BEV representations can function as dense feature spaces in which explicit geometric information and learned semantic information coexist.

This distinction is important for world models because the internal representation does not need to reconstruct every detail of the observable world. It should preserve information useful for predicting consequences and selecting actions. Texture on a distant building may be irrelevant to navigation, while the building boundary is essential. Similarly, precise visual appearance of vegetation may matter less than whether the region is traversable. BEV provides a structure in which task-relevant physical properties can receive greater representational priority.

BEV can also support multimodal robustness. Cameras provide rich semantic and appearance information but can be sensitive to illumination and weather, while LiDAR provides direct geometric measurements and radar can provide useful range and motion information under difficult visual conditions. Projecting these modalities into a compatible spatial frame enables the system to exploit their complementary strengths and reduces dependence on any single sensor when constructing its internal representation of the environment.

However, BEV is not a complete representation of the physical world. Projection onto a ground-oriented plane can lose vertical structure, overlapping geometry, fine object shape, and information important for manipulation or highly three-dimensional environments. Multi-level BEV features, height channels, voxel representations, 3D occupancy fields, object-centric representations, or latent 3D features may therefore complement BEV when robots must reason about stairs, shelves, articulated objects, aerial space, or complex manipulation.

The most useful interpretation is therefore not that BEV replaces all other world representations, but that it provides a highly effective spatial interface for embodied intelligence. It converts heterogeneous observations into a relatively stable coordinate system, supports temporal memory and multimodal fusion, exposes physical relationships relevant to motion, and connects naturally to future prediction and planning. These properties explain why BEV has become a foundational representation for many mobile Physical AI world-model architectures.

Within a broader Physical AI architecture, BEV can serve as one layer of a hierarchical world state. Sensor-specific encoders first extract observations, geometric or learned transformations place relevant information into BEV, temporal models maintain history, and predictive models estimate future states. Higher-level reasoning can then operate on objects, semantics, goals, and causal relationships, while lower-level planners use detailed spatial information for safe and dynamically feasible actions.

Ultimately, the importance of BEV comes from the requirement that Physical AI must convert perception into actionable understanding of space. Intelligence embodied in a robot cannot rely only on recognizing visual patterns; it must continuously estimate where things are, how they move, where the agent can go, and what may happen after an action. BEV provides a practical representation in which these questions can be expressed together, making it a powerful foundation for spatially grounded world models.

05.02. Camera to BEV Representation

Camera-to-BEV representation begins with the recognition that a camera observes the physical world through perspective projection rather than directly in a ground-aligned coordinate system. Image coordinates describe where visual patterns appear on the sensor, but they do not directly provide the metric relationships required for physical reasoning. The same object can appear at different image locations and scales as the robot moves, even though its position relative to the environment changes smoothly. Camera-to-BEV processing therefore transforms visual observations into a spatial representation that is more suitable for embodied intelligence and world modeling.

The process begins with one or more camera images that capture complementary portions of the surrounding environment. A camera encoder converts these images into feature representations containing visual information such as edges, textures, object appearance, semantic categories, and spatial cues. Modern systems commonly use convolutional or transformer-based backbones to extract multi-scale features because objects and environmental structures appear at substantially different resolutions. The objective is not simply to recognize objects, but to preserve visual information that can later be associated with physical position, geometry, and motion.

The central challenge is to infer three-dimensional structure from perspective images. Unlike LiDAR, a monocular camera does not directly measure depth for every pixel, so the system must estimate how image features relate to distance and spatial location. Depth estimation, camera calibration, geometric projection, learned depth distributions, or implicit 3D representations can provide this missing relationship. The resulting transformation maps image features from perspective coordinates into locations in a ground-aligned reference frame, allowing information originally distributed across the image plane to be organized according to physical space.

Camera calibration provides the geometric foundation for this transformation. Intrinsic parameters describe properties such as focal length and principal point, while extrinsic parameters describe the camera pose relative to the robot or vehicle. These parameters allow image rays to be related to the physical coordinate system. When multiple cameras are used, their individual observations can be transformed into a common coordinate frame. Accurate calibration is therefore important because small geometric errors can cause features from different viewpoints to become spatially misaligned in the resulting BEV representation.

A common conceptual pipeline is image acquisition, visual feature extraction, depth or geometric estimation, coordinate transformation, and BEV feature aggregation. Each image feature can be associated with a distribution of possible locations in the physical environment rather than being forced immediately into a single point. The system can then accumulate these projected features into BEV cells or learned spatial features. This process converts perspective-dependent visual information into an environment-centered representation in which neighboring cells correspond more directly to neighboring physical regions.

Learned camera-to-BEV methods can perform this transformation without explicitly reconstructing a complete three-dimensional scene. Instead, the network learns relationships between image features and spatial locations using training data, geometric supervision, temporal consistency, or combinations of these signals. Depth-aware projection can explicitly model where features should be placed, while fully learned approaches can encode the transformation into neural attention or latent feature mechanisms. The important objective is to preserve spatial relationships that are useful for downstream prediction, planning, and action.

For multiple cameras, camera-to-BEV processing becomes a spatial fusion problem. Front, rear, side, or panoramic cameras observe different regions and viewing angles, and their image features must be aligned before they can form a coherent environmental representation. A shared BEV coordinate system provides the common reference required for this alignment. Overlapping observations can reinforce the representation, while complementary views can fill regions that are invisible from another camera. This is particularly useful for mobile robots operating in cluttered environments where a single viewpoint may leave substantial blind areas.

The resulting BEV representation can contain more than object locations. Different spatial cells or learned features may encode semantic information, estimated depth, occupancy, surface properties, free space, object boundaries, motion-related features, and uncertainty. This allows the camera-to-BEV transformation to produce a spatial feature field rather than merely a conventional top-down image. Such a representation can subsequently support semantic and geometric reasoning while remaining compatible with the prediction-oriented architecture of a world model.

Temporal information further improves camera-to-BEV representation. As the robot moves, the camera viewpoint changes continuously, causing stationary objects to move substantially within image coordinates. Ego-motion estimation allows observations from different time steps to be transformed into a consistent reference frame. Once aligned, information from previous observations can be accumulated into a temporal BEV representation. This makes the internal state less dependent on instantaneous visibility and allows the world model to distinguish actual environmental change from apparent image motion caused by the robot\'s own movement.

Camera-to-BEV also establishes an important connection between perception and predictive world modeling. Once visual observations have been transformed into a consistent spatial state, temporal models can learn how objects, free space, semantic regions, and other environmental properties evolve. The world model can then predict future BEV states rather than attempting to reproduce every pixel of a future camera image. This shifts the representation toward quantities that are more directly relevant to physical interaction, such as occupancy, trajectories, traversability, and spatial relationships.

The approach nevertheless has important limitations. Camera-based BEV depends strongly on the quality of depth estimation, calibration, camera placement, illumination, visibility, and scene understanding. Occlusion can hide important geometry, and monocular depth can remain ambiguous in regions with weak visual cues. Projection errors can accumulate when the estimated depth or camera pose is inaccurate. For highly three-dimensional environments, a purely planar BEV may also discard vertical relationships that are important for stairs, shelves, manipulation, aerial motion, or objects extending significantly above or below the ground plane.

For this reason, camera-to-BEV should be regarded as a spatial representation layer rather than a complete description of the physical world. It provides a practical bridge from perspective vision to a ground-aligned world state, while complementary representations can preserve information that BEV alone cannot express. Within the broader World Model architecture defined for this volume, camera-to-BEV naturally connects perception with semantic and geometric BEV, temporal BEV memory, future BEV prediction, and ultimately navigation and planning.

05.03. LiDAR to BEV Representation

LiDAR-to-BEV representation begins with a fundamentally different sensing principle from camera-to-BEV conversion. A LiDAR sensor directly measures ranges to surrounding surfaces and produces a three-dimensional point cloud in which each point contains spatial coordinates and may also include intensity or other attributes. Because the measurements already exist in metric 3D space, LiDAR does not require the same depth inference process as a camera. The central challenge is instead to organize sparse, irregularly distributed points into a structured representation that can support spatial reasoning and world modeling.

The raw LiDAR point cloud provides accurate geometric information, but its representation is not immediately convenient for neural networks or planning systems. Points are distributed unevenly according to distance, surface orientation, sensor resolution, occlusion, and the scanning pattern of the sensor. Nearby surfaces may contain many points while distant objects may be represented by only a small number. A LiDAR-to-BEV pipeline therefore transforms the point cloud from an irregular set of measurements into a structured spatial representation in which physical locations can be processed consistently.

The first stage is normally LiDAR preprocessing and coordinate transformation. Measurements may contain noise, outliers, ground reflections, multipath effects, or points that are not useful for the intended task. Filtering and range selection can remove unreliable measurements while preserving important environmental structure. The point cloud is then expressed in a suitable coordinate frame, often relative to the robot or vehicle. Accurate sensor calibration and pose estimation are essential because spatial errors introduced before BEV projection can propagate through the entire representation.

A major advantage of LiDAR is that each point already provides a direct estimate of its three-dimensional position. A point can therefore be represented by coordinates such as x, y, and z, together with additional attributes when available. For BEV construction, the horizontal coordinates are commonly used to determine the corresponding ground-plane cell, while height and other measurements can be accumulated as additional features. This allows the system to retain useful vertical information even though the primary spatial organization is performed in a top-down coordinate system.

Voxelization and pillarization provide two common mechanisms for converting LiDAR points into structured features. Voxelization divides three-dimensional space into small volumetric cells and aggregates points within each cell, producing a sparse or dense 3D representation. Pillar-based approaches instead organize points according to horizontal cells while retaining information about the points\' vertical distribution inside each cell. These approaches reduce the irregular point cloud into a representation that can be processed efficiently while preserving the spatial relationships required for BEV reasoning.

The aggregation process can produce multiple BEV feature channels rather than a single occupancy map. A cell may encode point density, maximum or minimum height, mean height, intensity, vertical distribution, surface characteristics, learned point features, or estimated occupancy. Such features allow the BEV representation to describe more than whether space is occupied. It can also provide information about the geometry and physical properties of structures, vehicles, pedestrians, terrain, and other objects. The resulting representation becomes a compact spatial feature field for downstream intelligence.

LiDAR BEV is particularly useful for geometric reasoning because the representation is naturally metric and largely independent of visual appearance. Roads, walls, curbs, shelves, vehicles, terrain boundaries, and obstacles can be represented according to their physical locations rather than their image positions. This makes distance, relative position, spatial extent, and free-space relationships easier to reason about. For mobile Physical AI, such information is directly relevant to localization, mapping, obstacle avoidance, navigation, trajectory planning, and collision checking.

However, LiDAR measurements are inherently sparse and incomplete. Surfaces that are hidden from the sensor remain unobserved, and distant or small objects may contain relatively few points. Thin structures can be difficult to represent, while reflective or absorptive materials may produce unreliable returns. The point cloud also changes as the robot moves, so a single scan represents only a partial observation of the environment at one moment. Consequently, LiDAR BEV should be understood as a spatial observation that can be strengthened through temporal accumulation, sensor fusion, and world-model prediction.

Ego-motion compensation is therefore an important component of LiDAR-to-BEV processing. When the robot moves, successive point clouds are naturally expressed from different sensor poses. Localization and odometry information can transform previous measurements into a common reference frame, allowing multiple scans to be accumulated. Temporal aggregation increases spatial coverage and point density while preserving structures that may disappear from the current scan because of occlusion or sensor viewpoint. This creates a richer BEV state for mapping and subsequent world-model processing.

Dynamic objects introduce an additional challenge because temporal accumulation can incorrectly smear moving objects across multiple locations. A world model must therefore distinguish changes caused by ego motion from changes caused by environmental dynamics. Object tracking, motion estimation, temporal attention, or learned dynamic representations can help separate relatively stable structures from moving agents. In this form, LiDAR BEV can contain not only static geometry but also object position, velocity, orientation, motion history, and uncertainty, providing a foundation for dynamic state estimation.

LiDAR BEV also provides a natural interface to occupancy and future-state prediction. The current BEV can describe observed occupied regions, free space, uncertain regions, and geometric structures, while a predictive world model can estimate how these spatial states may evolve. Future occupancy, object trajectories, free-space changes, traversability, and motion fields can then be represented in the same ground-aligned coordinate system. This allows prediction to remain closely connected to the physical quantities required by navigation and planning rather than requiring reconstruction of future sensor measurements.

For planning and control, LiDAR BEV can provide a spatial representation that is directly compatible with trajectory evaluation. Candidate paths can be compared against predicted obstacles, occupied regions, free space, terrain boundaries, and other spatial constraints. A planner can therefore evaluate collision risk, clearance, reachability, and path cost using information derived from the same coordinate system used by the world model. This creates a relatively direct chain from LiDAR observation to spatial state estimation, future prediction, planning, and action.

LiDAR BEV nevertheless does not completely describe the physical world. Compared with cameras, LiDAR generally provides less direct information about visual appearance and fine semantic distinctions, although modern neural networks can learn substantial semantic information from point clouds and multimodal fusion can compensate for missing appearance information. Complex vertical structures may also be compressed when using a single-layer BEV. For manipulation, stairs, shelves, articulated objects, or environments with significant three-dimensional structure, voxel representations, multi-height BEV features, occupancy fields, or object-centric 3D representations may therefore complement the primary BEV state.

The most effective interpretation is consequently to treat LiDAR-to-BEV as a geometric foundation within a larger Physical AI world model rather than as an isolated perception technique. LiDAR provides accurate metric observations, preprocessing and coordinate alignment establish a consistent spatial frame, voxel or pillar encoding converts irregular measurements into structured features, and BEV aggregation produces a compact representation of geometry and occupancy. Temporal processing can then build persistent spatial memory, while multimodal fusion can add semantic and complementary sensing information.

05.04. Multimodal BEV Fusion

Multimodal BEV fusion combines complementary observations from cameras, LiDAR, radar, depth sensors, IMU, and other sources within a common spatial representation. Each modality observes different aspects of the physical world: cameras provide rich appearance and semantic information, LiDAR provides accurate geometric structure, radar provides range and motion information with greater robustness in difficult weather, and inertial or proprioceptive sensors provide motion and state information. BEV offers a practical common frame in which these heterogeneous observations can be spatially aligned and integrated.

The central objective is not simply to combine more sensor data, but to construct a more complete and reliable representation of the physical environment. Individual modalities have different strengths, failure modes, resolutions, and measurement characteristics. A camera may recognize an object clearly while providing uncertain depth, whereas LiDAR may measure its geometry accurately but provide weaker semantic appearance information. Radar may detect motion when visual sensing becomes unreliable. Multimodal BEV fusion therefore seeks to preserve complementary information while reducing the dependence on any single sensing modality.

Before fusion can occur, spatial and temporal alignment must be established. Each sensor operates in its own coordinate system and often at a different sampling frequency or with different processing delays. Extrinsic calibration transforms measurements into a common coordinate frame, while intrinsic calibration corrects sensor-specific geometric characteristics. Temporal synchronization, interpolation, buffering, and delay compensation help ensure that observations corresponding to the same physical state are associated correctly. Without accurate alignment, even highly informative sensors can produce contradictory or spatially blurred BEV representations.

The camera branch typically extracts semantic and visual features before projection into BEV. The LiDAR branch transforms point clouds into geometric features through voxelization, pillarization, or other spatial encoders, while radar can produce range, velocity, and detection features. Depth sensors contribute explicit distance structure, and IMU or odometry information can provide ego-motion estimates for temporal alignment. Each modality may therefore use a specialized encoder before its information enters the common BEV representation. This preserves modality-specific characteristics while preparing the outputs for spatial fusion.

Multimodal fusion can occur at several levels. Early fusion combines relatively raw or low-level representations after geometric projection and alignment, allowing direct interaction between modalities but potentially creating high computational and calibration demands. Intermediate or feature-level fusion first encodes each modality separately and then combines their learned features in a shared BEV or latent space. This approach provides a useful balance between computational efficiency and cross-modal interaction. Late fusion keeps modality-specific processing largely independent and combines predictions or decisions, improving modularity and robustness but potentially losing fine-grained spatial relationships.

For BEV-based Physical AI, intermediate feature-level fusion is particularly important because different sensors can be converted into compatible spatial features before being integrated. Camera features may contribute semantic identity and appearance, LiDAR features may provide precise boundaries and elevation, and radar features may contribute velocity or motion evidence. A fusion network can combine these signals through concatenation, addition, learned gating, attention, or cross-attention. The resulting BEV features can therefore represent physical regions using multiple complementary sources of evidence rather than relying on a single sensor.

Attention and cross-attention provide a flexible mechanism for learning relationships between modalities. Instead of treating all sensor information as equally important, the network can learn to focus on features that are relevant to the current spatial or semantic context. One modality can query another modality for supporting evidence, such as using visual features to identify an object represented geometrically by LiDAR or using radar motion information to reinforce the dynamic interpretation of a visually detected object. This allows fusion to become context-dependent rather than being limited to fixed feature concatenation.

Reliability-aware fusion is also important because sensor quality changes with environmental conditions. Camera observations may degrade because of darkness, glare, rain, fog, or occlusion, while LiDAR can become sparse or unreliable for particular surfaces and radar can contain clutter, multipath effects, or ambiguous detections. A robust fusion system can estimate modality-specific confidence or uncertainty and adjust the contribution of each source accordingly. Rather than simply averaging sensor information, the system can emphasize reliable evidence and reduce the influence of degraded or contradictory measurements.

After fusion, the shared BEV representation can contain geometric, semantic, dynamic, and probabilistic information within the same spatial framework. Individual cells or learned features may encode occupancy, object category, object identity, height, surface characteristics, velocity, traversability, uncertainty, or latent features. The representation can therefore serve as a common spatial state for downstream tasks including detection, segmentation, localization, tracking, prediction, navigation, and planning. Importantly, the shared representation does not require every modality to contain identical information; it allows different modalities to contribute different evidence to the same physical state.

Temporal alignment extends multimodal fusion from a single observation into a persistent world representation. Sensors operate at different frequencies, and observations may arrive with different delays, so the system must associate measurements with appropriate physical states. Once aligned, BEV features from multiple time steps can be accumulated or processed by temporal models. Ego-motion compensation allows static structures to remain spatially stable while moving objects can be modeled separately. This creates a foundation for temporal BEV memory and world-model prediction, where the system can estimate how the fused spatial state will evolve.

Multimodal BEV fusion also improves robustness through redundancy. When one sensor becomes uncertain or temporarily unavailable, other modalities can provide supporting evidence. A camera may lose reliable information in darkness while LiDAR maintains geometric observations; LiDAR may provide sparse evidence while camera features preserve semantic context; radar may continue providing useful motion information under conditions that degrade both vision and depth sensing. Missing-modality handling and graceful degradation therefore become important capabilities for Physical AI systems operating continuously in real environments.

The fused BEV representation ultimately provides a bridge between heterogeneous perception and predictive world modeling. A world model can use the current multimodal BEV state to estimate future occupancy, object trajectories, free-space evolution, traversability, motion fields, and uncertainty. Planning algorithms can then evaluate candidate trajectories against these predicted states using the same ground-aligned spatial representation. Multimodal BEV fusion therefore connects sensing, spatial understanding, temporal memory, future prediction, planning, and action into a coherent pipeline for embodied intelligence. The broader architecture places this process within the BEV representation chapter alongside camera-to-BEV, LiDAR-to-BEV, semantic and geometric BEV, dynamic object representation, temporal memory, future prediction, and navigation and planning.

05.05. Semantic and Geometric BEV

Semantic and geometric BEV represent two complementary aspects of the physical environment within a common ground-aligned spatial framework. Geometric BEV describes where things are and how space is physically structured, including position, distance, shape, height, occupancy, surfaces, and spatial relationships. Semantic BEV describes what those regions or objects represent, such as roads, sidewalks, buildings, vehicles, pedestrians, vegetation, obstacles, or traversable areas. Their combination allows a Physical AI system to reason about both the physical structure of the environment and the meaning associated with that structure.

Geometric BEV provides the spatial foundation required for physical interaction. A robot must estimate where obstacles are located, how much space they occupy, how far they are from the agent, and whether a region is free or occupied. These quantities can be represented using metric coordinates, occupancy grids, point clouds, voxel-derived features, surface information, or learned spatial features. Because BEV organizes these quantities according to a common ground-aligned coordinate system, spatial relationships become easier to preserve across different viewpoints and sensor observations. This makes geometric BEV particularly suitable for localization, mapping, collision checking, navigation, and motion planning.

Semantic BEV adds an interpretation layer to the same spatial structure. Instead of representing a cell only as occupied or free, the system can associate it with a semantic class, object category, functional region, or task-relevant property. A region may therefore be identified as road, sidewalk, vegetation, building, vehicle, pedestrian, or another environmental component. Semantic information can also describe affordances such as whether an area is drivable, walkable, manipulable, restricted, or otherwise relevant to the robot\'s current task. This allows spatial reasoning to move beyond geometry toward meaningful environmental understanding.

The relationship between semantic and geometric information is especially important because neither representation is sufficient by itself. Geometry can indicate that a region contains an elevated structure or an obstacle, but it may not explain what that structure represents or how the robot should interpret it. Semantics can identify an object as a pedestrian or a vehicle, but without accurate position, extent, orientation, and distance, the robot cannot safely reason about interaction. A unified BEV state therefore associates semantic meaning with metric spatial structure, allowing the world model to represent both "what is there" and "where it is."

Semantic and geometric BEV can be constructed from multiple sensing modalities. Camera features are particularly valuable for semantic recognition and appearance-based understanding, while LiDAR provides accurate three-dimensional geometry and depth structure. Radar can contribute range and motion evidence, and depth sensors can provide additional spatial measurements. After modality-specific encoding and spatial alignment, these observations can be projected or transformed into a shared BEV space. The resulting representation can combine semantic labels, geometric features, occupancy, height, motion, traversability, uncertainty, and learned latent features within the same spatial framework.

The representation can be organized as multiple feature channels rather than a single map. Geometric channels may describe occupancy, height or elevation, surface structure, distance, and spatial boundaries, while semantic channels can encode classes, instances, object identities, and environmental categories. Other channels can represent dynamic objects, velocity, motion, traversability, uncertainty, or learned features. This layered structure allows downstream models to select or combine the information required for a particular task. A navigation system may emphasize free space and obstacles, while a manipulation system may require object identity, geometry, and precise spatial relationships.

A key advantage of combining semantics and geometry is that the representation becomes more directly useful for world modeling. A world model does not need to preserve every visual detail of the environment if those details do not influence future physical states. Instead, it should preserve information that supports prediction and action. The boundary of a building may be more important than its texture, and the traversability of a terrain region may matter more than its exact visual appearance. Semantic and geometric BEV provides a structured way to retain these task-relevant properties while reducing the dependence on raw sensor appearance.

Temporal processing further strengthens the combined representation. As the robot moves, observations from different time steps must be transformed into a consistent reference frame using ego-motion estimation. Once aligned, semantic and geometric BEV states can be accumulated into temporal memory. Static structures such as roads, walls, terrain, and buildings can remain persistent, while dynamic objects such as pedestrians, vehicles, robots, or machinery can be represented through changing position, velocity, orientation, and trajectories. This separation allows the world model to distinguish stable environmental structure from dynamic behavior and to maintain information that may temporarily disappear from the current observation.

The combined BEV state also provides a foundation for prediction. A world model can estimate how geometric occupancy and semantic states will evolve over future horizons. Future prediction may include changing object positions, trajectories, free-space evolution, traversability, motion fields, semantic changes, and uncertainty. Because these quantities remain expressed in the same spatial representation as the current state, the predicted world can be compared directly with candidate actions. This creates a continuous connection between current perception, internal world state, future prediction, and decision making.

Semantic and geometric BEV is particularly valuable for planning because planning requires both spatial feasibility and semantic interpretation. A trajectory must remain within physically available space, avoid occupied regions, and satisfy the robot\'s kinematic and dynamic constraints. At the same time, the planner may need to understand that one region is a roadway, another is a pedestrian area, and another is restricted or non-traversable terrain. The combination therefore allows planning to evaluate collision risk, clearance, reachability, path cost, and task-specific constraints within a common spatial representation.

Despite these advantages, semantic and geometric BEV remains an abstraction of the physical world rather than a complete representation of it. A single ground-plane representation can lose fine vertical structure, overlapping geometry, and detailed three-dimensional relationships. Depth ambiguity, occlusion, calibration errors, sensor sparsity, and semantic misclassification can also introduce uncertainty. For environments involving stairs, shelves, articulated objects, manipulation, or substantial aerial space, multi-level BEV, 3D voxels, occupancy fields, object-centric representations, or latent 3D features may therefore be required to complement the primary BEV state.

Within the broader Physical AI world-model architecture, semantic and geometric BEV acts as a bridge between multimodal perception and higher-level physical reasoning. Sensor-specific encoders first extract observations, camera and LiDAR information can be transformed into compatible spatial features, and multimodal fusion produces a unified BEV state. Temporal memory then maintains the state over time, predictive models estimate future spatial and semantic evolution, and planners evaluate actions against the predicted state. In this architecture, semantic information answers what exists and what it means, geometric information describes where it exists and how it is structured, while the world model learns how this combined state changes and supports intelligent action. The chapter structure places this topic directly after camera-to-BEV, LiDAR-to-BEV, and multimodal BEV fusion, and before dynamic object representation, temporal BEV memory, future BEV prediction, and BEV-based navigation and planning.

05.06. Dynamic Object Representation in BEV

Dynamic object representation in BEV extends the spatial representation from describing where static structures exist to describing how moving entities occupy and evolve through space. Roads, walls, buildings, and terrain boundaries can remain relatively stable, while pedestrians, vehicles, robots, machinery, and other agents change position and orientation over time. A Physical AI system therefore needs a representation that preserves not only object location, but also velocity, direction, motion history, uncertainty, and relationships with the surrounding spatial state.

A dynamic object in BEV can be represented through a combination of geometric, semantic, and temporal attributes. Position describes where the object is located in the ground-aligned coordinate system, while size and shape describe its spatial extent. Orientation indicates the direction in which the object is facing or moving, and velocity describes its current motion. Additional attributes can include acceleration, object category, identity, confidence, and predicted trajectory. Together, these variables form an object state that can be updated continuously as new sensor observations arrive.

The BEV coordinate system makes dynamic objects particularly suitable for physical reasoning because their motion can be expressed directly relative to the robot and surrounding environment. A pedestrian moving across a road, a vehicle approaching an intersection, or another robot crossing a workspace can be represented as a sequence of spatial states rather than as changing image patterns. This allows the system to reason about relative distance, closing speed, future position, collision risk, and available free space within the same coordinate framework used for static geometry and traversability.

Dynamic object representation normally begins with detection or segmentation from one or more sensing modalities. Camera observations provide appearance and semantic information, while LiDAR supplies accurate spatial geometry and radar can provide useful range and velocity evidence. Multimodal BEV fusion can combine these observations into a unified spatial state. Object candidates can then be associated with spatial regions or learned BEV features, allowing the system to distinguish individual dynamic entities from the surrounding static background. The resulting representation can be maintained as an object-aware component of the BEV state.

Object tracking is required because detection at a single time step does not establish temporal identity. The system must determine whether an observation at the current time corresponds to an object observed previously, whether a new object has appeared, or whether an existing object has disappeared temporarily. Tracking therefore links observations across time and maintains persistent identities. Position, velocity, orientation, and other state variables can be updated as new measurements arrive, while confidence can indicate how reliable the current estimate remains under occlusion, sparse sensing, or ambiguous observations.

A particularly important distinction is between ego motion and object motion. When the robot moves, stationary objects appear to move relative to the sensor, even though their physical state has not changed. Ego-motion compensation transforms observations into a consistent reference frame so that static structures remain stable in BEV. After this compensation, residual changes can be interpreted more reliably as environmental dynamics. This separation is essential for preventing the robot\'s own motion from being incorrectly learned as object motion and for building a coherent dynamic world state.

Dynamic objects can be represented explicitly as entities or implicitly as spatial feature patterns. An object-centric representation stores attributes such as identity, category, position, velocity, orientation, size, and uncertainty for each tracked entity. A dense BEV representation can instead encode dynamic information directly across spatial cells using occupancy, motion, velocity, and learned latent features. Hybrid representations can combine both approaches, using dense BEV features for spatial context while maintaining explicit object states for agents that require individual tracking and prediction.

Motion history provides information that cannot be obtained reliably from a single observation. A sequence of past object states can reveal whether an agent is accelerating, slowing down, turning, stopping, or maintaining a stable trajectory. Historical information also helps distinguish temporary measurement noise from meaningful changes in behavior. Within a temporal BEV system, these histories can be stored through recurrent states, temporal attention, feature aggregation, or explicit trajectory buffers. The objective is to provide the world model with sufficient temporal context to estimate how dynamic states are evolving.

Uncertainty is an essential part of dynamic object representation because future motion is rarely deterministic. Occlusion, sensor noise, limited observation frequency, incorrect association, ambiguous intent, and interactions with other agents can all make the estimated state uncertain. Instead of representing an object with a single perfectly known trajectory, the system can maintain confidence, probability distributions, multiple hypotheses, or uncertainty-aware latent states. This allows downstream prediction and planning systems to consider both expected motion and the range of plausible alternatives.

Dynamic object representation becomes particularly important when multiple agents interact. The future motion of one pedestrian or vehicle may depend on nearby people, vehicles, robots, traffic rules, obstacles, or environmental constraints. Consequently, the representation should preserve relationships between objects rather than treating each trajectory as completely independent. Relative position, distance, velocity, orientation, proximity, and potential interaction can provide the context required for a world model to learn coordinated or reactive behavior. This transforms BEV from a simple object map into a structured representation of dynamic spatial relationships.

The dynamic BEV state provides a direct input to future prediction and world modeling. A predictive model can use current object states and motion history to estimate future positions, trajectories, occupancy changes, motion fields, and free-space evolution over multiple horizons. Static structure can remain persistent while dynamic entities evolve around it. The resulting future BEV states can then be evaluated by a planner to determine whether candidate robot trajectories remain safe and feasible. This establishes a continuous relationship between perception, tracking, prediction, planning, and action.

Dynamic object representation also supports closed-loop behavior because predictions can be corrected continuously by new observations. After the robot executes an action, new sensor measurements reveal whether tracked objects behaved as expected. The world model can update object states, revise motion estimates, and generate new future hypotheses. This predict--observe--correct cycle allows the system to adapt to unexpected behavior rather than relying on a single prediction made far in advance. Such persistent updating is consistent with the broader world-model architecture in which dynamic entities, environmental history, uncertainty, and predicted future states are maintained as part of an internal representation.

Within the BEV world-model pipeline, dynamic object representation therefore forms an intermediate layer between unified spatial perception and future-state prediction. Camera, LiDAR, radar, depth, and other observations are transformed into a common BEV representation; static and dynamic components are distinguished; dynamic entities are tracked with spatial and temporal attributes; and their states are passed to predictive models and planners. The broader structure places this topic after semantic and geometric BEV and before temporal BEV memory, future BEV prediction, and BEV-based navigation and planning, making dynamic object representation a key mechanism for converting a static spatial map into a continuously evolving physical world state.

05.07. Temporal BEV Memory

Temporal BEV memory extends a single BEV observation into a persistent spatial state that evolves over time. A current BEV frame describes what the sensors observe now, but it may contain occlusions, blind regions, sparse measurements, or temporary visibility gaps. By retaining aligned BEV information from previous time steps, the system can preserve environmental structure and motion context that is no longer directly visible. This transforms BEV from an instantaneous spatial representation into a continuously updated memory for world modeling.

The first requirement for temporal BEV memory is that observations from different time steps must be expressed in a consistent spatial reference frame. As the robot moves, a stationary building, road boundary, or obstacle appears to shift within the sensor observations even though its physical location has not changed. Ego-motion estimation from IMU, odometry, localization, or other sources can compensate for this apparent motion. Previous BEV states can then be transformed into the current reference frame, allowing persistent structures to remain spatially stable while genuine environmental changes remain visible.

After alignment, BEV features from multiple time steps can be accumulated or integrated into a temporal representation. The memory does not necessarily need to preserve every previous frame independently; instead, it can maintain a compact state containing information that remains useful for future reasoning. Static geometry, semantic regions, occupancy, traversability, object states, motion history, and uncertainty can be updated as new observations arrive. This creates a persistent world state that contains both recent evidence and information inherited from earlier observations.

Temporal memory is particularly valuable when the current sensor observation is incomplete. An object hidden behind another object may disappear temporarily from the camera or LiDAR view, while a previously observed road boundary or traversable region may remain outside the current field of view. Instead of immediately removing these elements from the internal state, temporal BEV memory can retain them according to their confidence and expected persistence. This allows the world model to reason about partially observed environments and reduces the instability caused by temporary visibility changes.

Static and dynamic information should be treated differently within temporal memory. Roads, walls, buildings, terrain boundaries, shelves, and other stable structures can accumulate over relatively long periods and form a persistent spatial layer. Pedestrians, vehicles, robots, and machinery require more frequent state updates because their positions and velocities change continuously. Dynamic object tracking can therefore maintain identities, positions, orientations, velocities, and trajectories while static information is integrated more conservatively. This separation helps prevent moving objects from contaminating the persistent representation of the environment.

Temporal BEV memory can be implemented using several forms of state representation. A simple system may accumulate aligned BEV features with temporal decay or confidence weighting, while a learned system may use recurrent networks, temporal convolution, attention mechanisms, transformers, or latent memory states. The memory can operate directly on BEV feature maps or maintain separate representations for static structure, dynamic objects, semantics, and uncertainty. The appropriate design depends on the required temporal horizon, computational resources, sensor frequency, and downstream world-model tasks.

Memory should also preserve uncertainty rather than treating historical observations as permanently correct. Older information may become inaccurate because the environment has changed, localization may contain errors, or an object may have moved after its last observation. Confidence or uncertainty can therefore control how strongly historical features influence the current state. Recent observations may receive greater weight when they contradict older information, while stable structures can retain longer-term memory. This allows temporal BEV memory to balance persistence with adaptation instead of becoming a rigid map of outdated observations.

Temporal memory provides important context for dynamic prediction. A single BEV state can show where an object is located, but a sequence of aligned states reveals how that object has moved. From this history, the world model can estimate velocity, acceleration, turning behavior, trajectory tendencies, and interactions with nearby objects. Similarly, changes in occupancy, free space, traversability, or semantic regions can reveal environmental dynamics that are difficult to infer from one observation. Temporal memory therefore supplies the historical context required for learning state transitions and future BEV evolution.

The memory state becomes an important input to predictive world models. A world model can receive the current BEV state together with temporally accumulated information and estimate future states over multiple horizons. Future predictions may include occupancy evolution, object trajectories, free-space changes, traversability, motion fields, semantic changes, and uncertainty. Because historical and predicted states share the same BEV coordinate framework, the system can maintain continuity between observation, memory, prediction, and planning. Temporal BEV memory thus acts as the bridge between spatial perception and temporal world dynamics.

Temporal BEV memory also supports closed-loop decision making. After a planner selects an action and the robot executes it, new sensor observations provide evidence about whether the predicted environmental evolution actually occurred. The memory can incorporate these observations, correct outdated states, update object trajectories, and revise uncertainty. The world model can then generate new predictions based on the updated state. This creates a continuous observe--remember--predict--act cycle rather than a sequence of independent perception decisions.

Within the broader BEV world-model architecture, temporal BEV memory follows multimodal BEV fusion, semantic and geometric BEV, and dynamic object representation, and provides the foundation for future BEV prediction and BEV-based navigation and planning. Its essential role is to transform aligned observations from multiple time steps into a persistent but adaptable spatial state. In this structure, BEV provides the common spatial coordinate system, temporal memory preserves relevant history, the world model learns how the state evolves, and planning uses predicted states to select physically feasible actions. The chapter structure explicitly places Temporal BEV Memory between Dynamic Object Representation in BEV and Future BEV Prediction, reflecting this role as the temporal bridge within the overall world-model pipeline.

05.08. Future BEV Prediction

Future BEV prediction extends temporal BEV memory from remembering the current and past world states to estimating how the spatial environment will evolve into the future. A current BEV state describes the latest known configuration of geometry, semantics, occupancy, dynamic objects, motion, traversability, and uncertainty, while temporal memory provides the historical context required to understand change. The predictive world model uses this combined state to generate future BEV states at multiple time horizons, creating a compact representation of possible future physical conditions.

The central purpose of future BEV prediction is not to reconstruct future sensor images pixel by pixel, but to predict spatial variables that matter for physical interaction. A future BEV state can represent where objects are likely to move, which regions may become occupied, how free space may change, and whether a region will remain traversable. It can also represent future motion fields, semantic changes, and uncertainty. This makes prediction directly relevant to embodied decision making because the model estimates consequences in the same spatial representation used to understand the current environment.

Prediction begins with the current unified BEV state together with temporal memory. The input may contain static geometry, semantic regions, occupancy, dynamic object states, motion history, traversability, and uncertainty. Historical states provide information about how the environment has evolved, while the current state establishes the latest spatial configuration. The world model then learns or estimates the transition from the observed state to future states. In a simplified form, the model can be viewed as learning a function that maps the current and historical BEV state into one or more future BEV states.

Multiple prediction horizons are important because different decisions require different temporal ranges. A short horizon may capture immediate collision risk, braking requirements, or the next motion of a nearby pedestrian. A medium horizon can support local trajectory selection and obstacle avoidance, while a longer horizon can provide information for navigation and strategic planning. Future BEV prediction therefore produces a sequence such as t+1, t+2, ..., t+T rather than a single deterministic future frame. Each horizon represents an increasingly uncertain hypothesis about how the physical environment may evolve.

Dynamic object prediction is a major component of future BEV prediction. Tracked pedestrians, vehicles, robots, and machinery can be propagated forward using their current position, velocity, orientation, acceleration, motion history, and interactions with nearby agents. The world model can learn patterns such as continued motion, stopping, turning, acceleration, or interaction-dependent behavior. Instead of assuming that every object follows an independent constant-velocity trajectory, learned dynamics can incorporate scene context and relationships between agents to produce more realistic future states.

Future occupancy prediction complements object trajectory prediction by estimating how spatial occupancy itself will evolve. An object may move into a previously free region, causing that region to become occupied, while another region may become free after an object moves away. Occupancy prediction can therefore provide a spatial description of possible future collision regions even when individual object identities are uncertain. This is particularly useful for planning systems that need to evaluate whether a candidate trajectory remains clear over time rather than only at the current instant.

Free-space evolution is another important prediction target. The world model can estimate which areas are likely to remain available for movement and which areas may become blocked by dynamic objects or changing environmental conditions. This is closely related to traversability prediction, because a region that is geometrically open may not remain physically usable. Future BEV prediction can therefore estimate changes in drivable, walkable, or otherwise traversable regions. Such predictions provide planners with a forward-looking spatial representation rather than requiring them to react only after an obstacle has already entered the current path.

Motion fields provide another way to represent future dynamics without requiring every moving entity to be explicitly tracked. A motion field assigns directional or velocity information to spatial regions, allowing the world model to represent collective movement patterns and environmental flow. This can be useful in crowded environments where many agents interact, or in situations where individual object identities are uncertain. Object-centric trajectories and dense motion fields can therefore complement each other, with the former representing identifiable agents and the latter describing broader spatial dynamics.

Future semantic states may also be predicted when environmental meaning changes over time. A region that is currently free may become occupied by a vehicle, a pedestrian area may become temporarily blocked, or a previously uncertain region may become confidently classified after additional observations. Semantic prediction therefore extends beyond geometry by estimating how the functional interpretation of spatial regions may change. Combined with geometric and occupancy prediction, it allows the world model to maintain a richer hypothesis about future environmental states.

Uncertainty must increase as prediction extends farther into the future. Short-term predictions may be relatively constrained by current observations and recent motion history, while long-horizon predictions can diverge because of uncertain intent, interactions, sensor errors, and unobserved environmental changes. A robust future BEV predictor should therefore represent not only an expected future state but also uncertainty or multiple plausible futures. Probability distributions, confidence maps, ensembles, multiple trajectory hypotheses, or latent stochastic representations can provide this information to downstream planning.

The prediction architecture can use different modeling strategies depending on the required representation and horizon. Autoregressive models can generate future BEV states sequentially, using each predicted state as input to the next prediction step. Parallel or multi-horizon models can predict several future states directly, reducing sequential latency and potentially limiting error accumulation. Latent predictive models can encode the current BEV history into a compact latent state and predict future latent states before decoding task-relevant spatial quantities. The appropriate architecture depends on computational constraints, prediction horizon, training data, and the required accuracy.

Temporal consistency is a central challenge because prediction errors can accumulate over multiple future steps. If an early predicted object position is slightly incorrect, subsequent predictions may increasingly diverge from the physically plausible trajectory. The world model therefore needs mechanisms that preserve spatial and temporal consistency across the predicted sequence. Training can compare predictions against future BEV observations at multiple horizons, while architectural constraints, recurrent state updates, attention mechanisms, or structured dynamics can help maintain coherent evolution. The goal is not merely accurate individual frames but a physically plausible future sequence.

Future BEV prediction becomes especially powerful when conditioned on possible actions. Instead of predicting only what is likely to happen under passive continuation, an action-conditioned world model can estimate how the environment may evolve under different robot actions. Candidate steering, velocity, braking, or manipulation actions can therefore produce different predicted BEV futures. The planner can compare these hypothetical outcomes and select an action whose predicted consequences satisfy safety, feasibility, cost, and task objectives. This transforms prediction into an internal simulation mechanism for decision making.

The predicted BEV states provide a natural interface to planning because both prediction and planning operate in the same ground-aligned spatial coordinate system. Candidate trajectories can be evaluated against predicted occupancy, dynamic objects, free-space evolution, traversability, motion fields, and uncertainty. A trajectory that is safe at the current moment but intersects a predicted future obstacle can be rejected, while a trajectory that remains clear across the relevant horizon can be preferred. Planning can therefore become predictive rather than purely reactive, using the world model to evaluate possible future consequences before committing to an action.

Future BEV prediction also supports closed-loop adaptation. Once the robot executes an action, new observations can be compared with the predicted future state. Differences between prediction and observation provide information about model error, unexpected object behavior, environmental changes, or incorrect uncertainty estimates. The temporal BEV memory can then be updated, and the world model can generate a revised set of future predictions. This creates an observe--remember--predict--plan--act cycle in which prediction is continuously corrected by real-world evidence.

Within the broader BEV world-model pipeline, Future BEV Prediction follows temporal BEV memory and precedes BEV-based navigation and planning. Multimodal sensors first contribute to a unified BEV representation, semantic and geometric information define the spatial state, dynamic object representation captures moving entities, and temporal memory preserves historical context. Future BEV prediction then estimates how occupancy, trajectories, free space, traversability, motion, semantics, and uncertainty may evolve across multiple horizons. These predicted states become the forward-looking world state used by planning and control, making future BEV prediction the key bridge from remembering the physical world to anticipating its consequences.

05.09. BEV for Navigation and Planning

BEV provides a natural spatial interface between perception and navigation because both can operate in the same ground-aligned coordinate system. Once camera, LiDAR, radar, depth, and other observations are transformed into BEV, the environment can be represented through metric position, occupancy, semantics, traversability, dynamic objects, motion, and uncertainty. Navigation and planning can therefore reason directly about where the robot is, where it can move, what may block its path, and how the surrounding world may change.

The first advantage of BEV for navigation is that it converts heterogeneous sensor observations into a representation that is consistent with the robot\'s physical movement. Instead of reasoning about obstacles through changing image coordinates, the planner can evaluate their positions and spatial extents relative to the robot. Free space, occupied regions, boundaries, and traversable areas can be represented directly in the same coordinate frame. This reduces the geometric translation normally required between perception outputs and navigation algorithms.

A BEV state can provide the planner with several complementary spatial quantities. Occupancy indicates which regions are blocked, while free space identifies areas that may be available for movement. Semantic information distinguishes roads, sidewalks, restricted regions, vegetation, buildings, pedestrians, vehicles, and other environmental components. Geometry provides position, distance, height, boundaries, and spatial extent, while traversability describes whether a region is physically suitable for the robot. Together, these channels form a task-relevant spatial state for navigation.

Planning in BEV can evaluate candidate trajectories directly against the current or predicted spatial state. A candidate trajectory is not judged only by whether it reaches a goal; it must also avoid occupied regions, maintain sufficient clearance, remain within traversable space, and satisfy the robot\'s kinematic and dynamic constraints. Additional objectives can include distance, travel time, smoothness, energy consumption, comfort, or task-specific priorities. BEV therefore provides a common spatial domain in which safety, feasibility, and cost can be evaluated together.

The integration with future BEV prediction makes planning predictive rather than purely reactive. The current BEV describes the environment now, while the world model predicts future occupancy, object trajectories, free-space evolution, traversability, motion fields, and uncertainty. A trajectory that appears safe at the current instant may become unsafe several seconds later because a pedestrian or vehicle enters its path. By evaluating candidate trajectories against predicted BEV states across multiple horizons, the planner can consider future consequences before committing to an action.

Dynamic objects require special treatment because their future positions are uncertain. Pedestrians, vehicles, robots, and machinery may accelerate, stop, turn, or react to other agents. Their predicted trajectories can therefore be represented as spatial hypotheses within future BEV states. The planner can evaluate whether a candidate path intersects these predicted regions and can incorporate uncertainty by maintaining additional clearance or selecting a trajectory with lower expected collision risk. This allows navigation to respond not only to where objects are, but also to where they may be.

Traversability is particularly important for Physical AI because geometric free space does not necessarily mean physically executable space. A region may appear open but contain terrain that is too steep, unstable, slippery, narrow, or otherwise unsuitable for a particular robot. BEV can encode traversability as a spatial cost or learned feature and combine it with geometry and semantics. The planner can then distinguish between merely empty space and space that the embodied agent can realistically occupy and traverse.

BEV also supports hierarchical planning by allowing different planning layers to use different spatial resolutions and prediction horizons. A high-level navigation layer can reason about broad traversable regions, goals, corridors, and long-range routes, while a local planner can operate on higher-resolution BEV states containing nearby obstacles, dynamic agents, and detailed geometric constraints. The same spatial representation can therefore connect global navigation with local trajectory generation without requiring a completely different representation at every stage.

The connection between BEV and control is established through trajectory evaluation and action selection. After candidate trajectories are generated, the system can compare their collision risk, clearance, reachability, kinematic feasibility, dynamic feasibility, path cost, and task constraints. The selected trajectory is converted into lower-level motion commands such as velocity, steering, acceleration, braking, or other platform-specific controls. New sensor observations then reveal the actual state of the world, allowing the system to update BEV and re-plan continuously.

Uncertainty is essential because both perception and prediction are imperfect. Sensor noise, occlusion, calibration errors, sparse measurements, semantic misclassification, localization uncertainty, and unpredictable agent behavior can all affect the BEV state. Navigation should therefore avoid treating the representation as perfectly known. Confidence maps, probabilistic occupancy, multiple future hypotheses, safety margins, or uncertainty-aware trajectory costs can allow the planner to choose actions that remain robust when the predicted world differs from reality.

The BEV planning interface is not limited to autonomous driving. The same principle can support AMRs, quadrupeds, mobile manipulators, service robots, agricultural robots, and other Physical AI platforms. The precise vehicle model and motion constraints may differ, but the fundamental planning problem remains similar: represent the environment spatially, estimate what is traversable, predict relevant changes, evaluate possible motions, and execute an action while continuously updating the internal state. BEV therefore provides a reusable spatial interface across different embodiments.

Within the broader world-model architecture, BEV-based navigation and planning forms the action-oriented stage after multimodal sensing, BEV fusion, semantic and geometric representation, dynamic object modeling, temporal memory, and future BEV prediction. The overall process transforms heterogeneous observations into a unified spatial state, preserves relevant history, predicts possible future states, evaluates candidate trajectories, and selects an action. New observations then close the loop by updating the BEV state and triggering re-planning. In this sense, BEV is not merely a perception format but a common computational interface connecting perception, memory, prediction, planning, and physical action. The chapter structure places BEV-based Navigation and Planning immediately after Future BEV Prediction and before the complete BEV World Model Pipeline, reflecting its role as the transition from predicted world state to executable behavior.

05.10. BEV World Model Pipeline [w/Code]

A BEV world-model pipeline integrates perception, spatial representation, temporal memory, prediction, planning, and physical action into a continuous process. Its purpose is not merely to convert sensor data into a top-down map, but to construct an internal spatial state that remains consistent across viewpoints and time, describes both static and dynamic elements, predicts how that state will evolve, and supports action selection. In this architecture, BEV functions as a common computational interface connecting sensing with world understanding and control. The chapter is positioned as the final integration point of the BEV representation sequence, following camera-to-BEV, LiDAR-to-BEV, multimodal fusion, semantic and geometric BEV, dynamic objects, temporal memory, future prediction, and navigation and planning.

The pipeline begins with multimodal sensing because no individual sensor provides a complete description of the physical environment. Cameras contribute rich visual and semantic information, LiDAR provides geometric and depth information, radar contributes range and motion evidence, while IMU, odometry, localization, and other sensors provide information about the agent\'s own state and movement. These observations have different coordinate systems, sampling frequencies, resolutions, and failure characteristics. The first stage therefore prepares heterogeneous observations for integration through calibration, synchronization, coordinate transformation, and ego-motion estimation.

The next stage converts modality-specific observations into a common BEV representation. Camera features can be geometrically or learnedly projected into a ground-aligned spatial frame, while LiDAR point clouds can be transformed through voxel, pillar, or other spatial encodings before BEV aggregation. Radar and depth information can be incorporated using corresponding spatial features. Multimodal fusion then combines these representations into a unified BEV state. The objective is to preserve complementary evidence rather than force all sensors to provide the same information, allowing semantic, geometric, occupancy, motion, and uncertainty information to coexist within one spatial representation.

The unified BEV state can be understood as a multi-channel representation of the environment. Geometric channels may encode position, distance, height, occupancy, boundaries, and surfaces, while semantic channels describe objects, regions, and functional meaning. Additional channels can represent velocity, motion, traversability, uncertainty, and learned latent features. This representation allows the system to distinguish what exists from where it exists and how it is structured. Such a combination is important because navigation and physical interaction require both semantic interpretation and metric spatial reasoning.

Dynamic objects are then separated from relatively stable environmental structure. Roads, walls, buildings, terrain, shelves, and other static elements can form a persistent spatial layer, while pedestrians, vehicles, robots, machinery, and other agents are represented through changing position, orientation, velocity, and trajectory. Object tracking maintains identity and temporal continuity, while ego-motion compensation prevents the robot\'s own movement from being confused with environmental motion. This produces a BEV state in which static structure and dynamic behavior can be reasoned about together but updated according to their different temporal characteristics.

Temporal BEV memory extends the current spatial state with information from previous observations. Earlier BEV states are transformed into the current reference frame using ego-motion compensation and then accumulated, integrated, or selectively retained. The memory can preserve static structures that are temporarily outside the sensor field of view, objects hidden by occlusion, and motion histories that cannot be inferred from a single observation. Rather than functioning as a rigid historical map, the memory should retain confidence and uncertainty so that outdated information can be weakened when new observations contradict it.

The temporal state becomes the input to the world model\'s dynamics and state-evolution process. The world model learns how spatial states change over time rather than merely describing the current environment. Historical BEV features, current object states, static structure, motion history, semantics, occupancy, traversability, and uncertainty can be used to estimate future states. In conceptual form, the world model learns a transition from the current and historical BEV state toward future spatial states. This makes the pipeline a predictive internal model rather than a conventional perception-and-mapping system.

Future prediction can produce multiple BEV states over different horizons, such as t+1, t+2, through t+T. These predicted states may contain future occupancy, object trajectories, free-space evolution, traversability changes, motion fields, semantic changes, and uncertainty. Different prediction mechanisms can be selected according to the required horizon and computational constraints, including sequential prediction, parallel or multi-horizon prediction, latent predictive models, probabilistic generative approaches, or interaction-aware representations. The important principle is that the predicted quantities remain expressed in the same spatial representation as the current BEV state.

Prediction can also be conditioned on possible robot actions. Instead of estimating only one passive future, the world model can evaluate how different steering, velocity, braking, or other actions may influence future BEV states. Candidate actions can therefore produce different predicted trajectories and environmental outcomes. This provides an internal simulation capability in which the system can imagine possible consequences before acting. Such action-conditioned prediction connects the BEV world model with broader world-model concepts such as state transition modeling, counterfactual reasoning, planning, and model-based control.

The predicted BEV states are passed to planning and decision making, where candidate robot trajectories are evaluated against future spatial conditions. Planning can consider collision risk, clearance, reachability, kinematic and dynamic feasibility, path cost, task constraints, and other objectives. A trajectory that is currently free but intersects a predicted future obstacle can be rejected, while a trajectory that remains safe and feasible across the required horizon can be selected. Because prediction and planning operate in the same ground-aligned representation, the system avoids repeatedly translating between incompatible sensor, map, and planning coordinate systems.

The selected plan is converted into physical action through a control layer. Depending on the embodiment, the output may include steering, velocity, acceleration, braking, or other motion commands for an autonomous vehicle, AMR, quadruped, mobile manipulator, or other Physical AI system. Execution does not terminate the pipeline. New observations are immediately collected and compared with the predicted state. The BEV state and temporal memory are updated, the world model generates revised predictions, and the planner can re-evaluate candidate trajectories. The complete process therefore forms a closed loop of observe, represent, remember, predict, plan, act, and update.

This closed-loop structure is essential because the physical world is only partially observable and future behavior is uncertain. Sensor failures, occlusion, localization errors, calibration errors, changing environmental conditions, and unexpected agent behavior can make predictions inaccurate. A robust BEV world model therefore needs to preserve uncertainty and allow new evidence to correct previous beliefs. Redundant sensing can also improve robustness: camera information can compensate for sparse LiDAR semantics, LiDAR can preserve geometry when vision becomes unreliable, and radar can provide motion evidence under difficult visual conditions. The internal state consequently remains adaptive rather than becoming a fixed interpretation of the environment.

Within a hierarchical Physical AI architecture, BEV forms the spatial bridge between low-level sensing and higher-level reasoning and action. Sensor encoders first extract modality-specific information, BEV projection and fusion construct a shared spatial state, temporal memory preserves relevant history, and the world model predicts state evolution. Higher-level reasoning can then interpret goals and task context, while a spatial planner evaluates candidate trajectories and a low-level controller converts the selected plan into executable commands. This hierarchy allows different computational layers to operate on representations appropriate to their responsibilities while maintaining a common spatial relationship through BEV.

The major strength of this pipeline is that it turns BEV from a perception representation into an integrated world-model interface. A ground-aligned and metric representation makes spatial relationships more consistent across viewpoints and time, simplifies multimodal fusion, supports persistent memory, separates static structure from dynamic behavior, and provides a natural interface to future prediction and planning. At the same time, BEV remains an abstraction and can lose fine vertical structure, detailed three-dimensional geometry, overlapping spatial relationships, and information important for manipulation or aerial environments. Multi-height BEV, 3D voxels, occupancy representations, object-centric representations, or latent 3D states can therefore complement the main BEV representation when required.

The complete BEV world-model pipeline can consequently be summarized as a continuous transformation from observation to action: multimodal sensors observe the environment, modality-specific encoders extract information, BEV projection and fusion construct a unified spatial state, semantic and geometric information describe what exists and where it exists, dynamic-object modeling captures motion, temporal memory preserves history, the world model predicts future states, planning evaluates possible actions, and control executes the selected behavior. New observations then return to the beginning of the process and update the internal state. In this architecture, BEV is not simply a top-down visualization; it is the spatial substrate through which Physical AI can understand where things are, how they move, what may happen next, where the agent can go, and which action should be taken. The chapter structure places this integrated pipeline at the end of the BEV representation chapter, before the subsequent chapter on occupancy networks and prediction, making it the architectural bridge from BEV representation toward broader physical world modeling.


Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go

Top comments (0)