DEV Community: T, Sato

How to Show “North” on Vision Pro—Even Without a Compass: Navigation with iPhone + Vision Pro

T, Sato — Thu, 26 Jun 2025 14:07:29 +0000

[Vision Pro] and the Limitations of Location Awareness

The Vision Pro cannot perform navigation based on latitude/longitude and directional heading. This is because the device lacks built-in hardware for GPS and magnetic compass (geomagnetic sensor), making it unable to acquire such information.

Note: While approximate latitude/longitude data can be obtained via Wi-Fi—making some location-based navigation possible—it is not possible to determine the direction the Vision Pro is facing based on any built-in sensors.

However, for users to fully benefit from the unique potential of spatial navigation, such as overlaying virtual information on the real world and doing so in 360 degrees, leveraging accurate location data becomes essential.

For example, imagine a user wearing the Vision Pro: being able to intuitively understand the direction and distance to a destination—such as a store or landmark—from their current standing position.

While smartphone map apps already allow users to check directions, having a constant visual reference for cardinal directions (North, South, East, West) overlaid in the real world feels fundamentally different. It's like being able to see the North Star at all times, even during the day.

Can Vision Pro Acquire [Latitude/Longitude and Directional Heading]?

To reiterate, the Vision Pro has no way to acquire latitude, longitude, or directional heading. This is due to the absence of hardware such as GPS or magnetic sensors.

Still, we want to make spatial navigation based on real-world location data possible.

So how can we achieve that?

Fortunately, we already have a device capable of obtaining accurate latitude, longitude, and heading—that device is the iPhone, or any modern smartphone.

Can the iPhone Work in Tandem with Vision Pro?

If the Vision Pro lacks certain capabilities, why not let the iPhone fill in the gaps?
But first—can the two devices even communicate with each other?

If they can, the iPhone can provide the Vision Pro with the location data it gathers, allowing that information to be reflected in the MR space.

[LLD Compass]: A Spatial Compass Experience on Vision Pro

To explore this possibility, we created a spatial compass app called LLD Compass, which links the Vision Pro and iPhone to recreate directional orientation within a mixed reality (MR) space.
The name LLD stands for Latitude, Longitude, and Direction.

In this article, we’ll explore how LLD Compass was made possible—covering the technical architecture, key implementation strategies, and the potential use cases it unlocks.

LLD Compass – App Overview and Use Cases

By linking the Vision Pro with an iPhone, LLD Compass enables the visualization of cardinal directions (North, South, East, West) within a mixed reality environment.
This is the core concept behind the app.

A red sphere represents North.

What Kind of Experience Does It Offer?

With the Vision Pro on, you can intuitively see spatial cues like:
“This direction leads to Tokyo Tower,”
“Haneda Airport is over that way,” or
“From Tokyo, Paris lies in this direction”—all visualized directly in your surrounding space.

While smartphone map apps can provide similar information, simply looking around to understand direction—without needing to glance at a screen—is a completely different experience.

By leveraging the Vision Pro’s unique ability to deliver spatial awareness, LLD Compass offers a new kind of value that traditional navigation tools can't provide.

Potential Use Cases

Tourism Navigation: Instantly see which direction famous landmarks, shops, or attractions are located—just by looking around.
Stargazing Guidance: Display constellations based on your orientation and current field of view.
Venue Navigation: In theme parks or zoos, intuitively understand where your destination lies within the venue.
Breaking News & Local Buzz: Get real-time directional cues for incidents, accidents, or trending reviews happening nearby.

Simply being able to see spatial and directional information at a glance greatly enriches the mixed reality experience.

System Architecture: Connecting Vision Pro and iPhone

Device Roles and Responsibilities
LLD Compass leverages the strengths of both devices by assigning them distinct roles:

iPhone Acquires latitude, longitude, and heading data, then sends it to the Vision Pro Technologies used: SwiftUI / CoreLocation / MultipeerConnectivity
Vision Pro Receives the location data and renders it spatially within the MR environment Technologies used: Unity / PolySpatial / Swift Bridge

How the Devices Communicate

Using Apple’s MultipeerConnectivity framework, the iPhone and Vision Pro communicate over a peer-to-peer (P2P) connection.
LLD Compass leverages this framework to send real-time location and heading data from the iPhone to the Vision Pro, where it’s immediately reflected in the mixed reality space.

iPhone
- Retrieves latitude, longitude, and heading information
- Serializes the data into JSON
- Sends it via MultipeerConnectivity
Vision Pro (Swift Layer)
- Receives the data using MultipeerConnectivity
- Passes the JSON to Unity
Unity
- Deserializes the JSON into usable latitude, longitude, and heading values
- Updates object positions and orientations accordingly in the spatial environment

App Structure

Given the system architecture described above, the solution requires two separate apps:

The iPhone app: Responsible for acquiring latitude, longitude, and heading data, and sending it to the Vision Pro
The Vision Pro app: Receives the location data and visualizes it within the MR environment

Implementation Challenges and Solutions

Bridging [iPhone], [Vision Pro], and [Unity]

The MR environment in this project was built using Unity.
One key challenge was how to reflect the data acquired on the iPhone within the Unity world running on Vision Pro.

Solution: Create a Swift Bridge on Vision Pro

To pass data from the native layer to Unity, we used UnitySendMessage.
A custom bridge was implemented in Swift, allowing the Swift layer on Vision Pro to forward received data directly into Unity.

As a result, the MultipeerConnectivity data exchanged between iPhone and Vision Pro can now be accessed and utilized within Unity.

Vision Pro Has No Concept of "North"

The Vision Pro has no way to determine true North.
While its internal sensors can detect relative rotation, it cannot identify any absolute orientation—such as the direction of magnetic North on Earth.

Solution: Perform an Initial Orientation Sync with iPhone

After the Vision Pro is worn, we initiate communication while both the Vision Pro and iPhone are facing the same direction.
This moment is treated as the reference point—a shared initial orientation.

From then on, the direction the iPhone is facing is assumed to be the same as that of the Vision Pro, effectively giving the Vision Pro a sense of absolute direction.

Determining the Direction of a Landmark

Suppose the user's current location—where they’re wearing the Vision Pro—is point A, and a landmark (e.g., Tokyo Tower) is point B.
We need to determine the relative direction from point A to point B.

Solution: Calculate the Bearing (Azimuth)

Note: A bearing (or azimuth) is the angle measured clockwise from a reference direction—typically true North—to the line connecting two points.
It’s widely used in fields such as geography, navigation, aviation, surveying, and GPS systems.

By using the latitude and longitude of both point A and point B, we can calculate the bearing from A to B, which tells us the direction in which the landmark lies.

double GetBearing(double lat1, double lon1, double lat2, double lon2)
{
    var phi1 = lat1 * Math.PI / 180;
    var phi2 = lat2 * Math.PI / 180;
    var deltaLambda = (lon2 - lon1) * Math.PI / 180;

    var y = Math.Sin(deltaLambda) * Math.Cos(phi2);
    var x = Math.Cos(phi1) * Math.Sin(phi2) -
            Math.Sin(phi1) * Math.Cos(phi2) * Math.Cos(deltaLambda);

    var theta = Math.Atan2(y, x);
    var bearing = (theta * 180 / Math.PI + 360) % 360;

    return bearing; // 北=0, 東=90, 南=180, 西=270
}

Future Outlook and Applications

Through the development of LLD Compass, we’ve come to truly appreciate the value and potential of bringing direction and location into MR space.

This technology can be applied to a wide range of services, such as:

Integration with Map Services
By projecting maps into spatial environments, users can experience the sensation of having a map unfold at their feet, making it easier to understand their surroundings.
Real-Time Flight Tracking and Live Positioning
Aircraft flying overhead can be displayed in real time—showing their direction and path—allowing users to visually understand where each plane is coming from and where it’s headed.
Shared Location Awareness Among Multiple Users
In fieldwork or at event venues, users can share their location data with others. This enables a shared experience where everyone can see each other’s position and orientation in the same MR space.

Final Thoughts

By combining the strengths of two distinct devices—Vision Pro and iPhone—LLD Compass delivers a simple yet powerful experience:
giving the Vision Pro a true concept of North.

While Vision Pro excels at spatial expression, it has inherent limitations in acquiring location data
The iPhone compensates for this, acting as a sensor to enable spatial navigation
Real-time integration is achieved through Unity × Swift × MultipeerConnectivity
From the user’s current position, it becomes possible to calculate the bearing to any external point

The true value of Vision Pro lies in how it overlays information onto the real world.
By embedding location and directional context into the spatial environment, we unlock entirely new and deeply immersive experiences.

LLD Compass is both a prototype and an experiment—a first step toward exploring that potential.

Moving forward, I hope to continue exploring what new kinds of experiences the Vision Pro can offer.

Getting Started with Vision Pro App Development Using Unity: “Metal” vs “RealityKit”

T, Sato — Mon, 31 Mar 2025 02:12:54 +0000

“Metal” vs “RealityKit”

When developing Vision Pro applications, the most common method is to use “Xcode”. However, it is also possible to develop using “Unity”.

This article explains two approaches available when developing Vision Pro apps with Unity: “Metal-based Apps on visionOS” and “RealityKit apps on visionOS”. We’ll refer to these two methods as “Metal mode” and “RealityKit mode” throughout this article.

Note: Unity does not officially define something called “Metal mode” or “RealityKit mode”. In this article, we use these terms simply to make the distinction clearer when discussing each approach’s features and use cases.

Note: In Unity, there is another development approach called “Windowed Apps in visionOS”. However, since it does not utilize the XR features unique to visionOS, it is not covered in this article.

Two Rendering Methods in Unity for Vision Pro

When developing for Vision Pro in Unity, two main rendering methods are available:

Metal

An approach that directly uses Apple’s low-level graphics API, “Metal”, for rendering
Allows full utilization of Unity’s built-in rendering pipeline (e.g., shaders, post-processing, lighting)

RealityKit (PolySpatial)

An approach that uses Apple’s high-level 3D/AR framework, “RealityKit”
Passes the Unity scene data to RealityKit, making it easier to build apps that leverage Vision Pro’s unique spatial computing capabilities

What is Metal?

Metal is Apple’s low-level graphics API, which allows applications to send commands directly to the GPU for maximum performance.
Unity has long supported rendering functions that target Metal, so if you choose “Metal mode”, the development process will be quite similar to creating other 3D games for iOS.

What is RealityKit?

On the other hand, RealityKit is a high-level 3D framework designed by Apple specifically for AR/VR. It includes not only 3D rendering but also a physics engine, multi-user features, and other functions tailored for AR and VR.
Because Vision Pro is designed with mixed reality (MR) capabilities in mind—overlapping virtual objects onto the real-world environment—RealityKit is especially well-suited for this device.

Metal Mode

Unity handles the rendering on its own
You can fully utilize Unity’s built-in rendering pipeline, setting up shaders, post-processing, and more with great flexibility.
Easy to use Unity’s standard XR toolkit
You can employ development methods similar to other VR platforms in Unity, such as XR Origin, camera setups, and Tracked Pose Driver.

RealityKit Mode

Delegates rendering to Apple’s AR engine
Unity sends its scene and object data to RealityKit for rendering, enabling native use of RealityKit’s spatial computing and AR/MR features.
Constraints on Unity-specific features
Custom shaders do not run directly on the RealityKit side. Instead, you must go through Shader Graph, and some features are limited.

Creating a Sample Scene

Below is a quick overview of how to set up a sample project in Unity for Vision Pro.

Note: This section focuses only on the unique steps for each “Metal mode” and “RealityKit mode”. Also, keep in mind that versions and settings may change in the future, so always refer to Unity’s official documentation for the latest details.

Creating a Project in Metal Mode

1. Project Settings

Install the Apple visionOS XR Plugin

From the “Package Manager”, install the “Apple visionOS XR Plugin”.
(Or skip this step initially and let the next steps prompt you to install it automatically.)

Set up XR Plug-in Management

In “Project Settings > XR Plug-in Management”, select “Apple visionOS”.

Select App Mode

In “XR Plug-in Management > Apple visionOS > App Mode”, set it to “Metal Rendering with Compositor Services”.

2. Scene Creation

Add AR Session and XR Origin

From the menu, go to “GameObject > XR > AR Session” to add an AR Session.
Then go to “GameObject > XR > XR Origin (Mobile AR)” to add an XR Origin.

Background Settings

If you want an immersive VR-like environment, set the Main Camera’s “Background Type” to SkyBox.

If you want an MR-like scene that shows the real world in the background, set “Background Type” to Solid Color, choose black (e.g., [0, 0, 0, 0]) with Alpha 0 for full transparency.

Place 3D Objects

Place a 3D object near the center of the scene.

3. Build and Check Rendering

Verify that the 3D object you placed is rendered correctly.

Creating a Project in RealityKit Mode (PolySpatial)

1. Project Settings

Install the Apple visionOS XR Plugin

From the “Package Manager”, install the “Apple visionOS XR Plugin”.
(Or skip this step initially and let the next steps prompt you to install it automatically.)

Set up XR Plug-in Management

In “Project Settings > XR Plug-in Management”, select “Apple visionOS”.

Install PolySpatial

From the “Package Manager”, install the “PolySpatial” package.
(Or skip this step and proceed to the final step, which may automatically prompt installation.)

Select App Mode

In “XR Plug-in Management > Apple visionOS > App Mode”, set it to “RealityKit with PolySpatial”.

2. Scene Creation

Add AR Session

From the menu, go to “GameObject > XR > AR Session” to add an AR Session.

Add Volume Camera

From the menu, go to “GameObject > XR > Setup > Volume Camera” to add a Volume Camera.

VolumeCamera Window Configuration

From the menu, “Assets > Create > PolySpatial > Volume Camera Window Configuration”.

Then choose either Bounded volume (a fixed display area) or Unbounded volume (extended across the entire space) and attach this configuration to the Volume Camera.

Place 3D Objects

Place a 3D object within the Volume Camera’s area.

3. Build and Check Rendering

Verify that the placed 3D object is rendered correctly.

Use Cases for Each Mode

How do you decide whether to use Metal mode or RealityKit mode? Below are some practical examples.

When Metal Mode Is Suitable

If you want a fully virtual space
For VR games, simulations, or immersive cinematic experiences, where you want rich graphics in a purely virtual environment.
If you want advanced shaders or post-processing
Ideal for utilizing custom shaders or high-end effects that come with Unity’s standard rendering capabilities.
If you want to align with other VR platforms
For instance, when porting an existing VR project (for PC VR or standalone devices) to Vision Pro and you prefer to maintain a similar Unity XR workflow.

When RealityKit Mode Is Suitable

If you want to overlay virtual objects onto the real world
Take advantage of Vision Pro’s spatial computing features (tracking floors, walls, etc.) to create MR experiences.
If you prefer a simple, utility-focused MR app
Great for displaying additional information or guides while viewing the real world.
If you need a persistent, always-on app that works alongside other apps
For example, placing widgets within your environment. Instead of complex custom shaders, you focus on smooth spatial placement and multitasking.

In Closing

When developing Vision Pro apps with Unity, choosing between “Metal mode” and “RealityKit mode” largely depends on your app’s purpose and the kind of experience you want to create.

Metal mode

Development follows a workflow similar to conventional VR, allowing for rich graphical expression and advanced shaders.

RealityKit mode

By leveraging RealityKit’s optimization for spatial computing, it becomes easier to create apps that incorporate MR elements.

Each approach has its pros and cons, so you should carefully consider factors such as “the direction of your app”, “the level of visual quality required”, and “the depth of interaction” early in development.

Since Vision Pro may undergo substantial updates and new SDK features, it’s crucial to keep an eye on the official documentation and release notes—and, of course, to experiment hands-on to find the best workflow for your needs.