SteamVR Overlay with Unity: Follow Device

Follow the HMD

Overlay following the HMD

Remove the position code

First, remove the absolute position code we added in the previous part.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);

-   var position = new Vector3(0, 2, 3);
-   var rotation = Quaternion.Euler(0, 0, 45);
-   SetOverlayTransformAbsolute(overlayHandle, position, rotation);

    ShowOverlay(overlayHandle);

    ...

Device Index

In SteamVR, connected devices are identified with Device Index that automatically allocates from the system. (read the wiki for details)
For HMD, it is defined as OpenVR.k_unTrackedDeviceIndex_Hmd and is always 0.

Prepare position and rotation

Let’s display the overlay at 2 m ahead (Z-axis) of the HMD. As in the previous part, let position and rotation variables.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

+   var position = new Vector3(0, 0, 2);
+   var rotation = Quaternion.Euler(0, 0, 0);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
}

Set relative position based on the HMD

Use SetOverlayTransformTrackedDeviceRelative() to set the relative position based on the HMD. (read the wiki for details)

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
    var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
    var matrix = rigidTransform.ToHmdMatrix34();
+   var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, ref matrix);
+   if (error != EVROverlayError.None)
+   {
+       throw new Exception("Failed to set overlay position: " + error);
+   }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
} 

Pass the HMD device index (OpenVR.k_unTrackedDeviceIndex_Hmd) and the transformation matrix.
Run the program, and check the overlay is shown 2 m ahead of the HMD.

Organize code

Move the relative position code into SetOverlayTransformRelative().

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
-   var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
-   var matrix = rigidTransform.ToHmdMatrix34();
-   var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, ref matrix);
-   if (error != EVROverlayError.None)
-   {
-       throw new Exception("Failed to set overlay position: " + error);
-   }
+   SetOverlayTransformRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, position, rotation);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
} 

...

+ // Pass deviceIndex as argument.
+ private void SetOverlayTransformRelative(ulong handle, uint deviceIndex, Vector3 position, Quaternion rotation)
+ {
+     var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
+     var matrix = rigidTransform.ToHmdMatrix34();
+     var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(handle, deviceIndex, ref matrix);
+     if (error != EVROverlayError.None)
+     {
+         throw new Exception("Failed to set overlay position: " + error);
+     }
+ }

Follow the controller

Overlay following the controller

Use controller device index instead of HMD to make the overlay follow a controller.

Get the controller device index

Get the left controller’s device index with GetTrackedDeviceIndexForControllerRole() of OpenVR.System.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
+   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
    SetOverlayTransformRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, position, rotation);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);
    ShowOverlay(overlayHandle);
}

The argument is:
Left controller: EtrackedControllerRole.LeftHand
Right controller: EtrackedControllerRole.RightHand

If it fails to get the device index like the controller is disconnected, GetTrackedDeviceIndexForControllerRole() returns k_unTrackedDeviceIndexInvalid.

Follow the controller

We have got the left controller index, then make the overlay follow the controller. Pass the controller index to SetOverlayTransformRelative() that we previously created for the HMD.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(0, 0, 2);
    var rotation = Quaternion.Euler(0, 0, 0);
    var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   SetOverlayTransformRelative(overlayHandle, OpenVR.k_unTrackedDeviceIndex_Hmd, position, rotation);
+   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+   {
+       SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
+   }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, 0.5f);

    ShowOverlay(overlayHandle);
}

Make sure the left controller is connected to the SteamVR window, then run the program.

The overlay should follow the controller instead of the HMD.

Adjust overlay position

To make a watch application, we will adjust the overlay position on the left wrist. Make position parameters editable at runtime on the Unity editor.

Add member variables

Add size, position, and rotation variables as class members. Use Range() attribute to show sliders on the inspector.

public class WatchOverlay : MonoBehaviour
{
    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;

+   [Range(0, 0.5f)] public float size = 0.5f;
+   [Range(-0.2f, 0.2f)] public float x;
+   [Range(-0.2f, 0.2f)] public float y;
+   [Range(-0.2f, 0.2f)] public float z;
+   [Range(0, 360)] public int rotationX;
+   [Range(0, 360)] public int rotationY;
+   [Range(0, 360)] public int rotationZ;

    ...

Sliders are shown on the inspector.

Replace variables in the code

Replace the existing size and position variables in the code with the added member variables.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

-   var position = new Vector3(0, 0, 2);
-   var rotation = Quaternion.Euler(0, 0, 0);
+   var position = new Vector3(x, y, z);
+   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ); 
    var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
    if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
    {
        SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
    }

    SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

-   SetOverlaySize(overlayHandle, 0.5f);
+   SetOverlaySize(overlayHandle, size);
    ShowOverlay(overlayHandle);
}

Update size and position in Update()

Make the size and position editable at runtime by adding code to the Update(). Note that this code will be deleted later. It is temporary for determining new positions and rotation.

private void Start()
{
    InitOpenVR();
    overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

    var position = new Vector3(x, y, z);
    var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
    var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
    if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
    {
        SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
    }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, size);
    ShowOverlay(overlayHandle);
}

+ private void Update()
+ {
+     SetOverlaySize(overlayHandle, size);
+ 
+     var position = new Vector3(x, y, z);
+     var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ); 
+     var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
+     if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+     {
+         SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
+     }
+ }

...

Run the program. Ensure the inspector slider changes the overlay size and position at runtime.

Adjust overlay position

Move sliders to adjust the overlay position to the left wrist. I recommend changing sliders from the desktop window that is opened in the SteamVR dashboard.

Control the Unity editor from the SteamVR dashboard.

Another way is looking at the HMD view on the desktop and adjusting parameters.

It’s helpful if you don’t want to put on the HMD.

Here are sample parameters.

size = 0.08
x = -0.044
y = 0.015
z = -0.131
rotationX = 154
rotationY = 262
rotationZ = 0

When adjusting is done, right click on the WatchOverlay component name, and select Copy Component.

Stop the program, right click the WatchOverlay component again, and paste the copied values with Paste Component Value.

Run the program. Check the overlay is on the left wrist.

Remove temporary code

Remove the code from the Update().

private void Update()
{
-   SetOverlaySize(overlayHandle, size);
-   
-   var position = new Vector3(x, y, z);
-   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ); 
-   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
-   {
-       SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
-   }
}

When the controller is not connected

Currently, the controller must be connected at launch because we get the controller device index at the Start().
Move the getting device index code from Start() to Update() to support the cases where the controller connects or disconnects in the middle.

private void Start()
{
    InitOpenVR();
    OverlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

-   var position = new Vector3(x, y, z);
-   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
-   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
-   {
-       SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
-   }

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    SetOverlayFromFile(overlayHandle, filePath);

    SetOverlaySize(overlayHandle, size);

    ShowOverlay(overlayHandle);
}

+ private void Update()
+ {
+     var position = new Vector3(x, y, z);
+     var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
+     var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
+     if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+     {
+         SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
+     }
+ }

Now, it works even if the controller is connected in the middle.

Final code

using UnityEngine;
using Valve.VR;
using System;

public class WatchOverlay : MonoBehaviour
{
    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;

    [Range(0, 0.5f)] public float size;
    [Range(-0.2f, 0.2f)] public float x;
    [Range(-0.2f, 0.2f)] public float y;
    [Range(-0.2f, 0.2f)] public float z;
    [Range(0, 360)] public int rotationX;
    [Range(0, 360)] public int rotationY;
    [Range(0, 360)] public int rotationZ;

    private void Start()
    {
        InitOpenVR();
        overlayHandle = CreateOverlay("WatchOverlayKey", "WatchOverlay");

        var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
        SetOverlayFromFile(overlayHandle, filePath);

        SetOverlaySize(overlayHandle, size);
        ShowOverlay(overlayHandle);
    }

    private void Update()
    {
        var position = new Vector3(x, y, z);
        var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
        var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
        if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
        {
            SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
        }
    }

    private void OnApplicationQuit()
    {
        DestroyOverlay(overlayHandle);
    }

    private void OnDestroy()
    {
        ShutdownOpenVR();
    }

    private void InitOpenVR()
    {
        if (OpenVR.System != null) return;

        var error = EVRInitError.None;
        OpenVR.Init(ref error, EVRApplicationType.VRApplication_Overlay);
        if (error != EVRInitError.None)
        {
            throw new Exception("Failed to initialize OpenVR: " + error);
        }
    }

    private void ShutdownOpenVR()
    {
        if (OpenVR.System != null)
        {
            OpenVR.Shutdown();
        }
    }

    private ulong CreateOverlay(string key, string name)
    {
        var handle = OpenVR.k_ulOverlayHandleInvalid;
        var error = OpenVR.Overlay.CreateOverlay(key, name, ref handle);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to create overlay: " + error);
        }

        return handle;
    }

    private void DestroyOverlay(ulong handle)
    {
        if (handle != OpenVR.k_ulOverlayHandleInvalid)
        {
            var error = OpenVR.Overlay.DestroyOverlay(handle);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to dispose overlay: " + error);
            }
        }
    }

    private void SetOverlayFromFile(ulong handle, string path)
    {
        var error = OpenVR.Overlay.SetOverlayFromFile(handle, path);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to draw image file: " + error);
        }
    }

    private void ShowOverlay(ulong handle)
    {
        var error = OpenVR.Overlay.ShowOverlay(handle);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to show overlay: " + error);
        }
    }

    private void SetOverlaySize(ulong handle, float size)
    {
        var error = OpenVR.Overlay.SetOverlayWidthInMeters(handle, size);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to set overlay size: " + error);
        }
    }

    private void SetOverlayTransformAbsolute(ulong handle, Vector3 position, Quaternion rotation)
    {
        var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
        var matrix = rigidTransform.ToHmdMatrix34();
        var error = OpenVR.Overlay.SetOverlayTransformAbsolute(handle, ETrackingUniverseOrigin.TrackingUniverseStanding, ref matrix);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to set overlay position: " + error);
        }
    }

    private void SetOverlayTransformRelative(ulong handle, uint deviceIndex, Vector3 position, Quaternion rotation)
    {
        var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
        var matrix = rigidTransform.ToHmdMatrix34();
        var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(handle, deviceIndex, ref matrix);
        if (error != EVROverlayError.None)
        {
            throw new Exception("Failed to set overlay position: " + error);
        }
    }
}

Here, We attach the overlay on the left wrist. Let’s draw Unity camera output to the overlay instead of the static image file in the next part.