kurohuku

Posted on Jun 18 • Edited on Oct 8

SteamVR Overlay with Unity: Controller Input

#unity3d #steamvr #openvr #vr

In this part, we will make the watch hidden by default and show it in a few seconds by pressing a controller button.

There are various ways to get controller input with Unity. This time, we will use OpenVR Input API (SteamVR Input).

Prerequisite setting

Open the SteamVR Settings.

Toggle Advanced Settings.

Toggle Developer > Enable debugging options in the input binding user interface to On.

Create action manifest

SteamVR Input uses abstracted actions instead of direct access to the physical buttons.
The developer predefines actions and may create default mappings for each controller. Users can assign physical buttons to the actions in the SteamVR Input setting window.

First, create the action manifest that defines actions in JSON format. We use the Unity SteamVR Plugin GUI tool to generate the action manifest easily.

Generate action manifest

Select Unity menu Window > SteamVR Input.

It asks you whether you want to use a sample action manifest file.
This time we will make up from the first so select No.

Input the action sets name to “Watch”, and select the below dropdown to per hand. An application can have multiple action sets.

Per hand means it can map different actions to the buttons of the left and right-hand controller separately.
Mirrored means it can map actions to only one controller and the other buttons are mirrored to the mapping.

Click the NewAction name that is in the In box of the Actions section.
The “Action Details” is shown to the right. Change the Name to “WakeUp”.
Click the “Save and generate” button at the left bottom to generate an action manifest file.

It will generate a new action manifest file as StreamingAssets/SteamVR/actions.json.

{
  "actions": [
    {
      "name": "/actions/Watch/in/WakeUp",
      "type": "boolean"
    }
  ],
  "action_sets": [
    {
      "name": "/actions/Watch",
      "usage": "leftright"
    }
  ],
  "default_bindings": [],
  "localization": []
}

Create default binding

Click the right bottom Open binding UI button.

If any VR controller is not shown, check your HMD is recognized from SteamVR.

Click the “Create New Binding”.

Set the Y button to show the watch for a few seconds. If your controller doesn’t have the Y button, use another button instead.

Click the + button to the right of the Y Button.

Select BUTTON.

Select None to the right of the Click, then select wakeup action.

Click the left bottom check icon to save changes.

Click the right bottom Replace Default Binding.

Click Save.

At this point, a new default binding file is added to the action manifest.

actions.json

{
   "action_sets" : [
      {
         "name" : "/actions/Watch",
         "usage" : "single"
      }
   ],
   "actions" : [
      {
         "name" : "/actions/Watch/in/WakeUp",
         "type" : "boolean"
      }
   ],
   "default_bindings" : [
      {
+        "binding_url" : "application_generated_unity_retryoverlay_exe_binding_oculus_touch.json",
+        "controller_type" : "oculus_touch"
      }
   ],
   "localization" : []
}

The default binding file is generated into the same directory to actions.json. These files will be included in the build for users.

Click the <- BACK button at the upper left, then you see the new setting is active.

Now we’ve made the default binding. Close the SteamVR Input window.

Also, close the Unity SteamVR Input window.
At this time, it asks you to save the change, so click Close.

In this tutorial environment, clicking the Save button overwrites the action manifest as an empty default_bindings.
(There is no problem with clicking Save when we open the binding setting window next time.)

How about other controller bindings?

You can create other controller’s default bindings by clicking the controller name on the right side.

Other controllers’ default bindings are added to default_bindings param of action manifest.

{
    "action_sets" : [
       {
          "name" : "/actions/NewSet",
          "usage" : "leftright"
       }
    ],
    "actions" : [
       {
          "name" : "/actions/NewSet/in/NewAction",
          "type" : "boolean"
       }
    ],
    "default_bindings" : [
      {
         "binding_url" : "application_generated_unity_steamvr_inputbindingtest_exe_binding_oculus_touch.json",
         "controller_type" : "oculus_touch"
      },
+     {
+        "binding_url" : "application_generated_unity_steamvr_inputbindingtest_exe_binding_vive_controller.json",
+        "controller_type" : "vive_controller"
+     }
   ],
   "localization" : []
}

You don’t necessarily have to create other controller default bindings because SteamVR automatically remaps actions for other controllers if at least one default binding exists.

Create script

Create a new script InputController.cs into Scripts folder.
We will add the controller input related code to this file.

On hierarchy, right click > Create Empty to create an empty object, change the object name to InputController.
Drag InputController.cs from the project window to the InputController object.

Initialize and cleanup OpenVR

Copy the below code to InputController.cs.

using System;
using UnityEngine;
using Valve.VR; 

public class InputController : MonoBehaviour
{
    private void Start()
    {
        OpenVRUtil.System.InitOpenVR();
    }

    private void Destroy()
    {
        OpenVRUtil.System.ShutdownOpenVR();
    }
}

Set action manifest path

Set the action manifest path with SetActionManifestPath() at launch. (read the wiki for details)
The action manifest is generated as StreamingAssets/SteamVR/actions.json so we set this.

using System;
using UnityEngine;
using Valve.VR; 

public class InputController : MonoBehaviour
{
    private void Start()
    {
        OpenVRUtil.System.InitOpenVR();

+       var error = OpenVR.Input.SetActionManifestPath(Application.streamingAssetsPath + "/SteamVR/actions.json");
+       if (error != EVRInputError.None)
+       {
+           throw new Exception("Failed to set action manifest path: " + error);
+       }
    }

    private void Destroy()
    {
        OpenVRUtil.System.ShutdownOpenVR();
    }
}

Get action set handle

An application can have some action sets that have multiple actions. We use an action set handle to determine which action set is used.
Get action set handle with GetActionSetHandle().

The action set is set with a string /actions/[action_set_name].
This time, it’s /actions/Watch.

public class InputController : MonoBehaviour
{
+   ulong actionSetHandle = 0;

    private void Start()
    {
        OpenVRUtil.System.InitOpenVR();

        var error = OpenVR.Input.SetActionManifestPath(Application.streamingAssetsPath + "/SteamVR/actions.json");
        if (error != EVRInputError.None)
        {
            throw new Exception("Failed to set action manifest path: " + error);
        }

+       error = OpenVR.Input.GetActionSetHandle("/actions/Watch", ref actionSetHandle);
+       if (error != EVRInputError.None)
+       {
+           throw new Exception("Failed to get action set /actions/Watch: " + error);
+       }
    }

    private void Destroy()
    {
        OpenVRUtil.System.ShutdownOpenVR();
    }
}

Get action handle

Next, get the action handle with GetActionHandle().
Action is set with /actions/[action_set_name]/in/[action_name].
This time, action is /actions/Watch/in/Wakeup.

public class InputController : MonoBehaviour
{
    ulong actionSetHandle = 0;
+   ulong actionHandle = 0;

    private void Start()
    {
        OpenVRUtil.System.InitOpenVR();

        var error = OpenVR.Input.SetActionManifestPath(Application.streamingAssetsPath + "/SteamVR/actions.json");
        if (error != EVRInputError.None)
        {
            throw new Exception("Faild to set action manifest path: " + error);
        }

        error = OpenVR.Input.GetActionSetHandle("/actions/Watch", ref actionSetHandle);
        if (error != EVRInputError.None)
        {
            throw new Exception("Failed to get action set /actions/Watch: " + error);
        }

+       error = OpenVR.Input.GetActionHandle($"/actions/Watch/in/WakeUp", ref actionHandle);
+       if (error != EVRInputError.None)
+       {
+           throw new Exception("Faild to get action /actions/Watch/in/WakeUp: " + error);
+       }
    }

    private void Destroy()
    {
        OpenVRUtil.System.ShutdownOpenVR();
    }
}

Update action state

Update each action state at the first of each frame.

Prepare action set to update

We use an action set to specify which actions will be updated.
Pass target action sets as an array of VRActiveActionSet_t type.
This time, the action set is only /actions/Watch so we make a single-element array VRActionActiveSet_t[].

private void Start()
{
    OpenVRUtil.System.InitOpenVR();

    var error = OpenVR.Input.SetActionManifestPath(Application.streamingAssetsPath + "/SteamVR/actions.json");
    if (error != EVRInputError.None)
    {
        throw new Exception("Failed to set action manifest path: " + error);
    }

    error = OpenVR.Input.GetActionSetHandle("/actions/Watch", ref actionSetHandle);
    if (error != EVRInputError.None)
    {
        throw new Exception("Failed to get action set /actions/Watch: " + error);
    }

    error = OpenVR.Input.GetActionHandle($"/actions/Watch/in/WakeUp", ref actionHandle);
    if (error != EVRInputError.None)
    {
        throw new Exception("Failed to get action /actions/Watch/in/WakeUp: " + error);
    }
}

+ private void Update()
+ {
+     // Action set list to update
+     var actionSetList = new VRActiveActionSet_t[]
+     {
+         // This time, Watch action only.
+         new VRActiveActionSet_t()
+         {
+             // Pass Watch action set handle
+             ulActionSet =  actionSetHandle,
+             ulRestrictedToDevice = OpenVR.k_ulInvalidInputValueHandle,
+         }
+     };
+ }

...

ulRestrictedToDevice limits input only from a specific device. For exmaple, to bind different actions for left and right controllers. We don't use this basically so set k_ulInvalidInputValueHandle.

Update state

We use UpdateActionState() to update action state. Pass the action set array to UpdateActionState().

private void Update()
{ 
   var actionSetList = new VRActiveActionSet_t[]
   {
       new VRActiveActionSet_t()
       {
           ulActionSet =  actionSetHandle,
           ulRestrictedToDevice = OpenVR.k_ulInvalidInputValueHandle,
       }
   };

+   var activeActionSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VRActiveActionSet_t));    
+   var error = OpenVR.Input.UpdateActionState(actionSetList, activeActionSize);
+   if (error != EVRInputError.None)
+   {
+       throw new Exception("Faild to update action state: " + error);
+   }
}

The second argument activeActionSize is the byte size of VRActiveActionSet_t the struct.

Get action value

We have updated the action state. Next, we will get the action value.
The function used to retrieve action value varies depending on the action type.

GetDigitalActionData()

This gets an on/off value likes the button is pushed or not.

GetAnalogActionData()

This gets analog values like thumb stick direction or how much trigger is pulled.

GetPoseActionData()

This gets a pose like controller position and rotation.

This time, we want to get the button on/off value so use GetDigitalActionData().
Get the action value with WakeUp action handle we prepared.

private void Update()
{
    var actionSetList = new VRActiveActionSet_t[]
    {
        new VRActiveActionSet_t()
        {
            ulActionSet =  actionSetHandle,
            ulRestrictedToDevice = OpenVR.k_ulInvalidInputValueHandle,
        }
    };

    var activeActionSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VRActiveActionSet_t));
    var error = OpenVR.Input.UpdateActionState(actionSetList, activeActionSize);
    if (error != EVRInputError.None)
    {
        throw new Exception("Failed to update action state: " + error);
    }

+   var result = new InputDigitalActionData_t();
+   var digitalActionSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(InputDigitalActionData_t));
+   error = OpenVR.Input.GetDigitalActionData(actionHandle, ref result, digitalActionSize, OpenVR.k_ulInvalidInputValueHandle);
+   if (error != EVRInputError.None)
+   {
+       throw new Exception("Failed to get WakeUp action data: " + error);
+   }
}

The 3rd argument digitalActionSize is the byte size of InputDigitalActionData_t structure.
The 4th argument ulRestrictToDevice is generally not used so set OpenVR.k_ulInvalidInputValueHandle.

The returned value type is InputDigitalActionData_t.

struct InputDigitalActionData_t
{
    bool bActive;
    VRInputValueHandle_t activeOrigin;
    bool bState;
    bool bChanged;
    float fUpdateTime;
};

The action on/off state is set to bState.
bChanged is true at the frame where the action state is changed.

Let’s detect the action.

private void Update()
{
    var actionSetList = new VRActiveActionSet_t[]
    {
        new VRActiveActionSet_t()
        {
            ulActionSet =  actionSetHandle,
            ulRestrictedToDevice = OpenVR.k_ulInvalidInputValueHandle,
        }
    };

    var activeActionSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VRActiveActionSet_t));
    var error = OpenVR.Input.UpdateActionState(actionSetList, activeActionSize);
    if (error != EVRInputError.None)
    {
        throw new Exception("Failed to update action state: " + error);
    }

    var result = new InputDigitalActionData_t();
    var digitalActionSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(InputDigitalActionData_t));
    error = OpenVR.Input.GetDigitalActionData(actionHandle, ref result, digitalActionSize, OpenVR.k_ulInvalidInputValueHandle);
    if (error != EVRInputError.None)
    {
        throw new Exception("Failed to get WakeUp action data: " + error);
    }

+   if (result.bState && result.bChanged)
+   {
+       Debug.Log("WakeUp is executed");
+   }
}

Run the program and push the Y button, then the message is logged to the Unity console.

Create event handler

Notify to Unity Event

Add UnityEvent into InputController.cs to notify WakeUp action, then call it with Invoke().

using System;
using UnityEngine;
using Valve.VR;
+ using UnityEngine.Events;

public class InputController : MonoBehaviour
{
+   public UnityEvent OnWakeUp; 

    private ulong actionSetHandle = 0;
    private ulong actionHandle = 0;

    // code omit.

    private void Update()
    {
        var actionSetList = new VRActiveActionSet_t[]
        {
            new VRActiveActionSet_t()
            {
                ulActionSet =  actionSetHandle,
                ulRestrictedToDevice = OpenVR.k_ulInvalidInputValueHandle,
            }
        };

        var activeActionSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VRActiveActionSet_t));
        var error = OpenVR.Input.UpdateActionState(actionSetList, activeActionSize);
        if (error != EVRInputError.None)
        {
            throw new Exception("Failed to update action state: " + error);
        }

        var result = new InputDigitalActionData_t();
        var digitalActionSize = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(InputDigitalActionData_t));
        error = OpenVR.Input.GetDigitalActionData(actionHandle, ref result, digitalActionSize, OpenVR.k_ulInvalidInputValueHandle);
        if (error != EVRInputError.None)
        {
            throw new Exception("Failed to get WakeUp action data: " + error);
        }

        if (result.bState && result.bChanged)
        {
-           Debug.Log("WakeUp is executed");
+           OnWakeUp.Invoke();
        }
    }

Attach event handler

Add method into WatchOverlay.cs that will be called when WakeUp action is executed.

public class WatchOverlay : MonoBehaviour
{

    ...

+   public void OnWakeUp()
+   {
+       // Show watch.
+   }
}

In the hierarchy, open InputController inspector, then set WatchOverlay.OnWakeUp() to OnWakeUp() field.

Add code to show/hide the watch

Update WatchOverlay.cs to show the watch only when WakeUp action is executed.

Hide watch by default

private void Start()
{
    OpenVRUtil.System.InitOpenVR();
    overlayHandle = Overlay.CreateOverlay("WatchOverlayKey", "WatchOverlay");

    Overlay.FlipOverlayVertical(overlayHandle);
    Overlay.SetOverlaySize(overlayHandle, size);
-   Overlay.ShowOverlay(overlayHandle);
}

Show watch by WakeUp action

Show overlay when the WakeUp action is executed.

public void OnWakeUp()
{
+   Overlay.ShowOverlay(overlayHandle);
}

Add hide method

Add HideOverlay() to OpenVRUtil.cs.

public static void ShowOverlay(ulong handle)
{
    var error = OpenVR.Overlay.ShowOverlay(handle);
    if (error != EVROverlayError.None)
    {
        throw new Exception("Failed to show overlay: " + error);
    }
}

+ public static void HideOverlay(ulong handle)
+ {
+     var error = OpenVR.Overlay.HideOverlay(handle);
+     if (error != EVROverlayError.None)
+     {
+         throw new Exception("Failed to hide overlay: " + error);
+     }
+ }

public static void SetOverlayFromFile(ulong handle, string path)
{
    var error = OpenVR.Overlay.SetOverlayFromFile(handle, path);
    if (error != EVROverlayError.None)
    {
        throw new Exception("Failed to draw image file: " + error);
    }
}

Hide watch after three seconds

This time, we use Unity Coroutine to wait three seconds.

WatchOverlay.cs

using System;
+ using System.Collections;
using UnityEngine;
using Valve.VR;
using OpenVRUtil;

public class WatchOverlay : MonoBehaviour
{
    public Camera camera;
    public RenderTexture renderTexture;
    public ETrackedControllerRole targetHand = ETrackedControllerRole.RightHand;
    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;
+   private Coroutine sleepCoroutine;

    ...

    public void OnWakeUp()
    {
        Overlay.ShowOverlay(overlayHandle);
+       if (sleepCoroutine != null)
+       {
+           StopCoroutine(sleepCoroutine);
+       }
+       sleepCoroutine = StartCoroutine(Sleep());
    }

+   private IEnumerator Sleep()
+   {
+       yield return new WaitForSeconds(3);
+       Overlay.HideOverlay(overlayHandle);
+   }
}

Run the program, and check the watch is shown for three seconds when the Y button is pushed.

Complete 🎉

The watch application is completed.
This tutorial ends here, so the final code organization is up to you.

Look back at what we touched on in this tutorial.

Initialize OpenVR
Create overlay
Draw image file
Change overlay size and position
Follow devices
Draw camera output
Create dashboard overlay
Process events
Controller input

We learned the basics of overlay application development.
OpenVR has many more APIs, so how about trying to make original tools?

The next page is additional information for more details.

Top comments (3)

Reavo End • Oct 5

Thank you for the tutorial! This is a wonderful starting point for anyone who wants to get into VR overlay development.

There were a few little issues and parts where you skipped steps, but I'll just point out the major ones on this page:

"Update action state" is missing several lines of code that exist in the next step, but are not highlighted in green
"Create event handler" step has two of the "using" keyword

Thanks again for your great contributions to the VR community.

kurohuku • Oct 8

Thank you for reading! I fixed it!

Dsel • Oct 11

The best Unity/VR tutorial I've ever seen. Very well done!

DEV Community