DEV Community: tststs

Unity MR Part 12: Improve Scene

tststs — Thu, 11 Jan 2024 08:51:30 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 The aim of this article is to enhance the visual and interactive elements of our scene. We will focus on several key improvements:

Graphics Enhancement: We'll enhance the graphics by configuring the Universal Render Pipeline (URP) assets.
Remove Debug Plane Material: The debug plane material, used for initial testing, will be removed for a cleaner look.
Door Shadow Effects: We'll set up the scene so that the door casts shadows on planes that will now be invisible, adding depth and realism.
Create a 'Behind the Door' Scene: We will develop a scene that is only revealed when the door is open, adding an element of surprise and exploration.

This article incorporates some advanced techniques that can significantly elevate your MR project. If you encounter any challenges or need further clarification, feel free to send us a message. Additionally, you can refer to the final project in our GitHub repository dedicated to this article series.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

Light Estimation

ℹ️ Light estimation for MR devices like the Meta Quest 3 involves assessing the real-world lighting conditions and applying that information to the virtual environment to create a more immersive and realistic experience.

Regrettably, as indicated by Unity in their forums, this feature is not currently supported.

Instantiate the door facing the user

Currently, the door Prefab is instantiated with a consistent rotation. Let's modify the MRArticleSeriesController Script so that the door faces the user upon instantiation. To do this, locate the OnButtonPressedRightAsync method in the MRArticleSeriesController Script and update it as follows.

private async void OnButtonPressedRightAsync(InputAction.CallbackContext context)
{
    Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync()");

    if (doorInstance != null) {
        Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync(): Door already instantiated");

        return;
    }

    if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
    {
        Pose pose = new(hit.point, Quaternion.identity);
        Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

        result.TryGetResult(out ARAnchor anchor);

        if (anchor != null) {
            // Instantiate the door Prefab
            doorInstance = Instantiate(door, hit.point, Quaternion.identity);

            // Unity recommends parenting your content to the anchor.
            doorInstance.transform.parent = anchor.transform;

            // Make the door face the user after instantiating
            doorInstance.transform.LookAt(new Vector3(
                Camera.main.transform.position.x,
                doorInstance.transform.position.y,
                Camera.main.transform.position.z
            ));
        }
    }
}

As you can see, we have added a call to doorInstance.transform.LookAt in the script. After making this addition, save the file and return to the Unity Editor to apply these changes.

ℹ️ For more information on the LookAt function and how it works, you can refer to the Unity documentation. This resource provides detailed insights into how LookAt can be used to orient objects in your scene: Unity - Scripting API: Transform.LookAt.

Update AR Default Plane

In this step, we will modify the material of the AR Default Plane to make it transparent while still allowing it to receive shadows. This adjustment enables us to see the shadows cast by the door on our floor. Implementing this change enhances the realism of our MR experience, as it allows virtual objects like the door to interact more naturally with the environment.

Go to Window → Package Manager and install the package https://github.com/Cyanilux/URP_ShaderGraphCustomLighting.git with Install package from git URL:

Install a package via git

Once you have completed the installation, close the Package Manager. Now, edit the AR Default Plane by double-clicking it in the Project window.

Replacing the default material with the ShadowReceiver material

Replace the default Material with ShadowReceiver as seen in the above screenshot. That’s all for the AR Default Plane.

If you can’t find ShadowReceiver via search you can also drag and drop the material from Packages/com.cyanilux.shadergraph-customlighting/Examples/ShadowReceiver.mat.

Optimize Lightning

In this step, we will focus on optimizing the Directional Light in our scene. You have the option to adjust the values as shown in the upcoming screenshot, or alternatively, you can modify the settings according to your own preferences and the specific requirements

Optimizing the Directional Light

Optimize URP

We are currently set to the Balanced quality level for the Android build. Let's switch to using the High Fidelity quality setting by default, as illustrated in the upcoming screenshot. This change will enhance the visual quality of our application, offering a more detailed and immersive experience on Android devices.

Set High Fidelity as the default quality on Android

Next, we must disable HDR in our URP asset. HDR is not supported in this context and will cause the application to crash if enabled. You can locate the URP asset at Assets/Settings/URP-HighFidelity.asset. The HDR option is available as the first option under the Quality section.

Disabling HDR in our URP asset

The High Fidelity settings utilize the Renderer Features Screen Space Ambient Occlusion, which significantly impacts performance. To optimize, you can either disable this feature or remove it entirely —both approaches are effective. You can find this setting in the asset located at Assets/Settings/URP-HighFidelity-Renderer.asset.

Disabling Screen Space Ambient Occlusion from the High Fidelity-Renderer

The behind the door

Let's create a straightforward effect that simulates a scene visible only when the door is open and the user looks through it. This approach will add an intriguing layer of interactivity to our project, enhancing the user experience by revealing a hidden scene.

Edit our door prefab by double-clicking Assets/Prefabs/Door in the Project window.

Disable the Door mesh through the inspector so we can look through the door.
Add a plane into the root of the prefab via Create → 3D → Plane.
Use the following Transform values as follows:

	X	Y	Z
Position	0	1.025	0
Rotation	90	0	0
Scale	0.1	1	0.205

The prefab should now resemble the appearance displayed in the upcoming screenshot.

Creating a Plane inside our door prefab

As you will notice, this method effectively creates the illusion where the "inside" of the door is only visible from the front. To observe this effect, take a look at the door from the back in the Scene view, as illustrated in the upcoming screenshot.

The “invisible” plane as seen from the back of the door

Now create a new Render Texture in Assets/Materials and name it DoorPlaneRenderTexture. Edit the Render Texture as follows:

DoorPlaneRenderTexture (Render Texture)

As you can observe, we have set the Size to 1024x2048, which correlates with our plane's approximate dimensions of a 1:2 ratio. This size selection ensures that the texture or material applied to the plane is properly scaled and aligned, reflecting the plane's physical proportions in the virtual environment.

Now create a new Material in Assets/Materials and name it DoorPlaneRenderTexture. Edit the Material as follows:

DoorPlaneRenderTexture (Material)

Now, choose the DoorPlaneRenderTexture as the Base Map as seen in the next screenshot.

Choosing the DoorPlaneRenderTexture as the Base Map

For the final step, assign the DoorPlaneRenderTexture material to the Mesh Renderer of the Plane.

Assign the DoorPlaneRenderTexture material to the Planes Mesh Renderer

Now, let's create a simple scene that will be rendered onto the plane. Before we start, make sure to return to the SampleScene and exit the prefab editor.

In your hierarchy, create an empty GameObject. You can do this by selecting Create Empty. Name this GameObject DoorPlaneScene.
Ensure the position of DoorPlaneScene is set to X: 0, Y: 0, Z: 0, unless it's already positioned there.
Create another empty GameObject as a child of DoorPlaneScene and name it CameraOffset . Set the Y position of CameraOffset to 1.1176 in its Transform properties.
To the CameraOffset GameObject, add a Camera component. This camera will capture the scene for rendering onto the plane.
Add another empty GameObject under DoorPlaneScene and name it Scene. This GameObject will hold the elements of your rendered scene.
Inside Scene, create a Cube (3D Object → Cube). Position this Cube with its Z coordinate set to 5.

These steps set up a basic scene structure, with a camera positioned to capture the scene, and a simple Cube as a visual element. The scene will be rendered from the perspective of the added camera.

The result looks as follows:

Hierarchy after adding the scene which will be rendered onto our plane

We aim for our new Camera to render only the contents of the Scene GameObject. To achieve this, navigate to Layers → Edit Layers..., as illustrated in the upcoming screenshot. Here, add a new user layer and name it DoorPlaneScene. This step is crucial for ensuring that the camera specifically captures the elements within the Scene, isolating its view to this particular part of your project for targeted rendering.

Adding a new user layer DoorPlaneScene

Now, select the DoorPlaneScene GameObject in the hierarchy and change its layer to DoorPlaneScene:

Selecting the DoorPlaneScene layer for our DoorPlaneScene GameObject

Confirm by also changing the layer for all children:

Changing the layer for all children

Lets configure the new Camera. Select our Main Camera in the hierarchy and copy the component values as seen in the next screenshot.

Copy the component values from Main Camera

Now select the new Camera again and paste the component values:

Paste component values to the new Camera

Now, let's configure some additional settings for the new Camera:

Set the tag of the Camera to Untagged.
Under the Rendering tab, change the Culling Mask to include only the DoorPlaneScene and Ignore Raycast. This ensures the camera renders only the objects in the DoorPlaneScene layer.
Assign DoorPlaneRenderTexture as the Output Texture. This means the camera's view will be rendered to this texture.
Set the Far value under Clipping Planes to 250. This determines how far the camera can see.
Choose Skybox as the Background Type under the Environment tab.

After making these changes, your camera should be set up as shown in the following screenshot.

The configured new Camera

Finally, create a prefab from the DoorPlaneScene and then remove it from the hierarchy, following the same process we used for the Reticle in the Raycasts article.

Our plan is to incorporate the DoorPlaneScene into our Door prefab. This way, DoorPlaneScene will only be rendered when necessary. To do this, edit the Door prefab and drag and drop the DoorPlaneScene into the root of the prefab. This action is depicted in the upcoming screenshot and ensures that the scene is efficiently integrated with the Door prefab.

Adding the DoorPlaneScene to our Door prefab

Save the prefab and return to the SampleScene.

Select the Main Camera again and set the Culling Mask from Everthing to: Default, TransparentFX, Ignore Raycast Water and UI.

Selecting the Culling Mask for the Main Camera

Drag and drop the prefab Door into your scene to investigate the changes we made. The plane now should look as follows:

Plain rendering the scene we just created

As you can observe, we now have our new scene projected onto the plane inside the door.

Next, re-enable the Door mesh, which we had disabled in a previous step. However, there's still an issue to address: the scene is static and doesn't adapt to the user's movements. Let's resolve this.

Create a new script and name it MimicCamera, then attach it to our newly added Camera. The script will be structured as follows:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MimicCamera : MonoBehaviour
    {
        public float offsetDistance = 30.0f;
        [SerializeField] private GameObject sceneCamera;
        private Camera mainCamera;
        public Transform DoorTransform { get; set; } = null;

        void Awake() {
            mainCamera = Camera.main;
        }

        void Update()
        {
            if (DoorTransform == null) {
                return;
            }

            // Calculate the direction and position offset from the door to the main camera
            Vector3 directionToCamera = mainCamera.transform.position - DoorTransform.position;
            Vector3 sceneCameraPosition = DoorTransform.position + directionToCamera.normalized * offsetDistance;

            sceneCameraPosition.y = sceneCameraPosition.y > 2f ? 2f : sceneCameraPosition.y;

            // Update the position and rotation of the scene camera
            sceneCamera.transform.position = sceneCameraPosition;
            sceneCamera.transform.rotation = Quaternion.LookRotation(DoorTransform.position - sceneCameraPosition);
        }
    }
}

Select the required values for the MimicCamera script as seen in the following screenshot:

Select the required values for the MimicCamera script

We still need to assign the DoorTransform when the Door is instantiated. To do this, edit the MRArticleSeriesController script and add a private field for the MimicCamera script. This modification is shown on line 1 in the following code snippet.

Afterward, assign the DoorTransform following our LookAt method call. If you need guidance, you can also refer to the code in the 12_ImproveScene project for a clearer understanding 12_ImproveScene.

...

// Make the door face the user after instantiating
doorInstance.transform.LookAt(new Vector3(
    Camera.main.transform.position.x,
    doorInstance.transform.position.y,
    Camera.main.transform.position.z
));

doorInstance.GetComponentInChildren<MimicCamera>().DoorTransform = doorInstance.transform;

...

Adding a complex scene behind the door

For the final step, we'll upgrade from the simple cube to a more complex scene. In our example, we utilized the Free Low Poly Nature Forest asset, which you can find here: Free Low Poly Nature Forest. Add the asset to your project.

Find the 'Demo_01 scene from the package within the Assets/Pure Poly/Free Low Poly Nature Pack/Scenes directory. Then, drag and drop it into your hierarchy, as demonstrated in the upcoming screenshot.

Drop the Demo_01 scene into our hierarchy

If the Demo_01 scene looks as follows, we need to replace the shader that is used for the material.

Demo_01 with an invalid material

Open the material located here: Assets/Pure Poly/Free Low Poly Nature Pack/Materials/PP_Standard_Material. Edit the material as follows:

Fixing the invalid material

Set the shader to Universal Render Pipeline/Lit.
As the Base Map choose the file PP_Color_Palette.
Set Smoothness to 0 for the Metallic Map.

Copy the Free_Forest GameObject, then edit our DoorPlaneScene prefab. Within the prefab, replace the cube with the Free_Forest GameObject, as illustrated in the next screenshot:

Replacing the cube with the Free_Forest GameObject in our DoorPlaneScene prefab

Set the Transform of the Free_Forest GameObject as follows:

	X	Y	Z
Position	0	-3.7	0
Rotation	0	-135	0
Scale	1	1	1

Set the Transform of the Free_Forest GameObject

Also ensure, that the layer of Free_Forest is set to DoorPlaneScene:

Setting the Free_Forest layer to DoorPlaneScene

Save the prefab and return to your scene. Now remove the Demo_01 scene from your hierarchy:

Remove the Demo_01 scene from your hierarchy

This enhancement significantly elevates the visual appeal and immersion of the scene, demonstrating the versatility and potential of using varied assets in your MR environment.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository. 12_ImproveScene.

Testing the app

We are now ready to test the final version of our app. Here's what you need to do:

Choose Build and Run in Unity.
Once the app is running, press the trigger on the left controller.
You should see a label displaying the message "...Listening...".
Say the phrase “open the door”.
After a short delay, the animation of the door opening should begin.
Feel free to move around and explore what lies behind the door.

This test will allow you to experience the full functionality of the app, from voice command recognition to the dynamic opening of the door and the revealing of the scene behind it. Enjoy the immersive experience of exploring the new environment you've created!

👏

Video of the app where the door opens when the phrase “open the door“ is recognized

Unity MR Part 11: Animation

tststs — Thu, 11 Jan 2024 08:51:23 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 The objective of this article is to animate the door in response to the voice command “open the door”. To achieve this, we will develop an animation specifically for the door and make necessary updates to the Player and OpenDoorConduit scripts. This integration of voice command functionality with animation will enhance the interactivity and realism of our application.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

Lets start with the animation. Edit our Door Prefab by double-clicking it within the Project window. Select the Door mesh as seen in the next screenshot.

Selecting the Door mesh on our door Prefab

Now add the component Animator via Add Component.

Adding the Animator component to the Door mesh

With the Door mesh still selected, navigate to Window → Animation → Animation to open the animation window in Unity. Once the animation window is open, click on the Create button. You will then be prompted to choose a location for saving the Animation Clip. Save it in a new folder Assets/Animations and name it DoorAnimation.

The animation window after clicking Create

The next step involves adding a property to animate. As we are crafting an “opening the door” animation, we need to animate the Transform rotation. To do this, click on Add Property, and then select Transform → Rotation. This action allows us to specifically target and animate the door's rotation, creating a realistic opening effect in our animation sequence.

Adding the rotation property to the animation

First move the Timeline to 60. You can either drag and drop the indicator or enter 60 on the left side of the Timeline. Then, enter -120 for the Rotation.y value.

The animation window with the configured Rotation.y property

In your Scene window the door should now look as follows:

Door state at the end of the animation

When creating an animation in Unity, the default setting typically causes the animation to loop continuously. However, since our objective is to trigger the door animation just once upon recognizing the correct voice command, we need to adjust the looping behavior in the Door Animation Controller.

To make this change, navigate to Assets/Animations in the Unity Editor and double-click on the Door Animation controller. This action will open the Animator window with the Door.controller loaded. In the Animator window, we can modify the settings to ensure that the door animation plays only once instead of looping endlessly.

Door controller in our Animator

Right-click on an empty space within the Animator window and select Create State -> Empty, as shown in the upcoming screenshot. This step will add a new, empty state to the animation controller.

Creating an empty state in the Animator

Right-click on the newly created state, named New State, and select Set as Layer Default State. This action will designate the New State as the default state for that particular layer in the animation controller, ensuring that this is the starting state when the animation sequence begins.

Setting New State as the default layer

The result should be as follows: Since the state is empty and no properties have been added to it, this results in the absence of any animation when the GameObject loads. This setup ensures that the door remains static initially, allowing for the animation to be triggered specifically by an event, such as a voice command, rather than starting automatically upon loading.

New State is the default layer

Now, select the DoorAnimation animation clip in your Projectwindow. Then, in the inspector, uncheck Loop Time.

We are now good to go to trigger the animation in our MRArticleSeriesController script.



using System.Collections;
using System.Collections.Generic;
using Meta.WitAi;
using Meta.WitAi.Requests;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using UnityEngine.XR.Interaction.Toolkit;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MRArticleSeriesController : MonoBehaviour
    {
        [SerializeField] private ARAnchorManager anchorManager;
        [SerializeField] private GameObject door;
        [SerializeField] private GameObject uI;
        [SerializeField] private InputActionReference buttonActionLeft;
        [SerializeField] private InputActionReference buttonActionRight;
        [SerializeField] private VoiceService voiceService;
        [SerializeField] private XRRayInteractor rayInteractor;
        private VoiceServiceRequest voiceServiceRequest;
        private VoiceServiceRequestEvents voiceServiceRequestEvents;
        private GameObject doorInstance;

        void OnEnable()
        {
            Debug.Log("MRArticleSeriesController -> OnEnable()");

            buttonActionRight.action.performed += OnButtonPressedRightAsync;
            buttonActionLeft.action.performed += OnButtonPressedLeft;
        }

        void OnDisable()
        {
            Debug.Log("MRArticleSeriesController -> OnDisable()");
            buttonActionRight.action.performed -= OnButtonPressedRightAsync;
            buttonActionLeft.action.performed -= OnButtonPressedLeft;
        }

        private void ActivateVoiceService()
        {
            Debug.Log("MRArticleSeriesController -> ActivateVoiceService()");

            if (voiceServiceRequestEvents == null)
            {
                voiceServiceRequestEvents = new VoiceServiceRequestEvents();

                voiceServiceRequestEvents.OnInit.AddListener(OnInit);
                voiceServiceRequestEvents.OnComplete.AddListener(OnComplete);
            }

            voiceServiceRequest = voiceService.Activate(voiceServiceRequestEvents);
        }

        private void DeactivateVoiceService()
        {
            Debug.Log("MRArticleSeriesController -> DeactivateVoiceService()");
            voiceServiceRequest.DeactivateAudio();
        }

        private void OnInit(VoiceServiceRequest request)
        {
            uI.SetActive(true);
        }

        private void OnComplete(VoiceServiceRequest request)
        {
            uI.SetActive(false);
            DeactivateVoiceService();
        }

        private async void OnButtonPressedRightAsync(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync()");

            if (doorInstance != null)
            {
                Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync(): Door already instantiated");

                return;
            }

            if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
            {
                Pose pose = new(hit.point, Quaternion.identity);
                Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

                result.TryGetResult(out ARAnchor anchor);

                if (anchor != null)
                {
                    // Instantiate the door Prefab
                    doorInstance = Instantiate(door, hit.point, Quaternion.identity);
                    // Unity recommends parenting your content to the anchor.
                    doorInstance.transform.parent = anchor.transform;
                }
            }
        }

        private void OnButtonPressedLeft(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressedLeft()");

            ActivateVoiceService();
        }

        public void OpenDoor()
        {
            Debug.Log("MRArticleSeriesController -> OpenDoor()");

            if (doorInstance == null)
            {
                Debug.Log("MRArticleSeriesController -> OpenDoor(): no door instantiated yet.");

                return;
            }

            Animator animator = doorInstance.GetComponentInChildren<Animator>();
            animator.Play("DoorAnimation", -1, 0);
        }
    }
}

Let's go over the modifications made to the MRArticleSeriesController script:

Added private GameObject doorInstance; This variable holds the reference to the instantiated door in the scene.
OnButtonPressedRightAsync: In this method, we now check if a door has already been instantiated in the scene. If a door is present, the method returns early to prevent another door from being instantiated.
OpenDoor: This public method will be called from the OpenDoorConduit. It triggers the door's animation using animator.Play, initiating the opening sequence of the door.

These changes enhance the functionality of the Player script, ensuring proper management and control of the door animation in response to user interactions and voice commands.

Now, edit the OpenDoorConduit script.



using System.Collections;
using System.Collections.Generic;
using Meta.WitAi;
using UnityEngine;

namespace Taikonauten.Unity.ArticleSeries
{
    public class OpenDoorConduit : MonoBehaviour
    {
        [SerializeField] private MRArticleSeriesController mRArticleSeriesController;
        private const string OPEN_DOOR_INTENT = "open_door";

        [MatchIntent(OPEN_DOOR_INTENT)]
        public void OpenDoor(string[] values)
        {
            Debug.Log("OpenDoorConduit -> OpenDoor()");

            string action = values[0];
            string entity = values[1];

            if (!string.IsNullOrEmpty(action) && !string.IsNullOrEmpty(entity))
            {
                if (action == "open" && entity == "door") {
                    mRArticleSeriesController.OpenDoor();
                }
            }
        }
    }
}

Let's go over the modifications made to the OpenDoorConduit script:

[SerializeField] private MRArticleSeriesController mRArticleSeriesController; This line declares a private variable and marks it with [SerializeField] so it can be assigned via the Unity Editor. This variable holds the reference to our Player component, allowing the script to interact with the Player component's public methods and properties.
OpenDoor: In this part of the script, we invoke the OpenDoor public method of the Player script which starts the door animation.

These modifications are essential for enabling communication and interaction between different components and scripts in our Unity project, particularly for handling the door-opening functionality.

Make sure to select the MRArticleSeriesController component in the inspector:

Selecting the MRArticleSeriesController for the Open Door Conduit script

Testing the app

We are now prepared to test the app. Select Build and Run.

Press the trigger on the left controller.
The label indicated "...Listening...".
Speak the word phrase “open the door”.
After a brief delay the door animation should now be playing.

Video of the app where the door opens when the phrase “open the door“ is recognized

In the next article, we will focus on enhancing the visual appeal of our scene. This will include implementing graphic improvements, adding a scene to be displayed behind the door, and disabling the plane material, which was previously used solely for debugging purposes. These refinements are aimed at elevating the aesthetic quality and immersive experience of our application, making it more engaging and visually appealing to users.

Unity MR Part 10: Voice SDK

tststs — Thu, 11 Jan 2024 08:51:19 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 The aim of this article is to incorporate the Meta - Voice SDK into our application, enabling it to respond to specific word sequences. This functionality will allow us to interact with the door we rendered in the previous article, providing a more immersive and interactive experience in our application.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

Login to the Unity Store Unity Asset Store - The Best Assets for Game Making and add Meta - Voice SDK - Immersive Voice Commands to your library.

Return to the Unity Editor and install the Meta - Voice SDK through the Package Manager. You can directly access My Assets via Window -> My Assets.

Adding the Meta - Voice SDK with the Package Manager

Before we begin utilizing the Meta - Voice SDK, it's necessary to create an account on Wit.ai . You can conveniently use your existing Meta developer account for this purpose by clicking on Continue with Meta on the Wit.ai landing page.

ℹ️ Wit.ai (often referred to as Won in Translation, by David Jacobs from its URL) is a natural language processing (NLP) service created by Facebook. It enables developers to build applications that can understand human language by providing a powerful and easy-to-use API.

Once you've set up your Wit.ai account, you can create a new application at Wit.ai. If you're unsure about what to name the application, simply go with unity_example.

After your app is created, click on unity_example. Then, as illustrated in the upcoming screenshot, add an Utterance for the Intent open_door. This step is crucial for training your application to recognize and respond to specific user inputs related to the action of opening a door.

ℹ️ An Utterance in the context of natural language processing (NLP), linguistics, and conversational AI, refers to a sequence of words or sounds made by a speaker. It's essentially a unit of speech. In practical terms, an utterance can be as short as a single word (like a command or an exclamation) or as long as a complete sentence or multiple sentences.

ℹ️ In Wit.ai, an Intent represents the purpose or goal behind a user's input, typically a spoken or written phrase. It's a fundamental concept in natural language understanding (NLU) and is used to categorize user utterances into specific actions that the application should perform.

1. Fill in the phrase “open the door” as the Utterance and select the open_door intent in the Intent dropdown.

Adding the Utterance for the Intent open_door

2. In the Utterance input field select the word “open”. This will open the entity form. Fill in action as the entity and click on + Create Entity.

Creating the action entity

3. In the Utterance input field select the word “door”. This will open the entity form. Fill in entity as the entity and click on + Create Entity.

Creating the entity entity

4. After creating the entities, they will now be highlighted as seen in the next screenshot.

The Utterance view after creating our entities

Now, click on Train and Validate. Once the training process is complete (indicator on the top left next to the app name), return to the Unity Editor and navigate to Oculus → Voice SDK → Get Started. In the first dialog enter the Wit Server Access Token. The access token can be found on the Wit.ai website under Management → Settings within your unity_example app.

Enter the Server Access Token

You will be prompted to choose a location for saving your Wit asset. Save it in your Assets/Settings folder and name it wit. Once you have saved the asset, the following screen will appear.

The Wit Configurations after entering the server token

Now, click on Specify Assemblies and uncheck everything except the first entry:

De-selecting unnecessary assemblies

Click on Generate Manifest then close the Voice Hub for now.

The next step is to respond to the Utterance “open the door”. Create a new Script under Assets/Scripts/VoiceSDK and name it OpenDoorConduit. Add the Script to the XR Origin (XR Rig) via Add Component.

Creating the OpenDoorConduit Script under Scripts/VoiceSDK

Adding the OpenDoorConduit Script to the XR Origin (XR Rig) GameObject

The Script looks as follows:

using System.Collections;
using System.Collections.Generic;
using Meta.WitAi;
using UnityEngine;

namespace Taikonauten.Unity.ArticleSeries
{
    public class OpenDoorConduit : MonoBehaviour
    {
        private const string OPEN_DOOR_INTENT = "open_door";

        [MatchIntent(OPEN_DOOR_INTENT)]
        public void OpenDoor(string[] values)
        {
            Debug.Log("OpenDoorConduit -> OpenDoor()");

            string action = values[0];
            string entity = values[1];

            if (!string.IsNullOrEmpty(action) && !string.IsNullOrEmpty(entity))
            {
                Debug.Log("OpenDoorConduit -> OpenDoor(): match");
            }
        }
    }
}

As you can see we are using MatchIntent with the open_door Intent which we created in a previous step. The method OpenDoor is automatically called by the ConduitDispatcher.

ℹ️ ConduitDispatcher is used to manage voice commands, direct them to the appropriate processing channels, or handle the distribution of responses or actions triggered by voice input.

Now, proceed by adding the App Voice Experience component to the XR Origin (XR Rig) GameObject. Once you have added this component, it's necessary to select the Wit configuration, which you have created during the Get Started process for Wit.

Adding App Voice Experience to the XR Origin (XR Rig) GameObject and selecting the Wit configuration

For the final step, we must add the Response Matcher. This component generates the Android Intent open_door in the Android manifest file, and upon receiving a successful response from Wit, it triggers our OpenDoor method, which we defined earlier.

To do this, open the Understanding Viewer via Oculus → Understanding Viewer. Enter “open the door“ in the Utterance field and click Send.

Now right-click value and select Add response matcher to XR Origin (XR Rig).

Adding the response matcher via the Understanding Viewer

Adding the value matcher via the Understanding Viewer

This will result in the following screenshot:

The XR Origin (XR Rig) after adding the response matcher and value matchers

Next, create an entry under On Multi Value Event and select the values as follows:

Selecting the OpenDoor method for the new On Multi Value Event

Android Setup

To ensure compatibility with the Voice SDK, some adjustments are needed for the Android build. Access the Project Settings by going to Edit → Project Settings. This step is crucial for configuring your project to work seamlessly with the Voice SDK on Android.

1. In the Project Settings, navigate to the Player section. There, change the Minimum API Level to Android 10.0 (API Level 29) and the Target API Level to Android 12L (API Level 32). These options are located under Other Settings.

Setting the Minimum and Target API level for Android

2. In the Project Settings, go to the Player section, find Application Entry Point under Other Settings and change the value from GameActivity to Activity.

Setting Activity instead of GameActivity as the Application Entry Point

ℹ️ You can find more information about application entry points in the Unity documentation Unity - Manual: Android application entry points. As of the time of writing, it's important to note that GameActivity is not compatible with the Meta Voice SDK.

3. In the Project Settings, go to the Player section, open the Publishing Settings tab and enable Custom Main Manifest. After activating this option, you will find the manifest file located at Assets/Plugins/Android/AndroidManifest.xml. This step is essential for gaining direct control over the Android manifest file, allowing you to make specific customizations needed for your project.

Enable the Custom Main Manifest option for Android

Open the AndroidManifest file in your code editor and modify its content as follows.

<?xml version="1.0" encoding="utf-8"?>
<manifest
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools">
    <application>
        <activity android:name="com.unity3d.player.UnityPlayerActivity"
                  android:theme="@style/UnityThemeSelector">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
            <meta-data android:name="unityplayer.UnityActivity" android:value="true" />
            <meta-data android:name="unityplayer.SkipPermissionsDialog" android:value="false" />
        </activity>
    </application>
</manifest>

Let's review the changes we've made. This will help us understand the adjustments and their impact on the application's functionality and compatibility.

We have removed the GameActivity block, as only one Activity is permitted and we previously opted for UnityActivity instead of GameActivity in the Project Settings.
We included the unityplayer.SkipPermissionsDialog setting with a value of false to ensure that required permission dialogs are not automatically bypassed. This adjustment is important for guaranteeing that the application appropriately prompts users for necessary permissions, aligning with best practices for user consent and app functionality.

Adding some UI

Before we can test our implementation, it is essential to integrate some user interface elements into the scene. We will add a UI Label that becomes visible when the application starts listening to the user's voice. This label will then disappear once the recording ceases, triggered by a successful response from Wit. This UI component plays a crucial role in providing visual feedback to the user about the state of voice recognition within the application.

To set up the user interface for voice recognition feedback, follow these steps:

1. Create an empty GameObject in your scene and name it UI.
2. Within the UI GameObject, add another empty GameObject and name it VoiceSDK.
3. With the VoiceSDK GameObject selected in the hierarchy attach the Lazy Follow script via Add Component. Configure the script as follows:

Enable the Custom Main Manifest option for Android

4. Add a Canvas to the VoiceSDK GameObject by right-clicking it and navigating to UI -> Canvas.

5. Inside the Canvas, add a Text element by choosing UI → Text - TextMeshPro.

This setup creates a structured UI hierarchy in your scene, with the VoiceSDK GameObject serving as a container for the elements that will provide visual feedback for voice recognition. The Lazy Follow script will manage the positioning, and the TextMeshPro element will display the necessary information or status messages.

Your hierarchy should now look as follows:

Hierarchy after adding the UI

Select the EventSystem GameObject in your hierarchy and delete and add components like follows:

The UI EventSystem

Don’t forget to remove the Standalone Input Module if any.

Next, we need to configure our Canvas and Text (TMP) elements. Select the Canvas GameObject in your hierarchy and set it up as follows (Add components as seen in the screenshot via Add Component).

ℹ️ We won't be delving into UI-related topics in this article series. For those who need assistance or guidance with Unity's UI system, I recommend checking out the Unity documentation. It provides comprehensive resources and tutorials that can help you understand and effectively use Unity's UI tools in your projects.

The complete Canvas configuration

The complete Text (TMP) configuration

Lastly, deactivate the VoiceSDK GameObject, as we won't be displaying it immediately. The visibility of the UI will be managed later through our script.

ℹ️ If you're not familiar with how to deactivate a GameObject in the Unity Inspector, I recommend consulting the Unity documentation: Unity - Manual: Deactivate GameObjects.

Updating our MRArticleSeriesController Script

For the final step in this article, we'll update our MRArticleSeriesController Script to enable and disable the Voice Service using the left Trigger. This modification will allow for straightforward control of the Voice Service directly through user input, enhancing the interactive capabilities of our application.

using System.Collections;
using System.Collections.Generic;
using Meta.WitAi;
using Meta.WitAi.Requests;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using UnityEngine.XR.Interaction.Toolkit;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MRArticleSeriesController : MonoBehaviour
    {
        [SerializeField] private ARAnchorManager anchorManager;
        [SerializeField] private GameObject door;
        [SerializeField] private GameObject uI;
        [SerializeField] private InputActionReference buttonActionLeft;
        [SerializeField] private InputActionReference buttonActionRight;
        [SerializeField] private VoiceService voiceService;
        [SerializeField] private XRRayInteractor rayInteractor;
        private VoiceServiceRequest voiceServiceRequest;
        private VoiceServiceRequestEvents voiceServiceRequestEvents;

        void OnEnable()
        {
            Debug.Log("MRArticleSeriesController -> OnEnable()");

            buttonActionRight.action.performed += OnButtonPressedRightAsync;
            buttonActionLeft.action.performed += OnButtonPressedLeft;
        }

        void OnDisable()
        {
            Debug.Log("MRArticleSeriesController -> OnDisable()");
            buttonActionRight.action.performed -= OnButtonPressedRightAsync;
            buttonActionLeft.action.performed -= OnButtonPressedLeft;
        }

        private void ActivateVoiceService()
        {
            Debug.Log("MRArticleSeriesController -> ActivateVoiceService()");

            if (voiceServiceRequestEvents == null)
            {
                voiceServiceRequestEvents = new VoiceServiceRequestEvents();

                voiceServiceRequestEvents.OnInit.AddListener(OnInit);
                voiceServiceRequestEvents.OnComplete.AddListener(OnComplete);
            }

            voiceServiceRequest = voiceService.Activate(voiceServiceRequestEvents);
        }

        private void DeactivateVoiceService()
        {
            Debug.Log("MRArticleSeriesController -> DeactivateVoiceService()");
            voiceServiceRequest.DeactivateAudio();
        }

        private void OnInit(VoiceServiceRequest request)
        {
            uI.SetActive(true);
        }

        private void OnComplete(VoiceServiceRequest request)
        {
            uI.SetActive(false);
            DeactivateVoiceService();
        }

        private async void OnButtonPressedRightAsync(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync()");

            if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
            {
                Pose pose = new(hit.point, Quaternion.identity);
                Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

                result.TryGetResult(out ARAnchor anchor);

                if (anchor != null)
                {
                    // Instantiate the door Prefab
                    GameObject _door = Instantiate(door, hit.point, Quaternion.identity);

                    // Unity recommends parenting your content to the anchor.
                    _door.transform.parent = anchor.transform;
                }
            }
        }

        private void OnButtonPressedLeft(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressedLeft()");

            ActivateVoiceService();
        }
    }
}

Let's review the updates quickly:

Added a field named uI to hold the GameObject that will be activated or deactivated based on the VoiceService state.
Included a field called voiceService to reference the App Voice Experience component added to the XR Origin (XR Rig) GameObject.
Introduced a field voiceServiceRequest to store the active request to the VoiceService.
Added a voiceServiceRequestEvents field, which is passed to the VoiceService. This ensures that the OnInit and OnComplete methods are called by the VoiceService.
In OnEnable and OnDisable we add and remove the OnButtonPressedLeft action, so we can respond to the left controller Trigger press.
ActivateVoiceService method, which activates the VoiceService, creates VoiceServiceRequestEvents if not already initialized, and is called via OnButtonPressedLeft when the user presses the left Trigger.
DeactivateVoiceService which simple deactivates the recording of the VoiceService.
OnInit Invoked when the VoiceService starts listening. It enables the UI GameObject to inform the user that the app is now listening.
OnComplete called when a voice request succeeds. It also invokes DeactivateVoiceService to stop the VoiceService from listening.
OnButtonPressedLeft triggered when the left Trigger is pressed.

Please be aware that we have also renamed some variables and methods. After saving these changes, remember to return to the Unity Editor and update the fields for the Player component accordingly.

Updating our values on the MRArticleSeriesController component

Testing the app

We are now prepared to test the app. Select Build and Run, and once the app is running, press the trigger on the left controller. You should see a label appear in front of you indicating "...Listening...". At this point, say the phrase "open the door". After a brief delay, the label should disappear. This process will allow you to verify the functionality of the voice recognition feature in your application.

View of the app after a left Trigger press

Additionally, you can verify in the console if the Intent was triggered, as shown in the following screenshot. This will provide a clear indication of whether the voice command was successfully recognized.

Console output after the voice command was successfully recognized

In our upcoming article, we will take an exciting step forward by leveraging the voice command functionality we've established to initiate an animation that opens the door. This integration represents a significant enhancement in our application, blending voice recognition with dynamic visual feedback.

Unity MR Part 9: Instantiate Prefab

tststs — Thu, 11 Jan 2024 08:51:13 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 The objective of this article is to demonstrate how to instantiate a Prefab — in our example, a door — and subsequently anchor it within the MR environment. This process is key to effectively integrating virtual objects, like the door, into the MR space, ensuring they are correctly positioned and maintained within the real-world context.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

First, let's create the door Prefab required for this article. Download the following archive door.rar and extract its contents into your Assets/3DModel folder. Confirm the following dialog with Fix now after dropping the files.

Confirm with Fix now to mark the texture as a normal map

Then, create a new Prefab in your Assets/Prefabs folder using the same method as we did for the reticle in a previous article. Once you've completed these steps, your Prefab should look like this:

The Door Prefab in edit mode

We are now set to instantiate the door Prefab and anchor it within the environment. This action will occur when the user hovers over a valid plane and presses the Trigger button.

Here's how the updated MRArticleSeriesController script will look:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using UnityEngine.XR.Interaction.Toolkit;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MRArticleSeriesController : MonoBehaviour
    {
        public ActionBasedController controller;
        public InputActionReference buttonAction;
        public XRRayInteractor rayInteractor;
        public ARAnchorManager anchorManager;
        public GameObject door;

        void OnEnable()
        {
            Debug.Log("MRArticleSeriesController -> OnEnable()");
            buttonAction.action.performed += OnButtonPressedAsync;
        }

        void OnDisable()
        {
            Debug.Log("MRArticleSeriesController -> OnDisable()");
            buttonAction.action.performed -= OnButtonPressedAsync;
        }

        private async void OnButtonPressedAsync(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressed()");

            if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
            {
                Pose pose = new(hit.point, Quaternion.identity);
                Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

                result.TryGetResult(out ARAnchor anchor);

                if (anchor != null)
                {
                    // Instantiate the door Prefab
                    GameObject \_door = Instantiate(door, hit.point, Quaternion.identity);

                    // Unity recommends parenting your content to the anchor
                    // instead of adding a ARAnchor component to the GameObject
                    \_door.transform.parent = anchor.transform;
                }
            }
        }
    }
}

Let's review the updates quickly:

A public GameObject member named door has been added, enabling us to select the door Prefab in the Inspector.
Instead of simply logging the anchor instance, we now verify if the returned anchor is not null.
The door Prefab is instantiated at the raycast hit point.
The door (our content) is parented to the anchor, following Unity's recommended practices.

With these updates to the MRArticleSeriesController Script, don't forget to select the door Prefab in the Inspector to complete the setup.

Selecting the door Prefab in our MRArticleSeriesController Script

To test the application, choose Build and Run, as detailed in the previous articles. Hover over a Plane and press the Trigger button. If all configurations are correct, you should see a similar result as seen in the next video:

Selecting the door Prefab in our MRArticleSeriesController Script

In our forthcoming article, we will take an exciting step in our MR journey by integrating the VoiceSDK. This addition will enable voice interaction within our MR environment, particularly focusing on using voice commands to interact with the door we've created. We'll guide you through the process of integrating the VoiceSDK into your Unity project, setting up voice recognition, and configuring it to recognize specific commands -- such as "open the door".

Unity MR Part 8: Anchors

tststs — Thu, 11 Jan 2024 08:51:08 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 This article will guide you through the concept of Anchors MR, explaining their significance and functionality. Additionally, it will provide a comprehensive tutorial on implementing Anchors within the Unity platform.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

The aim of this article is to conduct hit tests on planes we created previously, using our controller. We'll then instantiate an Anchor at the identified position. This process is fundamental for accurately positioning virtual objects in relation to the real world.

ℹ️ Anchors in MR are virtual reference points that are fixed to a specific location in the real world. They are used to maintain the position and orientation of virtual objects consistently as the user moves around or interacts with the MR environment.

A typical use case for anchors is to place virtual content in the physical world.

AR Anchor Manager component

The AR Anchor Manager component creates GameObjects for each anchor and is a critical element for managing AR anchors. It acts as a central point for creating, tracking, and updating anchors in the AR environment. This component is essential for ensuring that virtual objects are anchored accurately to real-world locations, maintaining their position and orientation consistently.

⚠️ Throughout its lifespan, the Meta Quest 3 device usually undertakes extra tasks to keep the anchor's position and orientation up to date. Given that anchors are typically resource-intensive objects, it's advisable to use them wisely.

Adding AR Anchor Manager component to the Scene

In the hierarchy view find the XR Origin (XR Rig) GameObject and add the AR Anchor Manager script via Add Component.

The XR Origin (XR Rig) GameObject with the added AR Anchor Manager Script

There's no need to modify any fields in the AR Anchor Manager. The trackables Changed is a list of ARTrackables, managed by the ARTrackableManager. Additionally, the Anchor Prefab field is only necessary if there's a requirement to enhance the default Prefab instantiated for each Anchor.

ℹ️ You can read more about the different Managers in ARFoundation here: Managers | AR Foundation | 6.0.0-pre.5.l.

Let's move on to updating our MRArticleSeriesController.cs script. In this update, we'll adjust the script to instantiate an Anchor at the Pose of a valid Raycast Hit. For the moment, we'll simply log the creation of the Anchor. In the following article, we will focus on instantiating a Prefab and attaching it to this Anchor.

ℹ️ Pose is a representation of a Position, and a Rotation in 3D Space. This structure is used primarily in XR applications to describe the current "pose" of a device in 3D space.

The revised script will be structured as follows:



using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using UnityEngine.XR.Interaction.Toolkit;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MRArticleSeriesController : MonoBehaviour
    {
        public ActionBasedController controller;
        public InputActionReference buttonAction;
        public XRRayInteractor rayInteractor;
        public ARAnchorManager anchorManager;

        void OnEnable()
        {
            Debug.Log("MRArticleSeriesController -> OnEnable()");
            buttonAction.action.performed += OnButtonPressedAsync;
        }

        void OnDisable()
        {
            Debug.Log("MRArticleSeriesController -> OnDisable()");
            buttonAction.action.performed -= OnButtonPressedAsync;
        }

        private async void OnButtonPressedAsync(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressed()");

            if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
            {
                Pose pose = new(hit.point, Quaternion.identity);
                Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

                result.TryGetResult(out ARAnchor anchor);

                Debug.Log(anchor);
            }
        }
    }
}

Before testing the app, it's important to select our AR Anchor Manager. To do this, locate the XR Origin (XR Rig) GameObject in your hierarchy, and then, as indicated in the following screenshot, choose the AR Anchor Manager.

Selecting the AR Anchor Manager for the MRArticleSeriesController.cs Script

Console output for the Meta Quest 3 device

In the upcoming article, we will delve into the process of instantiating a prefab in a MR environment using Unity. This technique is key to bringing virtual objects into your MR space, allowing for a richer and more interactive experience.

Unity MR Part 7: Raycasts

tststs — Thu, 11 Jan 2024 08:51:03 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 This article will guide you through the concept of Raycasts in MR, explaining their significance and functionality. Additionally, it will provide a comprehensive tutorial on implementing Raycasts within the Unity platform.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

The aim of this article is to conduct hit tests on planes we created previously, using our controller. We'll then, in the next article, place an Anchor at the identified position. This process is fundamental for accurately positioning virtual objects in relation to the real world.

ℹ️ Raycasts in Unity and MR are used to determine if and where a virtual ray intersects with objects in the game world or MR environment. They're crucial for interaction, allowing the detection of objects, surfaces, or points in 3D space. In MR, Raycasts can be used for selecting or interacting with virtual objects overlaid on the real world.

Creating a custom Reticle

ℹ️ In AR, a reticle is a visual indicator, often a small pointer or dot, used to help users accurately target and interact with virtual elements overlaid onto the real world. It provides a focal point for interaction, indicating where actions like tapping, selecting, or placing virtual objects will occur. Reticles are especially useful in AR experiences to improve precision and user experience in environments where direct touch interactions are not possible.

To create a custom Reticle, we'll utilize the asset provided in the link. Download the file directly into your Assets folder. For the purposes of this article, we'll organize it in a subfolder named 3DModels. Alternatively, you can download the asset to a different folder and then drag and drop it into the 3DModels folder within Unity.

⬇️ arrow.fbx

Let's begin by creating a visual indicator, a Reticle, which will appear when we hover over any of the planes managed by the AR Plane Manager.

First, create a new empty GameObject via Create → Empty in your hierarchy and name it 'Reticle'.
Next, drag and drop this GameObject into your Prefabs folder, then remove the Reticle GameObject from the hierarchy.
Double-click the prefab Reticle you just created to start editing it.
Make sure the Reticle GameObject's position is set to X: 0, Y: 0, Z: 0.
Then, drag and drop the .fbx file from your 3DModels folder onto the Reticle GameObject in your hierarchy view.
Make sure the arrow GameObject's position is set to X: 0, Y: 0, Z: 0.
Finally, adjust the position, rotation, and scale of the Cone and Cylinder to suit your needs.

Cone	X	Y	Z
Position	0	0	-0.1
Rotation	90	180	0
Scale	0.2	0.2	0.2

Cylinder	X	Y	Z
Position	0	0	-0.3
Rotation	90	180	0
Scale	0.2	0.2	0.2

With the specified values, you will achieve the Prefab configuration as shown in the upcoming screenshot. If you are designing a custom Reticle, remember to modify only the position, rotation, and scale of the Mesh components. It's important to avoid altering these values for the Reticle GameObject itself to maintain its intended functionality and alignment.

Custom Reticle after updating Position, Rotation and Scale

Updating our AR Default Plane

We aim for the Reticle to be visible only when hovering over one of our AR Default Planes. To achieve this, modify the AR Default Plane Prefab (by double clicking the AR Default Plane in the Assets/Prefabs folder), add the XR Simple Interactable Script and select the newly created Reticle Prefab as the Custom Reticle.

Setting the custom Reticle on the AR Default Plane

This Prefab will now be instantiated at the position where our ray intersects with any of the planes registered in our AR Plane Manager. When you run your app with these settings, you'll observe that the custom Reticle is now accurately indicating the plane being hovered over.

The custom Reticle in MR

Improve the appearance of the XR Interactor Line Visual

In this step, we'll adjust the color gradients for Valid, Invalid, and Blocked states in the XR Interactor Line Visual. This modification will change the line's color based on interaction: it will display one color when hovering over a valid target (in our scenario, detected planes) and another color when no valid target is hovered over (anything other than planes). This visual cue enhances the user's understanding of interaction possibilities within the AR environment.

You can use the following color gradients as seen in the upcoming screenshots or use your own custom gradients.

XR Interactor Line Visual with updated color gradients

The Valid color gradients with a starting color of #00A0FF.

Valid Color Gradient

The Invalid color gradients with a starting color of #FFFFFF.

Invalid Color Gradient

The Blocked color gradients with a starting color of #FFEB04.

Blocked Color Gradient

Ensure that you also apply the same values to the Right Controller.

That concludes the adjustments to the color gradients of the XR Interactor Line Visual. We'll examine the outcome of these changes at the end of this article, providing a comprehensive view of how these modifications enhance the visual feedback in our application.

Improve appearance of the Reticle

For the final visual enhancement, let's create a new Material for the custom Reticle. Go ahead and make a new Material in your Assets/Materials folder and name it ‘Reticle’. You can either use the following suggested values or select your own based on your specific needs. This step will ensure that the custom Reticle has a distinctive and appropriate appearance in your application.

The new Reticle Material

Base Map color for the Reticle Material

Edit the Reticle Prefab by double-clicking it in Assets/Prefabs. Now, drag and drop the new Material onto both meshes within the Reticle Prefab (Cone and Cylinder). Once you've done this, your Reticle Prefab should resemble the appearance shown in the upcoming screenshot.

Retacle Prefab with the updated Material

Retacle Prefab with the updated Material in our app

Reacting to a OnButtonPressed Event and check if the Ray hits a Plane

For the final step in this article we dive a bit into code. Create a Script named MRArticleSeriesController.cs in your Assets/Scripts folder and use the following code.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.Interaction.Toolkit;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MRArticleSeriesController : MonoBehaviour
    {
        public ActionBasedController controller;
        public InputActionReference buttonAction;
        public XRRayInteractor rayInteractor;

        void OnEnable()
        {
            Debug.Log("MRArticleSeriesController -> OnEnable()");
            buttonAction.action.performed += OnButtonPressed;
        }

        void OnDisable()
        {
            Debug.Log("MRArticleSeriesController -> OnDisable()");
            buttonAction.action.performed -= OnButtonPressed;
        }

        private void OnButtonPressed(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressed()");

            if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
            {
                Debug.Log(hit);
            }
        }
    }
}

Drag and drop the script onto your XR Origin (XR Rig). Then, as shown in the upcoming screenshot, select an ActionBasedController, InputActionReference, and XRRayInteractor. This step integrates the script with the XR Rig, ensuring that the controller, input actions, and ray interactions are properly configured.

Adding the MRArticleSeriesController script to our XR Origin (XR Rig) GameObject

Let's proceed to build and deploy the app onto our headset to see the results. Navigate to File → Build Settings and, as depicted in the upcoming screenshot, enable the Development Build option under the Android section.

Enable Development Build in Build Settings

To ensure proper functionality, it's necessary to disable Vulkan in the Graphics APIs in the Player section. Navigate to Edit -> Project Settings, and as illustrated in the following screenshot, remove Vulkan from the list of enabled Graphics APIs. This adjustment is crucial for compatibility.

As of Unity version 2023.2.3f and Unity OpenXR: Meta 1.0.1, creating a development build with Vulkan enabled leads to crashes. This known issue highlights the importance of avoiding Vulkan in your development builds with these specific versions to ensure stability and prevent unexpected crashes.

Removing the Vulkan Graphics API from Graphics APIs

Select Build and Run to test the application. Since we have configured only the Right Controller, which suffices for our purposes, you should observe the following console output when the Trigger button is pressed.

To view the output from the build on your headset, you need to choose the connected Oculus Quest in your Console window.

Selecting the Meta Quest 3 in Console

As demonstrated in the upcoming screenshot, the Debug.Log statement generates the expected output. This visual confirmation in the screenshot effectively showcases that the application is functioning as intended, aligning with our anticipated results.

AndroidPlayer "Oculus_Quest" Player -> OnButtonPressed()
AndroidPlayer "Oculus_Quest" UnityEngine.RaycastHit

The debug logs after pressing the Trigger button of the right controller

In our next article, we'll be focusing on an essential aspect of AR development: Planes and the AR Plane Manager in Unity. Planes are fundamental to understanding and interacting with the real world in AR applications. We will explore how the AR Plane Manager detects and manages these planes, allowing for the placement of virtual objects on flat surfaces like floors, tables, or walls.

Unity MR Part 6: Planes

tststs — Thu, 11 Jan 2024 08:50:58 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 In this article, we'll focus on rendering a GameObject for every detected horizontal plane obtained from the Space Setup. Additionally, we will create a Material specifically for each of these GameObjects. This process is integral to visualizing spatial data and enhancing the interactive elements of our project.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

ℹ️ Before beginning, it's necessary to execute the Space Setup on your Meta Quest, found under Physical Space → Space Setup. This process enables mapping of various objects like doors, walls, tables, couches, and more. The Meta Quest memorizes these details, allowing for an enhanced virtual reality experience that fully utilizes your area's boundaries.

Adding the AR Plane Manager

Add the AR Plane Manager to our existing XR Origin (XR Rig) GameObject by using Add Component.

The XR Origin (XR Rig) GameObject with the added AR Plane Manager

Creating a Prefab for our detected planes

As illustrated in the screenshot, it's necessary to define a Plane Prefab, which we will create in the following steps. To do this, go to your project's hierarchy and create an AR Default Plane by right-clicking and navigating to XR → AR Default Plane.

ℹ️ The AR Default Plane in Unity is a preset GameObject used to visually represent detected planes in an AR environment. It's often used as a default visual marker for floors, walls, or other flat surfaces that the AR system identifies in the real world, providing a reference point for placing virtual objects in relation to these physical surfaces.

Hierarchy with the added AR Default Plane

Create a new folder named Prefabs in your Assets folder.

Then, create a Prefab by dragging the AR Default Plane GameObject from your hierarchy into your Assets/Prefabs folder. For those unfamiliar with the concept of Prefabs, it's advisable to refer to Unity's official documentation on Prefabs Unity - Manual: Prefabs for a thorough understanding.

By dragging and dropping the GameObject into the Prefabs folder, you create a reusable Prefab. After you have created this Prefab, remove the AR Default Plane GameObject from your hierarchy.

Creating the AR Default Plane Prefab

Next, select your XR Origin (XR Rig) in the hierarchy and assign the AR Default Plane Prefab to the AR Plane Manager. The AR Plane Manager will instantiate an AR Default Plane GameObject at each plane's position based on spatial data.

Selecting the AR Default Plane for the AR Plane Manager

For the purposes of this article series, we only require horizontal planes. As shown in the upcoming screenshot, select Horizontal as the Detection Mode, preparing the system to specifically detect and interact with horizontal surfaces in the environment.

Setting the Detection Mode for the AR Plane Manager

Setup the default plane Prefab

Finally, we need to modify the AR Default Plane Prefab. Open it in the Prefab Editor by double-clicking on it in your Project window. The primary adjustment involves assigning an appropriate material to the mesh.

ℹ️ For those unfamiliar with creating materials in Unity, more information can be found in the Unity documentation, which provides comprehensive guidance on material creation and customization in Unity environments.

Create a new Material in your Assets/Materials folder and name it PlaneMaterial. You’ll find the Assets folder under Project. Create an empty folder Materials if not already done. As this material is intended for debugging purposes, we'll construct a basic one. This straightforward approach streamlines the process, ensuring that the material effectively serves its debugging function without the need for complex configurations or properties. You can choose the values as in the next screenshot or choose your own Material.

PlaneMaterial Surface Options/Inputs (Example)

Also update the Base Map color under Surface Inputs. Choose FFFFFFF and for Alpha 0.

Base Map color for PlaneMaterial

Now that we've created the material, it's time to assign it to the Mesh Renderer of your newly created AR Default Plane prefab. Edit the Prefab by double-clicking the Ar Default Plane under Assets/Prefabs/AR Default Plane.

Assigning the PlaneMaterial to the Mesh Renderer

If there's an existing entry in the Mesh Renderer, remove it first. Then, add PlaneMaterial as Element 0. This step ensures that our custom material is properly applied, allowing for the intended visual representation and debugging functionality within the AR environment.

The second and final modification for our AR Default Plane involves turning off the Cast Shadows option. We don't want planes casting shadows on each other in our space, as this would appear unrealistic. The real environment already casts shadows, so disabling this feature ensures a more natural and coherent visual experience in our AR setup.

Disable Cast Shadows on our AR Default Plane

Testing the app

That's all for now. We're ready to test our application. Simply hit Build And Run in the Unity Editor as explained in the last articles.

As shown in the upcoming screenshot, you'll see planes like the floor the couch are now rendered. Additionally, notice that raycasting is operational. In the next article about Anchors, we'll focus on enhancing the color of the raycast when a hit test is successful, further refining our application.

The app after adding planes to the scene

In our upcoming article, we will explore the critical role of Raycasts in MR environments. Raycasts are fundamental in MR for detecting and interacting with objects in a virtual space. We will cover how Raycasts work, their applications in our scenario, and how to effectively implement them in Unity.

Unity MR Part 5: Left and Right Controllers

tststs — Thu, 11 Jan 2024 08:50:53 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 This article will teach you how to set up the left and right controllers for the Meta Quest 3 in Unity. This setup is essential for enabling interaction with various elements within your Unity project, enhancing the user experience on the Meta Quest 3 platform.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

To keep the process straightforward, we'll use the starter assets from the XR Interaction Toolkit plugin. This approach simplifies the setup process, focusing on achieving a practical outcome quickly and efficiently.

Import XR Interaction Toolkit Starter Assets

We will utilize some of the XR Interaction Toolkit starter assets in this article. Go to the Package Manager via Window → Package Manager and import Starter Assets which are located in the tab Samples as seen in the next screenshot.

You can locate the XR Interaction Toolkit in the Package Manager by accessing Unity Registry and searching for 'interact'.

Importing Starter Assets for the XR Interaction Toolkit

Enable the new Input System

ℹ️ The New Input System in Unity is a flexible and extensible system designed to replace the older input methods. It provides a more unified and consistent way to handle different input devices like keyboards, mice, gamepads, and touchscreens. This system allows for more customized and complex input configurations, making it easier to develop games and applications that can be controlled across various platforms and devices.

Go to your Project Settings and enable the new input system. You’ll find it under Active Input Handling under the section Player and the tab Other Settings.

After selecting the new input system you will be asked to restart the Unity editor.

After restarting, the new input is now active. No additional steps required for now.

Adding the AR Input Manager

ℹ️ The AR Input Manager in Unity's AR Foundation is a component that manages input in AR applications. It serves as a central point for handling AR-specific input data, such as touch gestures, camera movement, and device orientation. This manager is essential for creating interactive and user-responsive AR experiences, as it efficiently processes and translates real-world user interactions into actions within the AR environment.

Select the Session GameObject in your hierarchy and add the AR Input Manager via Add Component.

Our Session GameObject with the added AR Input Manager

The Input Action Manager

The Input Action Manager in Unity is a component that handles input actions within the New Input System. It allows developers to define and manage complex input actions like button presses, gestures, or movement inputs. This manager makes it easier to organize and respond to various user inputs in a structured and efficient manner, crucial for creating responsive and interactive gaming or application experiences.

ℹ️ For those interested in exploring the Input Manager in greater detail, additional information is available in the provided reference. This resource offers a comprehensive look into the functionalities and applications of the Input Manager, enhancing your understanding and proficiency in utilizing this component in your projects. Input System | Input System | 1.7.0.

Select the XR Origin (XR Rig) GameObject in your hierarchy.

Add XRI Default Input Actions to the list of Action Assets

In the screenshot provided, you'll notice that we've added XRI Default Input Actions to the list of Action Assets. These assets were included as part of the Starter Assets from the XR Interaction Toolkit plugin, highlighting how the toolkit simplifies the integration of default input actions into your project.

If you're curious about the specific Input Actions included in the Action Asset, simply double-click on the selected action asset. This action will reveal the details and configurations of the Input Actions, providing a clear understanding of the functionalities they encompass.

XRI Default Input Actions in detail

Adding XR Input Modality Manager

The XR Input Modality Manager in Unity dynamically switches between hands and controllers during runtime, depending on their tracking status.

if hands start being tracked, it switches to hand interactors.
if motion controllers are activated, it switches to controller interactors once they're tracked.

📚 This component is valuable even without hand tracking, as it deactivates controller GameObjects until they are tracked, preventing them from appearing by default.

Select the XR Origin (XR Rig) GameObject in your hierarchy and add the XR Input Modality Manager via Add Component. Select Left Controller and Right Controller as seen in the screenshot.

Selecting the left and right controller for the XR Input Modality Manager

Choose a controller preset

For configuring the left and right controllers, we choose a controller preset imported through the starter assets. In your hierarchy select the GameObject Left Controller. To select a preset, click on the two sliders as depicted in the accompanying screenshot. This step is essential for customizing the controller settings in your project, ensuring they align with the imported presets for optimal functionality and user experience.

Open the preset window for the left controller

Selecting a preset for the left controller

Similarly, for the Right Controller GameObject, follow the same process and select the corresponding preset for the right controller. This ensures that both controllers are appropriately configured with their respective presets, maintaining consistency and functionality in your setup.

Choose an Interaction Manager

For the final step in setting up our controllers, we need to assign the Interaction Manager to both the left and right controllers. As shown in the forthcoming screenshot, select XR Interaction Manager as the Interaction Manager. This action links the controllers to the central system that manages interactions, ensuring they function correctly within the XR environment.

ℹ️ Interaction Manager: The XR Interaction Manager that this interactor will communicate with.

Selecting the XR Interaction Manager as the Interaction Manager

Similarly, for the Right Controller GameObject, follow the same process and select XR Interaction Manager as the Interaction Manager.

Testing the app

Run your Unity project via Build And Run as explained in the last article. You should now see the Unity project running in your headset as seen in the next screenshot. Move your Meta Quest 3 controllers to verify if everything is working as expected.

Testing the app if our controllers work

In our next article, we will delve into the aspect of detecting planes in MR environments. Plane detection is a cornerstone of MR, allowing virtual content to be realistically placed and interacted with in the physical world. We will explore how this technology works and the ways you can implement plane detection in your projects using Unity.

Unity MR Part 4: Enter MR

tststs — Thu, 11 Jan 2024 08:50:48 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 In this article, you will learn the setup process for URP (Universal Render Pipeline) and the camera configuration necessary to enable passthrough functionality. This guide will provide step-by-step instructions to ensure that passthrough operates effectively in your project, essential for creating immersive augmented reality experiences.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

To activate passthrough in our Android build, adjustments are required for the camera and URP asset. The camera's Background Type should be set to solid color, but to enable passthrough, modify the Background color's alpha value to 0. This can be done by clicking on the black bar to access the color picker, as demonstrated in the following two screenshots.

ℹ️ Background: the Camera clears its color buffer to this color before rendering.

Camera component in the Environment section

Color picker for the Background clear color

Close the color picker, access the Project Settings by navigating to Edit → Project Settings, then proceed to the Quality section. You will notice that the Balanced level is set as the standard quality for our Android build.

Project Settings Quality

The Balanced URP Asset features an HDR (High Dynamic Range) option that is unsupported and results in passthrough being unavailable. To rectify this, disable HDR in your URP-Balanced asset, located under Assets/Settings under Project. This adjustment is crucial for ensuring that passthrough functionality is enabled correctly in your project.

Disable HDR in URP-Balanced

Testing the app

That's all for now. We're ready to test our application. From now on we need to build the application and deploy it to the Meta Quest 3 since Passthrough is not available via hitting Play.

Ensure that all configurations are correctly set by navigating to File → Build Settings. Once you've verified these settings, simply click on Build And Run to proceed.

You should now see the Unity project running in your headset as seen in the next screenshot.

Passthrough is now successfully enabled in our Android build, as evidenced by the recent adjustments. This achievement marks a significant step in ensuring that the application functions with the desired augmented reality capabilities on Android devices.

Entering MR in our office

In our upcoming article, we will focus on configuring the left and right controllers for the Meta Quest 3. This step is crucial for creating a responsive and intuitive user experience in virtual reality. We'll guide you through the process of setting up these controllers in Unity, discussing how to map their buttons.

Unity MR Part 3: Camera Component

tststs — Wed, 10 Jan 2024 17:21:31 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 This article will guide you through understanding cameras in the context of MR and their implementation in Unity. It covers adding the camera to a scene and examines the XR Origin (AR) Prefab, providing essential insights for integrating cameras effectively in MR environments using Unity.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

In Unity, the camera is a crucial component for creating visual experiences in both traditional game development and MR. It acts as the viewer's eye, determining what is seen on the screen. In MR, the camera not only renders the virtual environment but is often integrated with real-world inputs, allowing for a seamless blend of physical and digital elements.

The camera in AR Foundation for Unity is crucial for AR applications, offering features like camera components, image capture, and accessing EXIF data from device cameras. The Unity OpenXR Meta plugin supplements this by focusing on Meta Quest devices.

On Meta Quest devices, AR Foundation's Camera subsystem controls Meta Passthrough. Enable the AR Camera Manager component to enable Passthrough, and disable it to disable Passthrough.

Adding a Camera to the Scene

Remove the default Main Camera from the hierarchy.
Create an empty GameObject MR
Move the Session GameObject into the newly created MR GameObject
In the hierarchy view right click the MR GameObject and select XR → XR Origin (AR). After adding the XR Origin your scene should look like as follows:

Scene after adding the XR Origin (XR Rig)

The XR Origin (XR Rig) Prefab is composed of a Camera Offset, Main Camera, and two GameObjects for the left and right controllers. We will delve into the XR Origin (XR Rig) Prefab in this article, and the subsequent article in this series will cover the setup of both controllers in more detail.

You'll observe that an XR Interaction Manager has been automatically generated in your scene.

ℹ️ The XR Interaction Manager in Unity is a central component that manages interactive elements in XR (Extended Reality) environments. It handles the interactions between the user and virtual objects, including input from controllers and headsets. This manager coordinates actions like grabbing, touching, or selecting objects, ensuring smooth and intuitive interaction within the virtual world, which is essential for creating immersive and responsive XR experiences.

We’ll get back to the XR Interaction Manager in the next article.

Exploring the XR Origin (AR) Prefab

The XR Origin (XR Rig) GameObject in the inspector

XR Origin (XR Rig)

The XR Origin represents the session-space origin (0, 0, 0) in an XR scene.

ℹ️ session-space origin refers to the reference point or coordinate system that is established when an XR session starts. This origin is used to define the positions and orientations of objects and users within the virtual environment. It's crucial for accurately placing and moving virtual elements in relation to the real world or the user's environment, ensuring a coherent and immersive experience in XR applications.

The XR Origin component is typically attached to the base object of the XR Origin, and stores the GameObject that will be manipulated via locomotion. It is also used for offsetting the camera.

ℹ️ Locomotion refers to the methods by which a user moves within a virtual environment. This can include physical movements, like walking or turning, which are tracked and translated into movement within the virtual space. Alternatively, it can involve virtual movement methods such as teleportation, sliding, or using a controller to navigate. Effective locomotion techniques are key to creating a comfortable and immersive XR experience.

XROrigin Script fields

Field	Description
Origin Base GameObject	The `Origin` GameObject is used to refer to the base of the XR Origin, by default it is this GameObject. This is the GameObject that will be manipulated via locomotion.
Camera Floor Offset Object	The GameObject to move to desired height off the floor (defaults to this object if none provided). This is used to transform the XR device from camera space to XR Origin space.
Camera	The Camera used to render the scene from the point of view of the XR device. Must be a child of the GameObject containing this XROrigin component.
Tracking Origin Mode	Sets which Tracking Origin Mode to use when initializing the input device. `NotSpecified`: Uses the default Tracking Origin Mode of the input device. `Device`: Input devices will be tracked relative to the first known location. `Floor`: Input devices will be tracked relative to a location on the floor
Camera Y Offset	Camera height to be used when in `Device` Tracking Origin Mode to define the height of the user from the floor. This is the amount that the camera is offset from the floor when moving the `CameraFloorOffsetObject`.

Main Camera

Let's examine the Main Camera component, focusing specifically on the additional scripts and systems essential for creating a MR experience. This article will not cover the basic aspects of Unity's Camera component. Instead, it will delve into the AR Camera Manager, AR Camera Background and Tracked Pose Driver, which are crucial for implementing an MR experience.

The Main Camera in the inspector

AR Camera Manager

This component controls Passthrough on Meta Quest devices. Enable this component to enable Passthrough, and disable it to disable Passthrough. Properties such as Auto Focus and Facing Direction have no effect on Meta Quest devices.

ℹ️ Passthrough refers to a technology that allows users to see the real world through their headset's cameras while still being in a virtual or augmented environment. This blending of real-world visuals with virtual elements helps create a mixed reality experience, enabling users to interact with both physical and digital objects, enhancing safety and immersion.

AR Camera Background

This component has no effect on Meta Quest devices, so we disable it as seen in the above screenshot (checkmark next to AR Camera Background).

Tracked Pose Driver (Input System)

The Tracked Pose Driver is a component used in XR development. It tracks the position and rotation of the headset or controller, and applies these transformations to a GameObject in Unity. This allows the GameObject to mirror the real-world movements of the device.

In our scenario, the current pose value of the tracked device is applied to the Transform of the Main Camera. This ensures that the camera's position and orientation in the virtual environment accurately reflect the real-world movements of the tracked device, enhancing the realism and interactivity of the experience.

Testing the app

Run your Unity project as explained in the last article Session Component.

You should now see the Unity project running in your headset as seen in the next screenshot.

Result after adding the XR Origin (XR Rig) to our scene

Since we're utilizing Meta Quest Link, passthrough functionality is not available, when running the app from within the Unity Editor. The red lines represent the left and right controllers, automatically integrated via the XR Origin Prefab.

Additionally, the black background is a consequence of the solid color setting, configured through the Environment section of our camera component, resulting in this specific visual effect.

In our next article, we're excited to introduce the concept of Passthrough technology, a pivotal feature for entering the realm of MR. Passthrough acts as a bridge between the virtual and real worlds, allowing users to see their physical environment through a digital lens.

Unity MR Part 2: Session Component

tststs — Wed, 10 Jan 2024 17:12:24 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 In this article, you'll discover the concept of a session in MR and how to apply it in Unity. We'll discuss differences in sessions in ARFoundation and OpenXR: Meta, set up our Session GameObject, and conduct tests on the Meta Quest 3. This exploration will provide a comprehensive understanding of session management in Unity for MR applications, especially for the Meta Quest 3 platform.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

A session refers to the period during which the app is actively tracking and integrating real-world and virtual environments, handling user interactions, and maintaining continuity in the mixed reality experience.

About Session in ARFoundation

In AR Foundation, the AR Session component is crucial for controlling the lifecycle of an AR experience on the target platform. It represents an instance of AR, and other features like plane detection can be independently enabled or disabled within this framework. The AR Session component is responsible for tracking features in the environment. When this component is disabled, the system stops tracking these features, but if re-enabled later, it attempts to recover and maintain the previously detected features. It's important to note that multiple AR Session components in the same scene can conflict with each other, so it's recommended to add at most one AR Session component to a scene.

The AR Session component also checks for device support. Since some platforms have built-in AR capabilities, while others may require on-demand installation of AR software or might not support AR at all, it's vital for applications to detect support for AR Foundation to provide alternative experiences when AR is not supported. This might involve checking a remote server for software availability.

The AR Session state can be monitored using ARSession.state, and you can subscribe to the ARSession.stateChanged event to receive callbacks when the state changes. This state includes various statuses like None, Unsupported, CheckingAvailability, NeedsInstall, Installing, Ready, SessionInitializing, and SessionTracking, each indicating different stages of AR session readiness and functionality.

About Session in OpenXR: Meta

In the context of OpenXR: Meta, the Session is a supplement to the AR Foundation Session manual, focusing on the unique platform-specific behavior of the Meta Quest. This includes a feature called Scene Capture, where, unlike other AR platforms, OpenXR: Meta does not dynamically detect trackables at runtime. Instead, it queries the device's Room Setup data and returns information stored in the Scene Model. During scene capture, users can label surfaces and objects in their environment, like walls and furniture. Once the scene capture is complete, the Scene Model is saved to the device and persists across applications and sessions.

The lifecycle of scene capture in OpenXR: Meta involves several phases: initiating scene capture, suspension of the app during capture, completion of scene capture by the user, and resumption of the app post-capture.

These features highlight the comprehensive nature of AR Session components in managing and maintaining AR experiences in Unity's AR Foundation and OpenXR: Meta, showcasing the advancements and unique aspects of AR technology in different platforms.

Adding the Session component to the scene

Create an Empty GameObject

First, open your Unity project.
In the Unity editor, go to the Hierarchy panel.
Right-click in the Hierarchy panel and select "Create Empty" to create a new GameObject and name it Session.
This GameObject will act as a container for your AR components.

Add the AR Session Script

With the new GameObject selected, go to the Inspector panel.
Click on "Add Component".
Search for “AR Session” and add it to your GameObject.
The AR Session component is essential for managing the AR experience, including tracking and session management.

Initially, you'll be using the default settings of the AR Session component. These defaults are usually sufficient for basic AR experiences.

The hierarchy after adding the Session Component

ℹ️ You might observe that we are using a custom layout in the Unity Editor, which means that the screenshots provided may not exactly match the appearance of your Editor. Keep in mind that while the visual layout may vary, the fundamental features and functions of Unity remain consistent across different customizations.

Check for device support

Select the Session GameObject in your hierarchy. Create a new script DeviceSupport via Add Component.

Let's verify if the connected device is AR-capable. As this course focuses on the Meta Quest 3, AR support is typically guaranteed. By setting attemptUpdate = true, the device will attempt to install AR software if feasible. However, the availability of this feature is dependent on the platform.

DeviceSupport looks as follows:



using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.ARFoundation;

namespace Taikonauten.Unity.ArticleSeries
{
    public class DeviceSupport : MonoBehaviour
    {
        \[SerializeField\] ARSession session;

        IEnumerator Start()
        {
            session.attemptUpdate = true;

            if (session == null)
            {
                Debug.LogError("DeviceSupport -> Start(): session is null");

                yield break;
            }

            if ((ARSession.state == ARSessionState.None) ||
                (ARSession.state == ARSessionState.CheckingAvailability))
            {
                Debug.Log("DeviceSupport -> Start(): check availability");

                yield return ARSession.CheckAvailability();
            }

            if (ARSession.state == ARSessionState.Unsupported)
            {
                // Start some fallback experience for unsupported devices
                Debug.Log("DeviceSupport -> Start(): unsupported device");
            }
            else
            {
                // Start the AR session
                Debug.Log("DeviceSupport -> Start(): start AR session");

                session.enabled = true;
            }
        }
    }
}

To prevent any potential naming conflicts, we will adopt the namespace Taikonauten.Unity.ArticleSeries for all our scripts moving forward.

ℹ️ This practice will ensure a clear and organized code structure, helping to distinguish our script elements effectively within the project. Using a specific namespace like Taikonauten.Unity.ArticleSeries is a best practice in software development for maintaining code clarity and avoiding clashes with other code entities.

In the Unity Editor, choose Session (AR Session) as the value for the Session field for your newly created script. Your Session GameObject should now resemble the appearance shown in the following screenshot.

Inspector window of the Session GameObject

To maintain an organized project structure, create a Scripts folder within the Assets directory in the Project window. Then, move the DeviceSupport Script into this new folder. This approach helps in keeping your project tidy and ensures that all scripts are conveniently located in a dedicated folder.

Moving the DeviceSupport script into the Scripts folder

Testing the app

To run a Unity project on your Meta Quest 3 using the Link Cable, follow these steps:

1. Connect the Meta Quest 3 to Your PC

Use the Link Cable to connect your Meta Quest 3 headset to your computer.

2. Enable Oculus Link on Your Headset

Put on your Meta Quest 3 headset and enable Oculus Link when prompted. This will connect your headset to the Unity Editor running on your PC.

3. Press the Play Button in Unity

Go back to the Unity Editor on your PC and press the 'Play' button. This will start running your Unity project directly in your Meta Quest 3 headset.

ℹ️ If you're using a Mac, the standard approach might not work. Instead, use File → Build and Run while the Meta Quest 3 is connected to your computer. We cover this process in the article Enter MR, specifically in the Testing the App section

4. View the Output in Your Headset

You should now see the Unity project running in your headset as seen in the next screenshot.

If you encounter any issues, ensure that your Meta Quest 3 is set up correctly with your PC, the Oculus software is installed and up to date, and your Unity project is configured for Oculus VR development.

⚠️ No active UnityEngine.XR.ARSubsystems.XRSessionSubsystem is available. This feature is either not supported on the current platform, or you may need to enable a provider in Project Settings > XR Plug-in Management. UnityEngine.XR.ARFoundation.ARSession:OnEnable () (at ./Library/PackageCache/com.unity.xr.arfoundation@6.0.0-pre.4/Runtime/ARFoundation/ARSession.cs:331)

The message you're seeing in the Unity Editor Console is a warning that can be safely ignored. It's indicating that the XRSessionSubsystem is only available in built applications for their specific build targets, which is Android in this case. This means that the XRSessionSubsystem, a component used for handling XR (Extended Reality) sessions, won't be available or functional within the Unity Editor itself, but it will work as expected once the application is built and deployed to the Meta Quest 3.

In our upcoming article, we will delve into the integral aspects of the Camera Component in Unity, especially in the context of AR and VR development. The Camera Component is a fundamental part of any immersive experience, acting as the eyes of the user in the virtual world. We'll explore its capabilities, how it can be configured and manipulated to suit our needs.

Unity MR Part 1: Setting up Unity

tststs — Wed, 10 Jan 2024 10:11:57 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 This article provides a comprehensive guide on setting up Unity for MR development with the Meta Quest 3. It covers the installation of Unity, creation of a new 3D project, installation of necessary packages, and configuration of the XR-Plug-in, equipping you with the foundational steps needed to begin developing MR applications on the Meta Quest 3 platform using Unity.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

About Unity

Unity offers a robust and flexible development platform for MR applications, providing extensive support for immersive experiences with its advanced rendering capabilities, comprehensive asset library, and wide-ranging compatibility with various MR hardware.

Install Unity Editor via Unity Hub

Install Unity Hub, create an account and get Unity 2023.2.3f1 with Android Build Support.

ℹ️ If you're wondering why Android build support is necessary, it's because the Meta Quest 3 operates on a modified version of Android 12.

We are using the latest 2023.2.3f1 build to have access to ARFoundation 6.0. You can read more about the topic here: Access AR Foundation 6.0 in Unity 2023.2 | AR Foundation | 6.0.0-pre.5.

Install the Unity Editor and choosing Android build support

Installs tab after successful installation of the Unity Editor

Create a new Unity 3D (URP) project

This series of articles will utilize the Universal Render Pipeline (URP) instead of the default render pipeline.

ℹ️ If you're unfamiliar with URP, or the Universal Render Pipeline in Unity, you can gain a thorough overview by following this link: Universal Render Pipeline (URP) | Unity.

Create a new project 3D (URP) within the Projects tab. Make sure you select the Editor Version 2023.2.3f1.

Creating an new project using the 3D (URP) template

Install required packages

Open your projects manifest.json file and append the following required packages.

ℹ️ To find the manifest.json file in Unity, you need to navigate to your Unity project's directory on your computer. Inside the project folder, look for the Packages folder. The manifest.json file will be located within this folder. This file is used to manage project dependencies and Unity package versions, making it a critical part of Unity project configuration.



"com.unity.xr.arfoundation": "6.0.0-pre.4",



    "com.unity.xr.interaction.toolkit": "2.5.2",

    "com.unity.xr.meta-openxr": "1.0.1",

    "com.unity.xr.openxr": "1.9.1"

Save the file and return to the Unity Editor. If you get the following warning choose ‘yes’.

Configure the XR-Plug-in

Find the XR Plug-in Management settings via Edit → Project Settings. Make sure to edit both Windows, Mac, Linux (Desktop) settings and Android settings.

First configure Desktop by enabling OpenXR as seen in the next screenshot.

Enabling OpenXR in XR Plug-in Management for Desktop

Then, switch to Android and also enable OpenXR and the Meta Quest feature group as seen n the next screenshot.

Enabling OpenXR in XR Plug-in Management for Android

Add the Meta Quest Touch Pro Controller Profile to the list of Enabled Interaction Profiles for Desktop and Android build. For details on the available controls, refer to the Meta Quest Pro Touch Controller Profile | OpenXR Plugin | 1.9.1 documentation.

For desktop we only need to add the controller profile as seen in the next screenshot.

Adding the Meta Quest Touch Pro Controller Profile for Desktop

Switch to the Android tab, add the controller profile and enable all features in the Meta Quest feature group. The features are most likely already enabled.

Adding the Meta Quest Touch Pro Controller Profile for Android

At present, you might come across one or more issues during the Project Validation process. To address these, use the Fix All option. However, be aware that Fix All won't resolve all warnings in your project, as depicted in the forthcoming screenshots. This situation is acceptable, as we will be addressing and resolving these warnings manually throughout the course of our article series.

Issues Desktop tab

Issues Android tab

Remaining issues after Fix All

Using the Meta Quest Link Cable

Meta Quest Link provides a simplified method for skipping the Android build process in Unity. However, it lacks critical features such as Plane Detection and Passthrough, rendering it ineffective for real-world mixed reality application testing. Presently, there is no indication as to whether or when these capabilities might be implemented.

Alternatively, you can switch to Android within the Build Settings in Unity. Follow these steps:

Access the Build Settings by navigating to File → Build Settings.
Make sure to add the SampleScene via Add Open Scene to Scenes in Build. Normally the SampleScene is already included so you can properly skip this step.
Select Android as a Platform and press the Switch platform button at the bottom right.

With this setup, we now have two methods to launch apps on the Meta Quest 3: without passthrough directly from the Unity editor, and with passthrough using the Build and Run feature. The build and run process deploys the app as an Android app to your Meta Quest 3, but it takes significantly longer than simply pressing the play button in Unity. We’ll start by using the the first method and later switch to Build and Run.

In our next article, we will introduce a key component in MR development: the Session Component. This feature plays a crucial role in managing the MR session, effectively serving as the backbone of any mixed reality application.

DEV Community: tststs

Unity MR Part 12: Improve Scene

Light Estimation

Instantiate the door facing the user

Update AR Default Plane

Optimize Lightning

Optimize URP

The behind the door

Adding a complex scene behind the door

Testing the app

Unity MR Part 11: Animation

Testing the app

Next article

Unity MR Part 10: Voice SDK

Android Setup

Adding some UI

Updating our MRArticleSeriesController Script

Testing the app

Next article

Unity MR Part 9: Instantiate Prefab

Next article

Unity MR Part 8: Anchors

AR Anchor Manager component

Adding AR Anchor Manager component to the Scene

Next article

Unity MR Part 7: Raycasts

Creating a custom Reticle

Updating our AR Default Plane

Improve the appearance of the XR Interactor Line Visual

Improve appearance of the Reticle

Reacting to a OnButtonPressed Event and check if the Ray hits a Plane

Next article

Unity MR Part 6: Planes

Adding the AR Plane Manager

Creating a Prefab for our detected planes

Setup the default plane Prefab

Testing the app

Next article

Unity MR Part 5: Left and Right Controllers

Import XR Interaction Toolkit Starter Assets

Enable the new Input System

Adding the AR Input Manager

The Input Action Manager

Adding XR Input Modality Manager

Choose a controller preset

Choose an Interaction Manager

Testing the app

Next article

Unity MR Part 4: Enter MR

Testing the app

Next article

Unity MR Part 3: Camera Component

Adding a Camera to the Scene

Exploring the XR Origin (AR) Prefab

XR Origin (XR Rig)

XROrigin Script fields

Main Camera

AR Camera Manager

AR Camera Background

Tracked Pose Driver (Input System)

Testing the app

Next article

Unity MR Part 2: Session Component

About Session in ARFoundation

About Session in OpenXR: Meta

Adding the Session component to the scene

Create an Empty GameObject

Add the AR Session Script

Check for device support

Testing the app

Next article

Unity MR Part 1: Setting up Unity

About Unity

Install Unity Editor via Unity Hub

Create a new Unity 3D (URP) project

Install required packages

Configure the XR-Plug-in

Using the Meta Quest Link Cable

Next article