DEV Community: Taikonauten

Part 9: Asset Handling & Animation (WebXR with Babylon.js)

Bryan — Thu, 11 Jan 2024 08:51:41 +0000

👀 Stumbled here on accident? Start with the first part!

Welcome back to the 9th and final part of this series about asset handling and animation. In the last article we added a model in form a door to the scene. In the following sections we’re going to implement opening and closing the door on the press of a button.

ℹ️ Remember - you can always run the code associated with this article and follow along using

npm start --part=9

animating the door

Interacting with the door

handleControllerSelection() {
  ...
    this._xr.input.onControllerAddedObservable.add((motionControllerAdded) => {
        motionControllerAdded.onMotionControllerInitObservable.add((motionControllerInit) => {

            ...
            const buttonComponent = motionControllerInit.getComponent(motionControllerComponentIds[3]);

            if (buttonComponent) {
                buttonComponent.onButtonStateChangedObservable.add((component) => {
                    if (component.pressed) {
                        (this._doorIsOpen) ? this.closeDoor() : this.openDoor();
                    }
                });
            }

            triggerComponent.onButtonStateChangedObservable.add((component) => {
            ...
            });
        });
    });
}

First we’re going to add a new interaction to the handleControllerSelectionfunction by observing button presses to the controllers A-Button. The motionControllerComponentIds[3] is the component id related to this specific button.

📚 More about component ids can be found here.

If the button is pressed the door is either opened by the this.openDoor() or closed this.closeDoor() function, depending on the state of this._doorIsOpen.

Opening the door

openDoor() {
  if (this._door !== null) {
      this.animateDoor(30);
      this._doorIsOpen = true;
  }
}

Closing the door

closeDoor() {
  if (this._door !== null) {
      this.animateDoor(30);
      this._doorIsOpen = false;
  }
}

By adjusting the duration we can create an effect where the door opens normally but closes like it is shut. The lower the duration, the quicker the animation is.

Animating the door interaction

animateDoor(duration: number) {

    const animationName = this._doorIsOpen ? "doorOpenQuat" : "doorCloseQuat";
    const doorAnimation = new Animation(animationName, "rotationQuaternion", 30, Animation.ANIMATIONTYPE_QUATERNION, Animation.ANIMATIONLOOPMODE_CONSTANT);

    const startRotation = this._door!.rotationQuaternion!.clone();

    const axis = new Vector3(0, 1, 0); 
    const angle = this._doorIsOpen ? -Math.PI / 1.5 : Math.PI / 1.5; 
    const endRotation = Quaternion.RotationAxis(axis, angle).multiply(startRotation);

    const keyFrames : {frame: number, value: Quaternion}[] = [];

    keyFrames.push({
        frame: 0,
        value: startRotation
    });

    keyFrames.push({
        frame: duration,
        value: endRotation
    });

    doorAnimation.setKeys(keyFrames);

    if (!this._door!.rotationQuaternion) {
        this._door!.rotationQuaternion = new Quaternion();
    }

    this._door!.animations = [doorAnimation];

    this._scene.beginAnimation(this._door, 0, duration, false);
}

To enhance the immersion we implement a smooth animation for the opening and closing the door.

Animation Definition:
- animationName: Determines the name of the animation based on the current state of the door (open or closed).
- doorAnimation: Creates a new Animation object for quaternion rotation. This type of rotation is smooth and avoids issues like gimbal lock. The animation runs at 30 frames per second and doesn't loop.
Initial Rotation:
- Retrieves the door's current rotation in quaternion form (a complex number system that represents rotations).
End Rotation Calculation:
- Defines the rotation axis (y-axis) and the angle (90 degrees for opening, -90 degrees for closing).
- Calculates the end rotation quaternion by multiplying the start rotation by the rotation created from the specified axis and angle.
Keyframe Definition:
- Initializes an array for keyframes, which are crucial points in the animation.
- Adds the start (frame 0) and end (frame at duration) rotations to the keyframes. This defines the animation path from the current rotation to the target rotation.
Animation Assignment:
- Ensures the door is set to use quaternion rotation.
- Assigns the created animation to the door mesh, preparing it to be animated.
Begin Animation:
- Starts the animation using the Babylon.js beginAnimation method on the door mesh from frame 0 to the specified duration.
- The animation is not set to loop.

Conclusion

In this final part of our series, we delved into the nuances of animating a door in a 3D environment, highlighting the seamless integration of user interactions and realistic motion. Utilizing quaternion rotations and Babylon.js, we demonstrated how to create a smooth and responsive door animation that enhances user experience.

Part 8: Models & Assets (WebXR with Babylon.js)

Bryan — Thu, 11 Jan 2024 08:51:36 +0000

👀 Stumbled here on accident? Start with the first part!

Welcome back to the 8th instalment in this series. This part is about importing an asset in form of a 3D model of a door that gets attached to an anchor.

ℹ️ Remember - you can always run the code associated with this article and follow along using

npm start --part=8

loading and attaching a model to an anchor

Prerequisite

To be able to load model files in form of assets into a scene we first need to import the babylonjs model loader for glb/glTF files.

import '@babylonjs/loaders/glTF';

The provided 3D model file consists of 3 layers. A Handle, a Door and a Door_frame.

The loader itself add a parent node called __root__ to the whole model, therefore the 3 layers are stored as sub meshes to the __root__.

enum DoorMeshes {
    Handle = 'Handle',
    Door = 'Door',
    DoorFrame = 'Door_frame',
    Container = '__root__',
}

To make things easier when loading parts on their own, we create an enum for the layer names.

Loading the model

addDoor(): void {
    SceneLoader.Append("/models/", "door.glb", this._scene, ((scene: Scene) => {

        this._handle = scene.getMeshByName(DoorMeshes.Handle);
        this._door = scene.getMeshByName(DoorMeshes.Door);
        this._doorFrame = scene.getMeshByName(DoorMeshes.DoorFrame);
        this._doorContainer = scene.getMeshByName(DoorMeshes.Container);

        const meshes = this._doorContainer!.getChildMeshes();

        meshes.forEach((mesh) => {
            this._shadowGenerator!.addShadowCaster(mesh);
            mesh.receiveShadows = true;
        });

        this._handle!.isVisible = false;
        this._door!.isVisible = false;
        this._doorFrame!.isVisible = false;
    }));
}

Next we’re going to add the addDoor function to load the 3D model.

SceneLoader.Append loads a model from a specific local file path into the the this._scene.
Once the model is loaded a callback function is called. In the callback function each mesh is assigned to a variable making them easily accessible later on.

❗️The most important layer is the node layer which is assigned to this._doorContainer since this is going to be the one that is being attached to an anchor later on.

Then we loop through all the child meshes to:

add the mesh as a shadow caster to the this._shadowGenerator, allowing it to cast shadows to the scene
setting receiveShadows to true to enable the mesh to receive shadows from other objects in the scene

Finally we set each mesh to be invisible initially to be later explicitly make them visible again when attached to an anchor.

Adding the door to the scene

async createScene(): Promise<Scene> {

  ...
  this.addDoor();

  if (this._xrAnchors && this._xrAnchors.isCompatible()) {
      this.observeAnchors();
      this.handleControllerSelection();
  }

  return this._scene;
}

To be able for the scene to load the model, we add this.addDoor() to the createScene function.

Attaching the model to an anchor

addAnchorAtPosition(raycastHit: PickingInfo) {
    this._xrAnchors!.addAnchorAtPositionAndRotationAsync(raycastHit.pickedPoint!).then((anchor) => {
        const boxTransformNode = new TransformNode('boxTransformNode');

        this._box!.parent = boxTransformNode;
        this._box!.position = new Vector3(0, 1, .5);
        this._box!.isVisible = true;

        this._door!.isVisible = true;
        this._doorFrame!.isVisible = true;
        this._handle!.isVisible = true;

        this._doorContainer!.position = raycastHit.pickedPoint!;
        boxTransformNode.parent = this._doorContainer!;

        anchor.attachedNode = this._doorContainer!;
        anchor.attachedNode.position = raycastHit.pickedPoint!;
    });
}

Since we already know how to attach a mesh to an anchor, we adjust our previous implementation for the this._box to this._doorContainer. Instead of attaching the this._box directly to the anchor, we assign the this._doorContainer as the parent to the box and assign the doorContainer to anchor.attachedNode. Important in this step is to make the model layers visible by setting isVisible to true.

Conclusion

In this 8th installment, we covered the process of importing and integrating a 3D door model with Babylon.js. We detailed loading the model, managing its components, and attaching it to an anchor. The steps taken demonstrate a straightforward approach to incorporating interactive elements in virtual environments, suggesting practical applications in web technology.

In the ninth and final part we’re going to implement some animations for the door.

Unity MR Part 12: Improve Scene

tststs — Thu, 11 Jan 2024 08:51:30 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 The aim of this article is to enhance the visual and interactive elements of our scene. We will focus on several key improvements:

Graphics Enhancement: We'll enhance the graphics by configuring the Universal Render Pipeline (URP) assets.
Remove Debug Plane Material: The debug plane material, used for initial testing, will be removed for a cleaner look.
Door Shadow Effects: We'll set up the scene so that the door casts shadows on planes that will now be invisible, adding depth and realism.
Create a 'Behind the Door' Scene: We will develop a scene that is only revealed when the door is open, adding an element of surprise and exploration.

This article incorporates some advanced techniques that can significantly elevate your MR project. If you encounter any challenges or need further clarification, feel free to send us a message. Additionally, you can refer to the final project in our GitHub repository dedicated to this article series.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

Light Estimation

ℹ️ Light estimation for MR devices like the Meta Quest 3 involves assessing the real-world lighting conditions and applying that information to the virtual environment to create a more immersive and realistic experience.

Regrettably, as indicated by Unity in their forums, this feature is not currently supported.

Instantiate the door facing the user

Currently, the door Prefab is instantiated with a consistent rotation. Let's modify the MRArticleSeriesController Script so that the door faces the user upon instantiation. To do this, locate the OnButtonPressedRightAsync method in the MRArticleSeriesController Script and update it as follows.

private async void OnButtonPressedRightAsync(InputAction.CallbackContext context)
{
    Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync()");

    if (doorInstance != null) {
        Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync(): Door already instantiated");

        return;
    }

    if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
    {
        Pose pose = new(hit.point, Quaternion.identity);
        Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

        result.TryGetResult(out ARAnchor anchor);

        if (anchor != null) {
            // Instantiate the door Prefab
            doorInstance = Instantiate(door, hit.point, Quaternion.identity);

            // Unity recommends parenting your content to the anchor.
            doorInstance.transform.parent = anchor.transform;

            // Make the door face the user after instantiating
            doorInstance.transform.LookAt(new Vector3(
                Camera.main.transform.position.x,
                doorInstance.transform.position.y,
                Camera.main.transform.position.z
            ));
        }
    }
}

As you can see, we have added a call to doorInstance.transform.LookAt in the script. After making this addition, save the file and return to the Unity Editor to apply these changes.

ℹ️ For more information on the LookAt function and how it works, you can refer to the Unity documentation. This resource provides detailed insights into how LookAt can be used to orient objects in your scene: Unity - Scripting API: Transform.LookAt.

Update AR Default Plane

In this step, we will modify the material of the AR Default Plane to make it transparent while still allowing it to receive shadows. This adjustment enables us to see the shadows cast by the door on our floor. Implementing this change enhances the realism of our MR experience, as it allows virtual objects like the door to interact more naturally with the environment.

Go to Window → Package Manager and install the package https://github.com/Cyanilux/URP_ShaderGraphCustomLighting.git with Install package from git URL:

Install a package via git

Once you have completed the installation, close the Package Manager. Now, edit the AR Default Plane by double-clicking it in the Project window.

Replacing the default material with the ShadowReceiver material

Replace the default Material with ShadowReceiver as seen in the above screenshot. That’s all for the AR Default Plane.

If you can’t find ShadowReceiver via search you can also drag and drop the material from Packages/com.cyanilux.shadergraph-customlighting/Examples/ShadowReceiver.mat.

Optimize Lightning

In this step, we will focus on optimizing the Directional Light in our scene. You have the option to adjust the values as shown in the upcoming screenshot, or alternatively, you can modify the settings according to your own preferences and the specific requirements

Optimizing the Directional Light

Optimize URP

We are currently set to the Balanced quality level for the Android build. Let's switch to using the High Fidelity quality setting by default, as illustrated in the upcoming screenshot. This change will enhance the visual quality of our application, offering a more detailed and immersive experience on Android devices.

Set High Fidelity as the default quality on Android

Next, we must disable HDR in our URP asset. HDR is not supported in this context and will cause the application to crash if enabled. You can locate the URP asset at Assets/Settings/URP-HighFidelity.asset. The HDR option is available as the first option under the Quality section.

Disabling HDR in our URP asset

The High Fidelity settings utilize the Renderer Features Screen Space Ambient Occlusion, which significantly impacts performance. To optimize, you can either disable this feature or remove it entirely —both approaches are effective. You can find this setting in the asset located at Assets/Settings/URP-HighFidelity-Renderer.asset.

Disabling Screen Space Ambient Occlusion from the High Fidelity-Renderer

The behind the door

Let's create a straightforward effect that simulates a scene visible only when the door is open and the user looks through it. This approach will add an intriguing layer of interactivity to our project, enhancing the user experience by revealing a hidden scene.

Edit our door prefab by double-clicking Assets/Prefabs/Door in the Project window.

Disable the Door mesh through the inspector so we can look through the door.
Add a plane into the root of the prefab via Create → 3D → Plane.
Use the following Transform values as follows:

	X	Y	Z
Position	0	1.025	0
Rotation	90	0	0
Scale	0.1	1	0.205

The prefab should now resemble the appearance displayed in the upcoming screenshot.

Creating a Plane inside our door prefab

As you will notice, this method effectively creates the illusion where the "inside" of the door is only visible from the front. To observe this effect, take a look at the door from the back in the Scene view, as illustrated in the upcoming screenshot.

The “invisible” plane as seen from the back of the door

Now create a new Render Texture in Assets/Materials and name it DoorPlaneRenderTexture. Edit the Render Texture as follows:

DoorPlaneRenderTexture (Render Texture)

As you can observe, we have set the Size to 1024x2048, which correlates with our plane's approximate dimensions of a 1:2 ratio. This size selection ensures that the texture or material applied to the plane is properly scaled and aligned, reflecting the plane's physical proportions in the virtual environment.

Now create a new Material in Assets/Materials and name it DoorPlaneRenderTexture. Edit the Material as follows:

DoorPlaneRenderTexture (Material)

Now, choose the DoorPlaneRenderTexture as the Base Map as seen in the next screenshot.

Choosing the DoorPlaneRenderTexture as the Base Map

For the final step, assign the DoorPlaneRenderTexture material to the Mesh Renderer of the Plane.

Assign the DoorPlaneRenderTexture material to the Planes Mesh Renderer

Now, let's create a simple scene that will be rendered onto the plane. Before we start, make sure to return to the SampleScene and exit the prefab editor.

In your hierarchy, create an empty GameObject. You can do this by selecting Create Empty. Name this GameObject DoorPlaneScene.
Ensure the position of DoorPlaneScene is set to X: 0, Y: 0, Z: 0, unless it's already positioned there.
Create another empty GameObject as a child of DoorPlaneScene and name it CameraOffset . Set the Y position of CameraOffset to 1.1176 in its Transform properties.
To the CameraOffset GameObject, add a Camera component. This camera will capture the scene for rendering onto the plane.
Add another empty GameObject under DoorPlaneScene and name it Scene. This GameObject will hold the elements of your rendered scene.
Inside Scene, create a Cube (3D Object → Cube). Position this Cube with its Z coordinate set to 5.

These steps set up a basic scene structure, with a camera positioned to capture the scene, and a simple Cube as a visual element. The scene will be rendered from the perspective of the added camera.

The result looks as follows:

Hierarchy after adding the scene which will be rendered onto our plane

We aim for our new Camera to render only the contents of the Scene GameObject. To achieve this, navigate to Layers → Edit Layers..., as illustrated in the upcoming screenshot. Here, add a new user layer and name it DoorPlaneScene. This step is crucial for ensuring that the camera specifically captures the elements within the Scene, isolating its view to this particular part of your project for targeted rendering.

Adding a new user layer DoorPlaneScene

Now, select the DoorPlaneScene GameObject in the hierarchy and change its layer to DoorPlaneScene:

Selecting the DoorPlaneScene layer for our DoorPlaneScene GameObject

Confirm by also changing the layer for all children:

Changing the layer for all children

Lets configure the new Camera. Select our Main Camera in the hierarchy and copy the component values as seen in the next screenshot.

Copy the component values from Main Camera

Now select the new Camera again and paste the component values:

Paste component values to the new Camera

Now, let's configure some additional settings for the new Camera:

Set the tag of the Camera to Untagged.
Under the Rendering tab, change the Culling Mask to include only the DoorPlaneScene and Ignore Raycast. This ensures the camera renders only the objects in the DoorPlaneScene layer.
Assign DoorPlaneRenderTexture as the Output Texture. This means the camera's view will be rendered to this texture.
Set the Far value under Clipping Planes to 250. This determines how far the camera can see.
Choose Skybox as the Background Type under the Environment tab.

After making these changes, your camera should be set up as shown in the following screenshot.

The configured new Camera

Finally, create a prefab from the DoorPlaneScene and then remove it from the hierarchy, following the same process we used for the Reticle in the Raycasts article.

Our plan is to incorporate the DoorPlaneScene into our Door prefab. This way, DoorPlaneScene will only be rendered when necessary. To do this, edit the Door prefab and drag and drop the DoorPlaneScene into the root of the prefab. This action is depicted in the upcoming screenshot and ensures that the scene is efficiently integrated with the Door prefab.

Adding the DoorPlaneScene to our Door prefab

Save the prefab and return to the SampleScene.

Select the Main Camera again and set the Culling Mask from Everthing to: Default, TransparentFX, Ignore Raycast Water and UI.

Selecting the Culling Mask for the Main Camera

Drag and drop the prefab Door into your scene to investigate the changes we made. The plane now should look as follows:

Plain rendering the scene we just created

As you can observe, we now have our new scene projected onto the plane inside the door.

Next, re-enable the Door mesh, which we had disabled in a previous step. However, there's still an issue to address: the scene is static and doesn't adapt to the user's movements. Let's resolve this.

Create a new script and name it MimicCamera, then attach it to our newly added Camera. The script will be structured as follows:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MimicCamera : MonoBehaviour
    {
        public float offsetDistance = 30.0f;
        [SerializeField] private GameObject sceneCamera;
        private Camera mainCamera;
        public Transform DoorTransform { get; set; } = null;

        void Awake() {
            mainCamera = Camera.main;
        }

        void Update()
        {
            if (DoorTransform == null) {
                return;
            }

            // Calculate the direction and position offset from the door to the main camera
            Vector3 directionToCamera = mainCamera.transform.position - DoorTransform.position;
            Vector3 sceneCameraPosition = DoorTransform.position + directionToCamera.normalized * offsetDistance;

            sceneCameraPosition.y = sceneCameraPosition.y > 2f ? 2f : sceneCameraPosition.y;

            // Update the position and rotation of the scene camera
            sceneCamera.transform.position = sceneCameraPosition;
            sceneCamera.transform.rotation = Quaternion.LookRotation(DoorTransform.position - sceneCameraPosition);
        }
    }
}

Select the required values for the MimicCamera script as seen in the following screenshot:

Select the required values for the MimicCamera script

We still need to assign the DoorTransform when the Door is instantiated. To do this, edit the MRArticleSeriesController script and add a private field for the MimicCamera script. This modification is shown on line 1 in the following code snippet.

Afterward, assign the DoorTransform following our LookAt method call. If you need guidance, you can also refer to the code in the 12_ImproveScene project for a clearer understanding 12_ImproveScene.

...

// Make the door face the user after instantiating
doorInstance.transform.LookAt(new Vector3(
    Camera.main.transform.position.x,
    doorInstance.transform.position.y,
    Camera.main.transform.position.z
));

doorInstance.GetComponentInChildren<MimicCamera>().DoorTransform = doorInstance.transform;

...

Adding a complex scene behind the door

For the final step, we'll upgrade from the simple cube to a more complex scene. In our example, we utilized the Free Low Poly Nature Forest asset, which you can find here: Free Low Poly Nature Forest. Add the asset to your project.

Find the 'Demo_01 scene from the package within the Assets/Pure Poly/Free Low Poly Nature Pack/Scenes directory. Then, drag and drop it into your hierarchy, as demonstrated in the upcoming screenshot.

Drop the Demo_01 scene into our hierarchy

If the Demo_01 scene looks as follows, we need to replace the shader that is used for the material.

Demo_01 with an invalid material

Open the material located here: Assets/Pure Poly/Free Low Poly Nature Pack/Materials/PP_Standard_Material. Edit the material as follows:

Fixing the invalid material

Set the shader to Universal Render Pipeline/Lit.
As the Base Map choose the file PP_Color_Palette.
Set Smoothness to 0 for the Metallic Map.

Copy the Free_Forest GameObject, then edit our DoorPlaneScene prefab. Within the prefab, replace the cube with the Free_Forest GameObject, as illustrated in the next screenshot:

Replacing the cube with the Free_Forest GameObject in our DoorPlaneScene prefab

Set the Transform of the Free_Forest GameObject as follows:

	X	Y	Z
Position	0	-3.7	0
Rotation	0	-135	0
Scale	1	1	1

Set the Transform of the Free_Forest GameObject

Also ensure, that the layer of Free_Forest is set to DoorPlaneScene:

Setting the Free_Forest layer to DoorPlaneScene

Save the prefab and return to your scene. Now remove the Demo_01 scene from your hierarchy:

Remove the Demo_01 scene from your hierarchy

This enhancement significantly elevates the visual appeal and immersion of the scene, demonstrating the versatility and potential of using varied assets in your MR environment.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository. 12_ImproveScene.

Testing the app

We are now ready to test the final version of our app. Here's what you need to do:

Choose Build and Run in Unity.
Once the app is running, press the trigger on the left controller.
You should see a label displaying the message "...Listening...".
Say the phrase “open the door”.
After a short delay, the animation of the door opening should begin.
Feel free to move around and explore what lies behind the door.

This test will allow you to experience the full functionality of the app, from voice command recognition to the dynamic opening of the door and the revealing of the scene behind it. Enjoy the immersive experience of exploring the new environment you've created!

👏

Video of the app where the door opens when the phrase “open the door“ is recognized

Part 7: Anchors (WebXR with Babylon.js)

Bryan — Thu, 11 Jan 2024 08:51:29 +0000

👀 Stumbled here on accident? Start with the first part!

Welcome back to the 7th article in this WebXR/Babylon.js series. This part is about Anchors in WebXR.

ℹ️ Remember - you can always run the code associated with this article and follow along using

npm start --part=7

attaching a mesh to an anchor

What are Anchors?

Anchors are used to create a stable reference point in the physical space, which can be tracked by the device's sensors and cameras. When an anchor is set, the virtual object tied to it appears to be part of the real world, staying in the same place as if it were a physical object.

✅ Anchoring is crucial for maintaining consistency in the virtual environment as it relates to the real world.

Adding anchors to the scene

To add an Anchor to the scene, we go back into our handleControllerSelection, take the raycastHit value and call addAnchorAtPositionAndRotationAsync to add an anchor at the ray cast hit position.

Interacting with the controller

handleControllerSelection() {
  ...
    if (raycastHit.pickedMesh === this._box) {
        const mat = this._box!.material as StandardMaterial;
        mat.diffuseColor = Color3.Random();
        this._box!.material = mat;
    } else {

        this.addAnchorAtPosition(raycastHit);
    }
    this._box!.isVisible = true;
  ...
}

We first check if the mesh we picked is the box and if so, we just give it a randomly different color.

If we don’t pick the box but any other position in the 3D space we will call a new function addAnchorAtPosition with the raycastHit as a parameter.

Adding the Anchor

addAnchorAtPosition(raycastHit: PickingInfo) {
    this._xrAnchors!.addAnchorAtPositionAndRotationAsync(raycastHit.pickedPoint!).then((anchor) => {

        const boxTransformNode = new TransformNode('boxTransformNode');
        boxTransformNode.position = raycastHit.pickedPoint!;

        this._box!.parent = boxTransformNode;
        this._box!.position = Vector3.Zero();
        this._box!.isVisible = true;

        anchor.attachedNode = boxTransformNode;
    });
}

This is where we add a new anchor at our raycastHit.pickedPointposition.

Since anchors have a fixed position in space and don’t support animations, we first have to help ourselves by creating a so called TransformNode for the rotating box. A TransformNode is a versatile and flexible helper that, in our case, acts as a helper to the box by acting as parent to it. Therefore the box keeps it’s rotation animation.

The position of this transform node is set to the point where the ray cast hit, making it align with the XR anchor.

The box is then parented to boxTransformNode. This means any transformation applied to boxTransformNode will also affect the box. The box's position is set to Vector3.Zero(), which means it's placed at the origin relative to its parent (i.e., the boxTransformNode). Since the transform node is already positioned at the ray cast hit point, this effectively places the box there.

this._box!.isVisible = true; makes the box visible in the scene.

Finally, the transform node is attached to the XR anchor. This ensures that the transform node (and therefore the box) will maintain its position relative to the real world, even as the user moves around in the XR space.

Cleaning up

observeAnchors() {
    if (this._xrAnchors === null) {
        return;
    }
    this._xrAnchors.onAnchorAddedObservable.add((addedAnchor) => {
        this._xrAnchors!.anchors.forEach((anchor: IWebXRAnchor) => {
            if (anchor !== addedAnchor) {
                anchor.remove();
            }
        });
    })
}

this._xrAnchors.onAnchorAddedObservable.add(() => { ... }) sets up an observer that will execute the provided callback function every time a new anchor is added to the XR environment.

Within the callback, there's a loop this._xrAnchors!.anchors.forEach((anchor: IWebXRAnchor) => { ... }) that iterates over all the anchors currently in this._xrAnchors.

Inside the loop, anchor.remove() is called on each anchor except the one we just added. This means that every time a new anchor is added to the system, all existing anchors are removed.

The primary intent of this function is to maintain a single active anchor in the XR environment. When a new anchor is added, it triggers the removal of all existing anchors. This is useful in scenarios where only the latest anchor is relevant, and previous anchors should not persist.

Adding `observeAnchors` to the scene

async createScene(): Promise<Scene> {
  ...
  this.observeAnchors();

  return this._scene;
}

To make use of the function we have to add it to createScene.

Conclusion

In conclusion, this seventh installment of our WebXR/Babylon.js series has taken us through the pivotal concept of Anchors in WebXR. Anchors are more than just a technical component in the realm of extended reality; they serve as the vital bridge connecting the virtual and real worlds. By creating these stable reference points in physical space, which remain consistent regardless of the viewer's movement or perspective, we enhance the immersion and believability of our virtual environments.

The practical demonstrations and code examples provided, from handling controller selections to dynamically adding and managing anchors, have illustrated how these concepts are not just theoretical but highly applicable in real-world scenarios. The use of TransformNode as a flexible tool to maintain object animations while anchoring, and the strategic management of anchors to keep the XR environment clean and relevant, show the depth and versatility of Babylon.js in creating compelling XR experiences.

In the eighth part we dive into loading models and assets.

Unity MR Part 11: Animation

tststs — Thu, 11 Jan 2024 08:51:23 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 The objective of this article is to animate the door in response to the voice command “open the door”. To achieve this, we will develop an animation specifically for the door and make necessary updates to the Player and OpenDoorConduit scripts. This integration of voice command functionality with animation will enhance the interactivity and realism of our application.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

Lets start with the animation. Edit our Door Prefab by double-clicking it within the Project window. Select the Door mesh as seen in the next screenshot.

Selecting the Door mesh on our door Prefab

Now add the component Animator via Add Component.

Adding the Animator component to the Door mesh

With the Door mesh still selected, navigate to Window → Animation → Animation to open the animation window in Unity. Once the animation window is open, click on the Create button. You will then be prompted to choose a location for saving the Animation Clip. Save it in a new folder Assets/Animations and name it DoorAnimation.

The animation window after clicking Create

The next step involves adding a property to animate. As we are crafting an “opening the door” animation, we need to animate the Transform rotation. To do this, click on Add Property, and then select Transform → Rotation. This action allows us to specifically target and animate the door's rotation, creating a realistic opening effect in our animation sequence.

Adding the rotation property to the animation

First move the Timeline to 60. You can either drag and drop the indicator or enter 60 on the left side of the Timeline. Then, enter -120 for the Rotation.y value.

The animation window with the configured Rotation.y property

In your Scene window the door should now look as follows:

Door state at the end of the animation

When creating an animation in Unity, the default setting typically causes the animation to loop continuously. However, since our objective is to trigger the door animation just once upon recognizing the correct voice command, we need to adjust the looping behavior in the Door Animation Controller.

To make this change, navigate to Assets/Animations in the Unity Editor and double-click on the Door Animation controller. This action will open the Animator window with the Door.controller loaded. In the Animator window, we can modify the settings to ensure that the door animation plays only once instead of looping endlessly.

Door controller in our Animator

Right-click on an empty space within the Animator window and select Create State -> Empty, as shown in the upcoming screenshot. This step will add a new, empty state to the animation controller.

Creating an empty state in the Animator

Right-click on the newly created state, named New State, and select Set as Layer Default State. This action will designate the New State as the default state for that particular layer in the animation controller, ensuring that this is the starting state when the animation sequence begins.

Setting New State as the default layer

The result should be as follows: Since the state is empty and no properties have been added to it, this results in the absence of any animation when the GameObject loads. This setup ensures that the door remains static initially, allowing for the animation to be triggered specifically by an event, such as a voice command, rather than starting automatically upon loading.

New State is the default layer

Now, select the DoorAnimation animation clip in your Projectwindow. Then, in the inspector, uncheck Loop Time.

We are now good to go to trigger the animation in our MRArticleSeriesController script.



using System.Collections;
using System.Collections.Generic;
using Meta.WitAi;
using Meta.WitAi.Requests;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using UnityEngine.XR.Interaction.Toolkit;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MRArticleSeriesController : MonoBehaviour
    {
        [SerializeField] private ARAnchorManager anchorManager;
        [SerializeField] private GameObject door;
        [SerializeField] private GameObject uI;
        [SerializeField] private InputActionReference buttonActionLeft;
        [SerializeField] private InputActionReference buttonActionRight;
        [SerializeField] private VoiceService voiceService;
        [SerializeField] private XRRayInteractor rayInteractor;
        private VoiceServiceRequest voiceServiceRequest;
        private VoiceServiceRequestEvents voiceServiceRequestEvents;
        private GameObject doorInstance;

        void OnEnable()
        {
            Debug.Log("MRArticleSeriesController -> OnEnable()");

            buttonActionRight.action.performed += OnButtonPressedRightAsync;
            buttonActionLeft.action.performed += OnButtonPressedLeft;
        }

        void OnDisable()
        {
            Debug.Log("MRArticleSeriesController -> OnDisable()");
            buttonActionRight.action.performed -= OnButtonPressedRightAsync;
            buttonActionLeft.action.performed -= OnButtonPressedLeft;
        }

        private void ActivateVoiceService()
        {
            Debug.Log("MRArticleSeriesController -> ActivateVoiceService()");

            if (voiceServiceRequestEvents == null)
            {
                voiceServiceRequestEvents = new VoiceServiceRequestEvents();

                voiceServiceRequestEvents.OnInit.AddListener(OnInit);
                voiceServiceRequestEvents.OnComplete.AddListener(OnComplete);
            }

            voiceServiceRequest = voiceService.Activate(voiceServiceRequestEvents);
        }

        private void DeactivateVoiceService()
        {
            Debug.Log("MRArticleSeriesController -> DeactivateVoiceService()");
            voiceServiceRequest.DeactivateAudio();
        }

        private void OnInit(VoiceServiceRequest request)
        {
            uI.SetActive(true);
        }

        private void OnComplete(VoiceServiceRequest request)
        {
            uI.SetActive(false);
            DeactivateVoiceService();
        }

        private async void OnButtonPressedRightAsync(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync()");

            if (doorInstance != null)
            {
                Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync(): Door already instantiated");

                return;
            }

            if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
            {
                Pose pose = new(hit.point, Quaternion.identity);
                Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

                result.TryGetResult(out ARAnchor anchor);

                if (anchor != null)
                {
                    // Instantiate the door Prefab
                    doorInstance = Instantiate(door, hit.point, Quaternion.identity);
                    // Unity recommends parenting your content to the anchor.
                    doorInstance.transform.parent = anchor.transform;
                }
            }
        }

        private void OnButtonPressedLeft(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressedLeft()");

            ActivateVoiceService();
        }

        public void OpenDoor()
        {
            Debug.Log("MRArticleSeriesController -> OpenDoor()");

            if (doorInstance == null)
            {
                Debug.Log("MRArticleSeriesController -> OpenDoor(): no door instantiated yet.");

                return;
            }

            Animator animator = doorInstance.GetComponentInChildren<Animator>();
            animator.Play("DoorAnimation", -1, 0);
        }
    }
}

Let's go over the modifications made to the MRArticleSeriesController script:

Added private GameObject doorInstance; This variable holds the reference to the instantiated door in the scene.
OnButtonPressedRightAsync: In this method, we now check if a door has already been instantiated in the scene. If a door is present, the method returns early to prevent another door from being instantiated.
OpenDoor: This public method will be called from the OpenDoorConduit. It triggers the door's animation using animator.Play, initiating the opening sequence of the door.

These changes enhance the functionality of the Player script, ensuring proper management and control of the door animation in response to user interactions and voice commands.

Now, edit the OpenDoorConduit script.



using System.Collections;
using System.Collections.Generic;
using Meta.WitAi;
using UnityEngine;

namespace Taikonauten.Unity.ArticleSeries
{
    public class OpenDoorConduit : MonoBehaviour
    {
        [SerializeField] private MRArticleSeriesController mRArticleSeriesController;
        private const string OPEN_DOOR_INTENT = "open_door";

        [MatchIntent(OPEN_DOOR_INTENT)]
        public void OpenDoor(string[] values)
        {
            Debug.Log("OpenDoorConduit -> OpenDoor()");

            string action = values[0];
            string entity = values[1];

            if (!string.IsNullOrEmpty(action) && !string.IsNullOrEmpty(entity))
            {
                if (action == "open" && entity == "door") {
                    mRArticleSeriesController.OpenDoor();
                }
            }
        }
    }
}

Let's go over the modifications made to the OpenDoorConduit script:

[SerializeField] private MRArticleSeriesController mRArticleSeriesController; This line declares a private variable and marks it with [SerializeField] so it can be assigned via the Unity Editor. This variable holds the reference to our Player component, allowing the script to interact with the Player component's public methods and properties.
OpenDoor: In this part of the script, we invoke the OpenDoor public method of the Player script which starts the door animation.

These modifications are essential for enabling communication and interaction between different components and scripts in our Unity project, particularly for handling the door-opening functionality.

Make sure to select the MRArticleSeriesController component in the inspector:

Selecting the MRArticleSeriesController for the Open Door Conduit script

Testing the app

We are now prepared to test the app. Select Build and Run.

Press the trigger on the left controller.
The label indicated "...Listening...".
Speak the word phrase “open the door”.
After a brief delay the door animation should now be playing.

Video of the app where the door opens when the phrase “open the door“ is recognized

In the next article, we will focus on enhancing the visual appeal of our scene. This will include implementing graphic improvements, adding a scene to be displayed behind the door, and disabling the plane material, which was previously used solely for debugging purposes. These refinements are aimed at elevating the aesthetic quality and immersive experience of our application, making it more engaging and visually appealing to users.

Part 6: Animating a Mesh (WebXR with Babylon.js)

Bryan — Thu, 11 Jan 2024 08:51:22 +0000

👀 Stumbled here on accident? Start with the first part!

Welcome back to the 6th part of this series about Animations in Babylon.js.

ℹ️ Remember - you can always run the code associated with this article and follow along using

npm start --part=6

animating the box mesh

Essentials & Similarities

Babylon.js offers a robust animation system that allows for the creation and control of animations within 3D scenes.

Animations can be applied to various properties of objects, including position, rotation, scaling, and color.

The system supports keyframe-based animations, where you can define specific values at certain frames, creating smooth transitions over time.

There are similarities to CSS animations:

Keyframe-Based: Both Babylon.js and CSS animations can be keyframe-based, where you define certain states at specific points in time, and the system interpolates the values in between.
Easing Functions: Both can use easing functions to make the transitions between keyframes smooth and more natural.
Transformations: They can both animate transformations like scaling, rotating, and moving objects or elements.
Timeline Control: They offer control over the timing of animations, allowing you to set durations, delays, and iteration counts.

📚 Babylon.js also offers advanced features like blending multiple animations, easing functions for smooth transitions, and the ability to control the playback of animations, including starting, stopping, pausing, and looping.

This flexibility makes it suitable for creating complex and dynamic 3D animations in web applications.

Animating a box

For our experience we’re going to animate a rotation of the box we previously added into our scene.

In effect, this function will rotate the box by 180 degrees around the y-axis, with the rotation occurring smoothly over the span of 100 frames. A looping ensures that the animation will continue indefinitely, creating a rotating box effect in the scene.

These are the steps to accomplish our goal:

animateBox() {
    const rotateAnimation = new Animation("rotateAnimation", "rotation.y", 30, Animation.ANIMATIONTYPE_FLOAT, Animation.ANIMATIONLOOPMODE_CYCLE);

    const keyFrames: { frame: number, value: number }[] = [];

    keyFrames.push({
        frame: 0,
        value: 0
    });

    keyFrames.push({
        frame: 50,
        value: Math.PI / 2
    });

    keyFrames.push({
        frame: 100,
        value: Math.PI
    });

    rotateAnimation.setKeys(keyFrames as IAnimationKey[]);

    this._box!.animations = [rotateAnimation];

    this._scene.beginAnimation(this._box, 0, 100, true);
}

Create Animation Object: A new Animation object named rotateAnimation is created. This animation is set to change the rotation.y property, which affects the y-axis rotation of the box.
Animation Properties:
- 30 specifies the frame rate per second. This indicates how many times the animation values are updated per second.
- Animation.ANIMATIONTYPE_FLOAT indicates that the property being animated (rotation.y) is a float.
- Animation.ANIMATIONLOOPMODE_CYCLE ensures that the animation will repeat in a loop.
Define Key Frames:
- Keyframes are set up to define how the animation progresses over time. Each keyframe specifies a frame number and the corresponding value of the rotation.y at that frame.
- The first keyframe at frame 0 sets the rotation to 0 (start position).
- The second keyframe at frame 50 sets the rotation to Math.PI / 2, which is a 90-degree rotation.
- The third keyframe at frame 100 sets the rotation to Math.PI, a 180-degree rotation.
Apply Keyframes to Animation: The setKeys method of the rotateAnimation object is used to apply the keyframes. This method defines the animation sequence according to the keyframes.
Assign Animation to the Box: The animation array of the _box object (presumably a predefined mesh in the scene) is set to include the rotateAnimation.
Begin Animation: Finally, this._scene.beginAnimation is called with the _box, the start frame (0), the end frame (100), and a boolean to indicate that the animation should loop (true).

Adding the animation to the scene

async createScene(): Promise<Scene> {
  ...
  this.animateBox();

  return this._scene;
}

Finally, we’re adding the this.animateBox() function to the scene.

Conclusion

This article demonstrates the powerful and flexible animation system of Babylon.js, ideal for creating dynamic 3D animations in web applications. Through a practical example of animating a box, it showcases how to set up keyframe-based animations, define keyframes, and apply them to create smooth and looping animations, emphasizing Babylon.js's capability to bring 3D objects to life with ease and precision.

The seventh part of the series will be about Anchors.

Unity MR Part 10: Voice SDK

tststs — Thu, 11 Jan 2024 08:51:19 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 The aim of this article is to incorporate the Meta - Voice SDK into our application, enabling it to respond to specific word sequences. This functionality will allow us to interact with the door we rendered in the previous article, providing a more immersive and interactive experience in our application.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

Login to the Unity Store Unity Asset Store - The Best Assets for Game Making and add Meta - Voice SDK - Immersive Voice Commands to your library.

Return to the Unity Editor and install the Meta - Voice SDK through the Package Manager. You can directly access My Assets via Window -> My Assets.

Adding the Meta - Voice SDK with the Package Manager

Before we begin utilizing the Meta - Voice SDK, it's necessary to create an account on Wit.ai . You can conveniently use your existing Meta developer account for this purpose by clicking on Continue with Meta on the Wit.ai landing page.

ℹ️ Wit.ai (often referred to as Won in Translation, by David Jacobs from its URL) is a natural language processing (NLP) service created by Facebook. It enables developers to build applications that can understand human language by providing a powerful and easy-to-use API.

Once you've set up your Wit.ai account, you can create a new application at Wit.ai. If you're unsure about what to name the application, simply go with unity_example.

After your app is created, click on unity_example. Then, as illustrated in the upcoming screenshot, add an Utterance for the Intent open_door. This step is crucial for training your application to recognize and respond to specific user inputs related to the action of opening a door.

ℹ️ An Utterance in the context of natural language processing (NLP), linguistics, and conversational AI, refers to a sequence of words or sounds made by a speaker. It's essentially a unit of speech. In practical terms, an utterance can be as short as a single word (like a command or an exclamation) or as long as a complete sentence or multiple sentences.

ℹ️ In Wit.ai, an Intent represents the purpose or goal behind a user's input, typically a spoken or written phrase. It's a fundamental concept in natural language understanding (NLU) and is used to categorize user utterances into specific actions that the application should perform.

1. Fill in the phrase “open the door” as the Utterance and select the open_door intent in the Intent dropdown.

Adding the Utterance for the Intent open_door

2. In the Utterance input field select the word “open”. This will open the entity form. Fill in action as the entity and click on + Create Entity.

Creating the action entity

3. In the Utterance input field select the word “door”. This will open the entity form. Fill in entity as the entity and click on + Create Entity.

Creating the entity entity

4. After creating the entities, they will now be highlighted as seen in the next screenshot.

The Utterance view after creating our entities

Now, click on Train and Validate. Once the training process is complete (indicator on the top left next to the app name), return to the Unity Editor and navigate to Oculus → Voice SDK → Get Started. In the first dialog enter the Wit Server Access Token. The access token can be found on the Wit.ai website under Management → Settings within your unity_example app.

Enter the Server Access Token

You will be prompted to choose a location for saving your Wit asset. Save it in your Assets/Settings folder and name it wit. Once you have saved the asset, the following screen will appear.

The Wit Configurations after entering the server token

Now, click on Specify Assemblies and uncheck everything except the first entry:

De-selecting unnecessary assemblies

Click on Generate Manifest then close the Voice Hub for now.

The next step is to respond to the Utterance “open the door”. Create a new Script under Assets/Scripts/VoiceSDK and name it OpenDoorConduit. Add the Script to the XR Origin (XR Rig) via Add Component.

Creating the OpenDoorConduit Script under Scripts/VoiceSDK

Adding the OpenDoorConduit Script to the XR Origin (XR Rig) GameObject

The Script looks as follows:

using System.Collections;
using System.Collections.Generic;
using Meta.WitAi;
using UnityEngine;

namespace Taikonauten.Unity.ArticleSeries
{
    public class OpenDoorConduit : MonoBehaviour
    {
        private const string OPEN_DOOR_INTENT = "open_door";

        [MatchIntent(OPEN_DOOR_INTENT)]
        public void OpenDoor(string[] values)
        {
            Debug.Log("OpenDoorConduit -> OpenDoor()");

            string action = values[0];
            string entity = values[1];

            if (!string.IsNullOrEmpty(action) && !string.IsNullOrEmpty(entity))
            {
                Debug.Log("OpenDoorConduit -> OpenDoor(): match");
            }
        }
    }
}

As you can see we are using MatchIntent with the open_door Intent which we created in a previous step. The method OpenDoor is automatically called by the ConduitDispatcher.

ℹ️ ConduitDispatcher is used to manage voice commands, direct them to the appropriate processing channels, or handle the distribution of responses or actions triggered by voice input.

Now, proceed by adding the App Voice Experience component to the XR Origin (XR Rig) GameObject. Once you have added this component, it's necessary to select the Wit configuration, which you have created during the Get Started process for Wit.

Adding App Voice Experience to the XR Origin (XR Rig) GameObject and selecting the Wit configuration

For the final step, we must add the Response Matcher. This component generates the Android Intent open_door in the Android manifest file, and upon receiving a successful response from Wit, it triggers our OpenDoor method, which we defined earlier.

To do this, open the Understanding Viewer via Oculus → Understanding Viewer. Enter “open the door“ in the Utterance field and click Send.

Now right-click value and select Add response matcher to XR Origin (XR Rig).

Adding the response matcher via the Understanding Viewer

Adding the value matcher via the Understanding Viewer

This will result in the following screenshot:

The XR Origin (XR Rig) after adding the response matcher and value matchers

Next, create an entry under On Multi Value Event and select the values as follows:

Selecting the OpenDoor method for the new On Multi Value Event

Android Setup

To ensure compatibility with the Voice SDK, some adjustments are needed for the Android build. Access the Project Settings by going to Edit → Project Settings. This step is crucial for configuring your project to work seamlessly with the Voice SDK on Android.

1. In the Project Settings, navigate to the Player section. There, change the Minimum API Level to Android 10.0 (API Level 29) and the Target API Level to Android 12L (API Level 32). These options are located under Other Settings.

Setting the Minimum and Target API level for Android

2. In the Project Settings, go to the Player section, find Application Entry Point under Other Settings and change the value from GameActivity to Activity.

Setting Activity instead of GameActivity as the Application Entry Point

ℹ️ You can find more information about application entry points in the Unity documentation Unity - Manual: Android application entry points. As of the time of writing, it's important to note that GameActivity is not compatible with the Meta Voice SDK.

3. In the Project Settings, go to the Player section, open the Publishing Settings tab and enable Custom Main Manifest. After activating this option, you will find the manifest file located at Assets/Plugins/Android/AndroidManifest.xml. This step is essential for gaining direct control over the Android manifest file, allowing you to make specific customizations needed for your project.

Enable the Custom Main Manifest option for Android

Open the AndroidManifest file in your code editor and modify its content as follows.

<?xml version="1.0" encoding="utf-8"?>
<manifest
    xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:tools="http://schemas.android.com/tools">
    <application>
        <activity android:name="com.unity3d.player.UnityPlayerActivity"
                  android:theme="@style/UnityThemeSelector">
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
            <meta-data android:name="unityplayer.UnityActivity" android:value="true" />
            <meta-data android:name="unityplayer.SkipPermissionsDialog" android:value="false" />
        </activity>
    </application>
</manifest>

Let's review the changes we've made. This will help us understand the adjustments and their impact on the application's functionality and compatibility.

We have removed the GameActivity block, as only one Activity is permitted and we previously opted for UnityActivity instead of GameActivity in the Project Settings.
We included the unityplayer.SkipPermissionsDialog setting with a value of false to ensure that required permission dialogs are not automatically bypassed. This adjustment is important for guaranteeing that the application appropriately prompts users for necessary permissions, aligning with best practices for user consent and app functionality.

Adding some UI

Before we can test our implementation, it is essential to integrate some user interface elements into the scene. We will add a UI Label that becomes visible when the application starts listening to the user's voice. This label will then disappear once the recording ceases, triggered by a successful response from Wit. This UI component plays a crucial role in providing visual feedback to the user about the state of voice recognition within the application.

To set up the user interface for voice recognition feedback, follow these steps:

1. Create an empty GameObject in your scene and name it UI.
2. Within the UI GameObject, add another empty GameObject and name it VoiceSDK.
3. With the VoiceSDK GameObject selected in the hierarchy attach the Lazy Follow script via Add Component. Configure the script as follows:

Enable the Custom Main Manifest option for Android

4. Add a Canvas to the VoiceSDK GameObject by right-clicking it and navigating to UI -> Canvas.

5. Inside the Canvas, add a Text element by choosing UI → Text - TextMeshPro.

This setup creates a structured UI hierarchy in your scene, with the VoiceSDK GameObject serving as a container for the elements that will provide visual feedback for voice recognition. The Lazy Follow script will manage the positioning, and the TextMeshPro element will display the necessary information or status messages.

Your hierarchy should now look as follows:

Hierarchy after adding the UI

Select the EventSystem GameObject in your hierarchy and delete and add components like follows:

The UI EventSystem

Don’t forget to remove the Standalone Input Module if any.

Next, we need to configure our Canvas and Text (TMP) elements. Select the Canvas GameObject in your hierarchy and set it up as follows (Add components as seen in the screenshot via Add Component).

ℹ️ We won't be delving into UI-related topics in this article series. For those who need assistance or guidance with Unity's UI system, I recommend checking out the Unity documentation. It provides comprehensive resources and tutorials that can help you understand and effectively use Unity's UI tools in your projects.

The complete Canvas configuration

The complete Text (TMP) configuration

Lastly, deactivate the VoiceSDK GameObject, as we won't be displaying it immediately. The visibility of the UI will be managed later through our script.

ℹ️ If you're not familiar with how to deactivate a GameObject in the Unity Inspector, I recommend consulting the Unity documentation: Unity - Manual: Deactivate GameObjects.

Updating our MRArticleSeriesController Script

For the final step in this article, we'll update our MRArticleSeriesController Script to enable and disable the Voice Service using the left Trigger. This modification will allow for straightforward control of the Voice Service directly through user input, enhancing the interactive capabilities of our application.

using System.Collections;
using System.Collections.Generic;
using Meta.WitAi;
using Meta.WitAi.Requests;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using UnityEngine.XR.Interaction.Toolkit;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MRArticleSeriesController : MonoBehaviour
    {
        [SerializeField] private ARAnchorManager anchorManager;
        [SerializeField] private GameObject door;
        [SerializeField] private GameObject uI;
        [SerializeField] private InputActionReference buttonActionLeft;
        [SerializeField] private InputActionReference buttonActionRight;
        [SerializeField] private VoiceService voiceService;
        [SerializeField] private XRRayInteractor rayInteractor;
        private VoiceServiceRequest voiceServiceRequest;
        private VoiceServiceRequestEvents voiceServiceRequestEvents;

        void OnEnable()
        {
            Debug.Log("MRArticleSeriesController -> OnEnable()");

            buttonActionRight.action.performed += OnButtonPressedRightAsync;
            buttonActionLeft.action.performed += OnButtonPressedLeft;
        }

        void OnDisable()
        {
            Debug.Log("MRArticleSeriesController -> OnDisable()");
            buttonActionRight.action.performed -= OnButtonPressedRightAsync;
            buttonActionLeft.action.performed -= OnButtonPressedLeft;
        }

        private void ActivateVoiceService()
        {
            Debug.Log("MRArticleSeriesController -> ActivateVoiceService()");

            if (voiceServiceRequestEvents == null)
            {
                voiceServiceRequestEvents = new VoiceServiceRequestEvents();

                voiceServiceRequestEvents.OnInit.AddListener(OnInit);
                voiceServiceRequestEvents.OnComplete.AddListener(OnComplete);
            }

            voiceServiceRequest = voiceService.Activate(voiceServiceRequestEvents);
        }

        private void DeactivateVoiceService()
        {
            Debug.Log("MRArticleSeriesController -> DeactivateVoiceService()");
            voiceServiceRequest.DeactivateAudio();
        }

        private void OnInit(VoiceServiceRequest request)
        {
            uI.SetActive(true);
        }

        private void OnComplete(VoiceServiceRequest request)
        {
            uI.SetActive(false);
            DeactivateVoiceService();
        }

        private async void OnButtonPressedRightAsync(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync()");

            if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
            {
                Pose pose = new(hit.point, Quaternion.identity);
                Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

                result.TryGetResult(out ARAnchor anchor);

                if (anchor != null)
                {
                    // Instantiate the door Prefab
                    GameObject _door = Instantiate(door, hit.point, Quaternion.identity);

                    // Unity recommends parenting your content to the anchor.
                    _door.transform.parent = anchor.transform;
                }
            }
        }

        private void OnButtonPressedLeft(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressedLeft()");

            ActivateVoiceService();
        }
    }
}

Let's review the updates quickly:

Added a field named uI to hold the GameObject that will be activated or deactivated based on the VoiceService state.
Included a field called voiceService to reference the App Voice Experience component added to the XR Origin (XR Rig) GameObject.
Introduced a field voiceServiceRequest to store the active request to the VoiceService.
Added a voiceServiceRequestEvents field, which is passed to the VoiceService. This ensures that the OnInit and OnComplete methods are called by the VoiceService.
In OnEnable and OnDisable we add and remove the OnButtonPressedLeft action, so we can respond to the left controller Trigger press.
ActivateVoiceService method, which activates the VoiceService, creates VoiceServiceRequestEvents if not already initialized, and is called via OnButtonPressedLeft when the user presses the left Trigger.
DeactivateVoiceService which simple deactivates the recording of the VoiceService.
OnInit Invoked when the VoiceService starts listening. It enables the UI GameObject to inform the user that the app is now listening.
OnComplete called when a voice request succeeds. It also invokes DeactivateVoiceService to stop the VoiceService from listening.
OnButtonPressedLeft triggered when the left Trigger is pressed.

Please be aware that we have also renamed some variables and methods. After saving these changes, remember to return to the Unity Editor and update the fields for the Player component accordingly.

Updating our values on the MRArticleSeriesController component

Testing the app

We are now prepared to test the app. Select Build and Run, and once the app is running, press the trigger on the left controller. You should see a label appear in front of you indicating "...Listening...". At this point, say the phrase "open the door". After a brief delay, the label should disappear. This process will allow you to verify the functionality of the voice recognition feature in your application.

View of the app after a left Trigger press

Additionally, you can verify in the console if the Intent was triggered, as shown in the following screenshot. This will provide a clear indication of whether the voice command was successfully recognized.

Console output after the voice command was successfully recognized

In our upcoming article, we will take an exciting step forward by leveraging the voice command functionality we've established to initiate an animation that opens the door. This integration represents a significant enhancement in our application, blending voice recognition with dynamic visual feedback.

Part 5: Input/Controllers & Ray Casting (WebXR with Babylon.js)

Bryan — Thu, 11 Jan 2024 08:51:16 +0000

👀 Stumbled here on accident? Start with the first part!

Welcome back to the 5th installment of this WebXR/Babylon.js series. This part is about Input (specifically MetaQuests Motion Controllers) and the concept of Ray Casting.

ℹ️ Remember - you can always run the code associated with this article and follow along using

npm start --part=5

interacting with a mesh by changing its color

Ray Casting

What is a Ray Cast?

Ray casting is used to determine the line of sight, select or interact with virtual objects, or navigate through a virtual environment.

How does it work?

Ray casting involves projecting a line (ray) from a specific point into the virtual environment and determining what objects in the virtual world it intersects with. For example, in a VR game, you might use ray casting to select an item or to aim a weapon.

Hit Testing vs. Ray Casting

❗️ Hit testing and ray casting might be perceived as similar because both involve projecting rays in a 3D environment to determine intersections. This fundamental similarity in their core concept—using rays to find points of interaction—can make them seem alike, especially in scenarios where both methods are used for spatial analysis or object interaction within virtual or augmented reality environments.

✅ Hit testing in augmented and virtual reality determines where a virtual object intersects with the real-world environment, while ray casting involves projecting a line from a point in space to detect intersections primarily within a virtual environment.

Performing a Ray Cast from a Controller Input Source

createRayFromController(controller: WebXRInputSource): Ray {
    const origin = controller.pointer.position;
    const direction = controller.pointer.forward;
    return new Ray(origin, direction, length = 100);
}

The createRayFromController method in this context generates a Ray object originating from a WebXR controller. It takes a WebXRInputSource object (representing the controller) as an argument. The method sets the ray's origin to the controller's pointer position and its direction to the forward direction of the controller's pointer. The ray is given a length of 100 units. This functionality is typically used in virtual environments to create a directional line for purposes like aiming or selecting objects in the direction the controller is pointing.

Controller Detection and Interaction

📚 Babylon.js supports a wide range of motion controllers. Each controller's buttons, thumb sticks, and touchpads are updated every frame, providing real-time input data.

handleControllerSelection() {

    if (this._xr === null) {
        return;
    }
    this._xr.input.onControllerAddedObservable.add((motionControllerAdded) => {
        motionControllerAdded.onMotionControllerInitObservable.add((motionControllerInit) => {
            const motionControllerComponentIds = motionControllerInit.getComponentIds();
            const triggerComponent = motionControllerInit.getComponent(motionControllerComponentIds[0]);

            triggerComponent.onButtonStateChangedObservable.add((component) => {
                if (component.pressed && component.value > 0.8) {
                    const resultRay = this.createRayFromController(motionControllerAdded);
                    const raycastHit = this._scene.pickWithRay(resultRay);
                    if (this._debug) console.log(raycastHit);

                    if (raycastHit && raycastHit.hit && raycastHit.pickedMesh) {
                        if (raycastHit.pickedMesh === this._box) {
                            const mat = this._box!.material as StandardMaterial;
                            mat.diffuseColor = Color3.Random();
                            this._box!.material = mat;
                        }
                    }
                }
            });
        });
    });
}

The function handleControllerSelection() illustrates how to respond to inputs from a motion controller in an XR scene.

Initialization: First, it checks if the XR session is present and initialized.

Controller Events: When a controller is added, an observable onControllerAddedObservable is activated. This observable responds to the controller's initialization by listening to the controller’s onMotionControllerInitObservable.

Trigger Component: Once the controller is initialized, the trigger component of the controller (usually the trigger button) is identified. This component is then used to respond to press events by listening to the onButtonStateChangedObservable.

motionControllerComponentIds[0] corresponds to the standard trigger button on an XR controller like a MetaQuest 3 controller.

The observer reacts to us pressing the trigger button on the controller. Since the trigger button can be pressed to various degrees, we want to make sure the button is at least pressed for 80%, so quite strong, before reacting to it. If both requirements are met, the ray cast is created from the controller.

If our ray cast hits the box (if we are aiming at it and pulling the trigger on our controller), the color of the box will randomly change to a different color.

More about inputs and controllers can be found here

Adding the Controller Selection to the scene

async createScene(): Promise<Scene> {
  ...
  this.handleControllerSelection();

  return this._scene;
}

Finally, we add the handleControllerSelection to the scene.

Conclusion

Babylon.js provides robust support for various input sources and controllers in WebXR, effectively managing them through classes like WebXRInput and WebXRInputSource. This flexibility allows for seamless integration and real-time interaction with different types of controllers, enhancing the user experience in virtual environments.

This article demystifies the concepts and practical applications of ray casting and hit testing in virtual environments, illustrating their similarities and differences. It emphasizes the use of ray casting for interactions within virtual spaces, such as selecting or aiming at objects, and demonstrates how to implement ray casting in a practical scenario using WebXR controllers. This highlights the importance and versatility of these techniques in developing immersive and interactive virtual reality experiences.

In the sixth part we dive into Animations.

Unity MR Part 9: Instantiate Prefab

tststs — Thu, 11 Jan 2024 08:51:13 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 The objective of this article is to demonstrate how to instantiate a Prefab — in our example, a door — and subsequently anchor it within the MR environment. This process is key to effectively integrating virtual objects, like the door, into the MR space, ensuring they are correctly positioned and maintained within the real-world context.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

First, let's create the door Prefab required for this article. Download the following archive door.rar and extract its contents into your Assets/3DModel folder. Confirm the following dialog with Fix now after dropping the files.

Confirm with Fix now to mark the texture as a normal map

Then, create a new Prefab in your Assets/Prefabs folder using the same method as we did for the reticle in a previous article. Once you've completed these steps, your Prefab should look like this:

The Door Prefab in edit mode

We are now set to instantiate the door Prefab and anchor it within the environment. This action will occur when the user hovers over a valid plane and presses the Trigger button.

Here's how the updated MRArticleSeriesController script will look:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using UnityEngine.XR.Interaction.Toolkit;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MRArticleSeriesController : MonoBehaviour
    {
        public ActionBasedController controller;
        public InputActionReference buttonAction;
        public XRRayInteractor rayInteractor;
        public ARAnchorManager anchorManager;
        public GameObject door;

        void OnEnable()
        {
            Debug.Log("MRArticleSeriesController -> OnEnable()");
            buttonAction.action.performed += OnButtonPressedAsync;
        }

        void OnDisable()
        {
            Debug.Log("MRArticleSeriesController -> OnDisable()");
            buttonAction.action.performed -= OnButtonPressedAsync;
        }

        private async void OnButtonPressedAsync(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressed()");

            if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
            {
                Pose pose = new(hit.point, Quaternion.identity);
                Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

                result.TryGetResult(out ARAnchor anchor);

                if (anchor != null)
                {
                    // Instantiate the door Prefab
                    GameObject \_door = Instantiate(door, hit.point, Quaternion.identity);

                    // Unity recommends parenting your content to the anchor
                    // instead of adding a ARAnchor component to the GameObject
                    \_door.transform.parent = anchor.transform;
                }
            }
        }
    }
}

Let's review the updates quickly:

A public GameObject member named door has been added, enabling us to select the door Prefab in the Inspector.
Instead of simply logging the anchor instance, we now verify if the returned anchor is not null.
The door Prefab is instantiated at the raycast hit point.
The door (our content) is parented to the anchor, following Unity's recommended practices.

With these updates to the MRArticleSeriesController Script, don't forget to select the door Prefab in the Inspector to complete the setup.

Selecting the door Prefab in our MRArticleSeriesController Script

To test the application, choose Build and Run, as detailed in the previous articles. Hover over a Plane and press the Trigger button. If all configurations are correct, you should see a similar result as seen in the next video:

Selecting the door Prefab in our MRArticleSeriesController Script

In our forthcoming article, we will take an exciting step in our MR journey by integrating the VoiceSDK. This addition will enable voice interaction within our MR environment, particularly focusing on using voice commands to interact with the door we've created. We'll guide you through the process of integrating the VoiceSDK into your Unity project, setting up voice recognition, and configuring it to recognize specific commands -- such as "open the door".

Part 4: Hit Testing (WebXR with Babylon.js)

Bryan — Thu, 11 Jan 2024 08:51:10 +0000

👀 Stumbled here on accident? Start with the first part!

Welcome back to the next part of our series. This time we learn about Hit Testing.

ℹ️ Remember - you can always run the code associated with this article and follow along using

npm start --part=4

a torus making a hit test visible

What is a Hit Test?

ℹ️ A hit test in WebXR is typically used to determine where in the real world a virtual object should be placed or how it should interact with the real environment.

📚 We will achieve this by creating a torus mesh (looks like a Donut 🍩) that’ll show us on which planes a hit can be detected and where virtual objects can be placed in accordance to the physical environment. Like placing an object on the floor, or on a detected table.

How it works

A hit test involves sending a virtual ray from a specific point (often the user's viewpoint or a point in virtual space) and checking where this ray intersects with real-world surfaces.

📚 While it is common to use the camera (representing the user's view in most cases) as the origin for hit tests, especially in scenarios like first-person views or camera-based selections, Babylon.js is flexible enough to allow hit tests to be initiated from any point in the 3D space.

Use cases

Camera-Based Hit Testing: The most straightforward and common scenario, where the hit test is performed from the camera's viewpoint. This is typically used for selecting or interacting with objects in the scene based on the user's current view.

Arbitrary Point in Space: Developers can initiate a hit test from any arbitrary point in the 3D space, not just the camera. This is useful for scenarios where you need to check for intersections or collisions at specific locations in the scene, independent of the camera's position.

Moving Objects: Hit tests can also be initiated from moving objects within the scene, like a character's hand or a moving vehicle, to check for interactions or collisions with other objects in the scene.

📚 The choice of the origin for a hit test depends on the specific requirements of the application being developed.

ℹ️ Hit testing often relies on the detection of planes or surfaces in the real world to place virtual objects accurately.

Adding a Mesh as a marker

First we create a Mesh in form of a torus that serves us as a visual marker for a hit test.

addMarkerForHitTest() {
    this._marker = MeshBuilder.CreateTorus("marker", { diameter: 0.15, thickness: 0.05 }, this._scene);
    this._marker.isVisible = false;
    this._marker.receiveShadows = true;
    this._marker.checkCollisions = true;
    this._marker.rotationQuaternion = new Quaternion();
}

Performing a Hit Test

Next we perform hit testing to determine the interaction between the real world and virtual objects.

addFeaturesToSession() {
  ...
  try {
    ...
    this._xrHitTest = this._fm.enableFeature(WebXRFeatureName.HIT_TEST, "latest") as WebXRHitTest;
  } catch (error) {
    ...
  }
  ...
}

First we initialize and enable the hit testing feature in Babylon.js by adding it as a feature to our session.

performHitTest() {
    if (this._xrHitTest === null || this._marker === null) {
        return;
    }
    this._xrHitTest.onHitTestResultObservable.add((results) => {
        if (results.length) {
            this._marker!.isVisible = true;
            this._hitTest = results[0];
            this._hitTest.transformationMatrix.decompose(undefined, this._marker!.rotationQuaternion!, this._marker!.position);
        } else {
            this._marker!.isVisible = false;
            this._hitTest = undefined;
        }
    });
}

As soon as hit test results are available, the onHitTestResultObservable.add observer method is called.

When hit test results are present (results.length > 0):

Makes the marker visible.
Stores the first hit test result (results[0]) in a local variable (_hitTest).
Uses the transformation matrix of the hit test result to update the position and orientation of the marker in the scene. This positions and aligns the marker at the point where the hit test detected a collision with a real surface.

If no hit test results are present, the marker is made invisible again and the local variable _hitTest is reset.

Adding Hit Testing to the scene

async createScene(): Promise<Scene> {
  ...
  this.addMarkerForHitTest();
  this.performHitTest();

  return this._scene;
}

Once again, we have to add the addMarkerForHitTestand the performHitTest functions to the scene.

Conclusion

In summary, hit testing in WebXR, particularly with Babylon.js, is an essential tool for creating interactive augmented reality experiences. It allows developers to place virtual objects accurately within the real world, enhancing the realism and immersion of XR applications. The ability to initiate hit tests from different points, like the user’s camera or specific locations in 3D space, provides versatility in application design. Understanding and applying hit testing is key to develop engaging and believable content.

In the fifth part of this series we take a look at input controllers and ray casting.

Unity MR Part 8: Anchors

tststs — Thu, 11 Jan 2024 08:51:08 +0000

👀 Stumbled here on accident? Start with the introduction!

📚 This article will guide you through the concept of Anchors MR, explaining their significance and functionality. Additionally, it will provide a comprehensive tutorial on implementing Anchors within the Unity platform.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

The aim of this article is to conduct hit tests on planes we created previously, using our controller. We'll then instantiate an Anchor at the identified position. This process is fundamental for accurately positioning virtual objects in relation to the real world.

ℹ️ Anchors in MR are virtual reference points that are fixed to a specific location in the real world. They are used to maintain the position and orientation of virtual objects consistently as the user moves around or interacts with the MR environment.

A typical use case for anchors is to place virtual content in the physical world.

AR Anchor Manager component

The AR Anchor Manager component creates GameObjects for each anchor and is a critical element for managing AR anchors. It acts as a central point for creating, tracking, and updating anchors in the AR environment. This component is essential for ensuring that virtual objects are anchored accurately to real-world locations, maintaining their position and orientation consistently.

⚠️ Throughout its lifespan, the Meta Quest 3 device usually undertakes extra tasks to keep the anchor's position and orientation up to date. Given that anchors are typically resource-intensive objects, it's advisable to use them wisely.

Adding AR Anchor Manager component to the Scene

In the hierarchy view find the XR Origin (XR Rig) GameObject and add the AR Anchor Manager script via Add Component.

The XR Origin (XR Rig) GameObject with the added AR Anchor Manager Script

There's no need to modify any fields in the AR Anchor Manager. The trackables Changed is a list of ARTrackables, managed by the ARTrackableManager. Additionally, the Anchor Prefab field is only necessary if there's a requirement to enhance the default Prefab instantiated for each Anchor.

ℹ️ You can read more about the different Managers in ARFoundation here: Managers | AR Foundation | 6.0.0-pre.5.l.

Let's move on to updating our MRArticleSeriesController.cs script. In this update, we'll adjust the script to instantiate an Anchor at the Pose of a valid Raycast Hit. For the moment, we'll simply log the creation of the Anchor. In the following article, we will focus on instantiating a Prefab and attaching it to this Anchor.

ℹ️ Pose is a representation of a Position, and a Rotation in 3D Space. This structure is used primarily in XR applications to describe the current "pose" of a device in 3D space.

The revised script will be structured as follows:



using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.InputSystem;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
using UnityEngine.XR.Interaction.Toolkit;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MRArticleSeriesController : MonoBehaviour
    {
        public ActionBasedController controller;
        public InputActionReference buttonAction;
        public XRRayInteractor rayInteractor;
        public ARAnchorManager anchorManager;

        void OnEnable()
        {
            Debug.Log("MRArticleSeriesController -> OnEnable()");
            buttonAction.action.performed += OnButtonPressedAsync;
        }

        void OnDisable()
        {
            Debug.Log("MRArticleSeriesController -> OnDisable()");
            buttonAction.action.performed -= OnButtonPressedAsync;
        }

        private async void OnButtonPressedAsync(InputAction.CallbackContext context)
        {
            Debug.Log("MRArticleSeriesController -> OnButtonPressed()");

            if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
            {
                Pose pose = new(hit.point, Quaternion.identity);
                Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

                result.TryGetResult(out ARAnchor anchor);

                Debug.Log(anchor);
            }
        }
    }
}

Before testing the app, it's important to select our AR Anchor Manager. To do this, locate the XR Origin (XR Rig) GameObject in your hierarchy, and then, as indicated in the following screenshot, choose the AR Anchor Manager.

Selecting the AR Anchor Manager for the MRArticleSeriesController.cs Script

Console output for the Meta Quest 3 device

In the upcoming article, we will delve into the process of instantiating a prefab in a MR environment using Unity. This technique is key to bringing virtual objects into your MR space, allowing for a richer and more interactive experience.

Part 3: Meshes & Materials (WebXR with Babylon.js)

Bryan — Thu, 11 Jan 2024 08:51:04 +0000

👀 Stumbled here on accident? Start with the first part!

Welcome back to the next article in our WebXR series. In this article, we take a look at some helpful features implemented in Babylon.js, specifically related to creating objects.

ℹ️ Remember - you can always run the code associated with this article and follow along.

npm start --part=3

Creating a box mesh

What is Material?

ℹ️ Materials give your meshes color and texture.

They can be displayed as a wireframe, scaled and offset across a mesh, have degrees of transparency and be blended.

What is a Mesh?

ℹ️ In the 3D virtual world shapes are built from meshes, lots of triangular facets joined together, each facet made from three vertices.

a mesh

Babylon.js splits meshes into 4 broader categories.

Set Shapes - which usually have names in everyday use and their forms are well known and recognized. Examples include the box, sphere, cone and plane.

Set Shapes

Parametric Shapes - these have no everyday names and are formed through data sets and tend to have irregular shapes. Examples include ribbons, extrusions, and irregular convex or concave polygons.

Parametric Shapes

Polyhedra - three dimensional regular solids including octahedron, dodecahedron, icosahedron and icosphere. The method of creating these is createPolyhedron along with a number for the basic shapes and an array of vertices for a wider range of shapes or createIcoSphere.

Polyhedra

Custom - those you design yourself and build from your own set of vertices connected into triangular facets as you choose.

📚 More about Meshes can be found here.

✅ For the simplicity of this series we’re going to create a simple Box with the help of the Meshbuilder and give it a color using a StandardMaterial.

Creating a Box Mesh

The function createBox() illustrates how easily a 3D object can be created in Babylon.js.
In this example, a simple box is created.

createBox() {
    const material = new StandardMaterial("material", this._scene);

    material.diffuseColor = Color3.Random();

    this._box = MeshBuilder.CreateBox("box", { width: 0.5, height: 0.5, depth: 0.5 }, this._scene);
    this._box.material = material;
    this._box.rotation.y = Math.PI / 4;
    this._box.position = new Vector3(0, 1.5, 1.5);
}

First, a standard material is created, calling StandardMaterial("material", this._scene). Then, the material's diffuse color is set to a random color by calling Color3.Random().

The box itself is created with MeshBuilder.CreateBox("box", { width: 0.5, height: 0.5, depth: 0.5 }, this._scene). This method creates a cube-shaped mesh with the specified dimensions.

The box is then rotated by 45 degrees Math.PI / 4 and positioned at (-1, 1.5, 1.5) in 3D space.
This will place the box a little to the front left relative to us.

Adding the box to the scene

async createScene(): Promise<Scene> {
  ...
  this.createBox();

  return this._scene;
}

Finally, we have to add this.createBox() to the scene.

Conclusion

In wrapping up this article, we've touched on the fundamentals of creating 3D objects in Babylon.js within the WebXR framework. Through the simple box mesh example, we've seen how materials and meshes work together in 3D modeling.

The fourth part of this series will be about Hit Testing.

DEV Community: Taikonauten

Part 9: Asset Handling & Animation (WebXR with Babylon.js)

Interacting with the door

Opening the door

Closing the door

Animating the door interaction

Conclusion

Part 8: Models & Assets (WebXR with Babylon.js)

Prerequisite

Loading the model

Adding the door to the scene

Attaching the model to an anchor

Conclusion

Unity MR Part 12: Improve Scene

Light Estimation

Instantiate the door facing the user

Update AR Default Plane

Optimize Lightning

Optimize URP

The behind the door

Adding a complex scene behind the door

Testing the app

Part 7: Anchors (WebXR with Babylon.js)

What are Anchors?

Adding anchors to the scene

Interacting with the controller

Adding the Anchor

Cleaning up

Adding observeAnchors to the scene

Conclusion

Unity MR Part 11: Animation

Testing the app

Next article

Part 6: Animating a Mesh (WebXR with Babylon.js)

Essentials & Similarities

Animating a box

These are the steps to accomplish our goal:

Adding the animation to the scene

Conclusion

Unity MR Part 10: Voice SDK

Android Setup

Adding some UI

Updating our MRArticleSeriesController Script

Testing the app

Next article

Part 5: Input/Controllers & Ray Casting (WebXR with Babylon.js)

Ray Casting

What is a Ray Cast?

How does it work?

Hit Testing vs. Ray Casting

Performing a Ray Cast from a Controller Input Source

Controller Detection and Interaction

Adding the Controller Selection to the scene

Conclusion

Unity MR Part 9: Instantiate Prefab

Next article

Part 4: Hit Testing (WebXR with Babylon.js)

What is a Hit Test?

How it works

Use cases

Adding a Mesh as a marker

Performing a Hit Test

Adding Hit Testing to the scene

Conclusion

Unity MR Part 8: Anchors

AR Anchor Manager component

Adding AR Anchor Manager component to the Scene

Next article

Part 3: Meshes & Materials (WebXR with Babylon.js)

What is Material?

What is a Mesh?

Babylon.js splits meshes into 4 broader categories.

Creating a Box Mesh

Adding the box to the scene

Conclusion

Adding `observeAnchors` to the scene