DEV Community: Kae Bagg

Simple Unity VR Project

Kae Bagg — Sun, 23 Jun 2019 15:14:56 +0000

(cross-post from my personal blog)

I've been slipping back into coding after putting all of my focus on my secret writing project for a few months¹ and haven't had much to share in the way of guides. But I was kicking off a small VR experiment the other day and wanted to build something with just HMD and simple hand tracking and without including a whole bunch of Oculus stuff. This is a fairly small, and rudimentary tutorial but most of the information about setting up a project with just the most basic tools are tucked away in YouTube tutorials and while I know lots of people find it easiest to pick something up in videos I'm a big fan of the written word.

If you just want the code though, I put it all into a release here so I can continue messing around with that project.

Here's a little mini-guide here in lieu of responsible things like commenting the code. The goal was to avoid using any headset specific code but because I only have a Rift I can only confirm that it works with a Rift.

First, I create a new project with the 3D template:

Next, I go to the 'Edit' menu and select 'Project Settings'. I select the 'Player' tab and check off 'Virtual Reality Supported':

At this point if my Rift is connected and functioning properly when I hit the preview in Unity button the main camera should be attached to my HMD and I can look around the project.

For the hands I'll first create an empty and make the camera a child as well as a sphere mesh that I'll name 'Left Hand':

Then I removed the collider from the Left Hand -- you can optionally scale the Game Object down as well, I found that 0.1 was a good size for hands:

Now, you'll want to make a new script, I called mine Tracking.cs. Make sure that you have Unity's XR tools included by using UnityEngine.XR; You can then delete the "Start()" method because we won't be needing that. The whole script looks like this:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR;

public class Tracking : MonoBehaviour {
    // Unity has some built in means to detect what sort of input device you're using
    public XRNode NodeType;

    void Update() {
        // Set the Transform's position and rotation based on the orientation of the motion controller
        transform.localPosition = InputTracking.GetLocalPosition(NodeType);
        transform.localRotation = InputTracking.GetLocalRotation(NodeType);
    }
}

Click and drag the script you just wrote onto the Left Hand mesh, and set the public NodeType on the script to Left Hand:

If you duplicate the 'Left Hand' we've already made, rename it 'Right Hand' and set the NodeType to 'Right Hand' when you hit play you should see two spheres tracking to your hands.

If things look or track weird for whatever reason try resetting the transforms of your game objects, if the hands start clipping really far from your face change the Clipping Planes on the camera.

And that's how to do it. I'll come back and revise this later as it's a "write while I do" project but hopefully there's some helpful information.

It's still ongoing. ↩

3D Audio Primer

Kae Bagg — Tue, 11 Jun 2019 14:53:39 +0000

(cross-post from my personal blog)

When making VR experiences of all stripes the most important consideration is "how does this help or hurt immersion?" If sound design takes a user out of the space it is bad sound design. Now, unfortunately this can mean different things for different experiences and a focus on realism above everything else is not a panacea. Instead, the best method of building a soundscape is asking yourself "does this fulfill expectations?" When you add a fire in a game but a recording of an actual fire sounds wrong when compared to crumpling a paper bag it's more important to select the proper sounding audio over the more technically accurate sound ¹. As a result, before releasing your sound design into the world it's important to take a good long critical listen to all of your sound elements and make sure they sound good in general, in the context that they are played in, and in relation to the other sounds in the environment.

A major difficulty in 3D audio is everyone has a weirdly shaped head, with weirdly shaped ears with a body that occupies space weirdly and that it's entirely possible that something that sounds completely natural to you and your normal shaped head and ears and body will sound wrong to someone else. There's not a huge amount you can do about this shy of replacing the entire human population with clones of yourself so it's important to get lots of people to listen to what you're building regularly and if you realize you've just spent hours dropping a physics object on the floor unsure if it sounds right remember it's entirely possible that it's you that has the weirdly shaped head, body, etc. To build a somewhat general model of what something sounds like from the perspective of a human being we use a Head-Related Transfer Function.

A Head-Related Transfer Function (or HRTF) is a function used to help simulate the 3D positioning of a sound by calculating not only how that sound reflects off of inanimate objects in a scene like the walls, or tables, but also how sound bounces off our bodies and into our eardrums. HRTF takes into account the shape of ears, head, and body and calculates how a given audio clip will sound from a particular location in a way that is natural to humans.

Game sound for the longest time has been limited to interacting with a 2D plane of a screen and getting sound information out. This form factor opens the door to using hacks to optimize for performance. Background music and ambiences don't need to have a source in the world, and by using panning and attenuation a perfectly convincing soundscape can be derived. The angle of a player's head as well as orientation is extremely limited compared to VR experiences where they can be just about anywhere at any given time. Not only that but sound emitters themselves can also move either programmatically, or by player interaction.

Stepping up from games experienced on 2D planes are 360 degree videos and experiences that are not roomscale. Panning audio to the left and the right no longer is sufficient as the pitch, and roll of a player's head must be taken into account. To solve this problem a technology first developed in the 1970s has started seeing more widespread use -- ambisonics. Ambisonic recordings are made with a special array of (usually) four microphones positioned to record as close to 360 degrees as possible. Using ambisonics it is possible to get a full representation of where sounds have been emitted from and then play them back to a listener. Unlike surround sound ambisonics do not target specific speakers to play particular clips of audio but instead figure out which speaker to play from using the recording itself. Perhaps the main limitation of ambisonics is that it has a small sweet spot and requires a general knowledge of where the listener is -- roomscale isn't an option as soon as a player leaves the defined centrepoint of the audio experience things fall apart.

So instead of trying to emulate the 3D environment we occupy we try to represent the sound as it hits yours ears -- this is where binaural audio comes in. To record audio that emulates the way you hear sounds with your ears two microphones are used positioned on either side of a dummy head (as your brother might be called) as opposed to the array of 4 microphones that ambisonic recordings use. Binaural recordings attempt to create sounds as they are heard instead of how they are emitted and can only be listened to on headphones as a result.

For VR you're generally not actually recording the sounds using either of these methods but draw from concepts used to record binaural sounds -- which is to say, attempt to recreate audio as we hear it. It's simply not flexible enough to build out a single mix and call it a day when you have all these pesky humans listening to it. Instead we use mono sounds and code to generate binaural sounds on the fly. This is why audio can be CPU intensive if you have to calculate out where all the sound waves are bouncing from all the time.

There are three main factors in building a convincing audio mix for VR -- reverb, occlusion, and spatialization. Reverb is using echoes to add a sense of space to a scene, occlusion is what happens when there is an object between a sound emitter and the listener, and spatialization is the use of filters, delay, and attenuation to trick your ears into thinking a sound is coming from a particular location.

Spacialized audio can be very computationally expensive as reflections are calcuated at run time for the most part. I'm sure hand coding these calculations is completely reasonable for super humans but in most cases you'll want to use some kind of library or plugin to handle the math -- there are three main projects to do this Oculus' spatialization plugin, Steam Audio, and Google Resonance. There is a guy who has done an amazing comparison of these three options and if you're trying to decide on which to use for your project I'd urge you to listen to his samples here.

Audio persists in being the unsung hero of immersion in VR. It can be used actively to draw attention to things or passively to fill out a room. I hope to have given you a launchpad for the words, and concepts for audio in VR so you can go forth and make things that sound cool.

We can go ahead and blame Hollywood for messing with our expectations. ↩

Voice Control in Unity

Kae Bagg — Thu, 06 Jun 2019 17:38:50 +0000

(cross-post from my personal blog)

Being a Canadian, I can safely say that using your hands in VR still feels like you are trying to interact with something through about five layers of mittens. With the Oculus touch controllers, being able to point a finger to interact with things is leagues better than the not-for-human-hands Vive controllers (save us, knuckle controllers) but most of us don't use just a single finger to type¹ for a reason.

All this to say, you probably don't want to write your first virtual novel with your virtual hands. And while you would think speaking a novel is also out, it's apparently not unprecedented.

Of course, voice recognition comes with its own limitations as anyone who has tried to communicate with a phone menu or "interactive voice response" system can tell you. But microphone quality is increasing, as is the AI that figures out what you are trying to say² . While we might not be there yet, the tech is slowly coming over the horizon.

The primary hurdles in using voices in VR is twofold -- both social aspects. The first, as I discovered when I still worked in an office with other humans, is that it's really awkward to talk to your computer when other people are around. VR already involves flailing around with a huge chunk of plastic on your face, and a cable you must constantly gingerly step around. Voice is fundamentally not private, and even if you are comfortable dictating simple commands in a room full of people,³ the noise other people make in a room can create a "mergers and acquisitions" problem⁴.

The other major problem with voice is that it can skewer social VR. Communicating with other people and telling a machine what to do is just like interleaving all your report writing with your text messages and then sending that report to the person you are texting⁵. Swapping between command and communication contexts removes the synchronicity of human conversation, and reduces us to the VR equivalent of ignoring that text from your mother to waste time on the internet.

Now that I've spent three paragraphs explaining in detail how much of a pessimist I am, I can safely concede that a single player experience that you can run in a quiet, private setting isn't an extremely tall order for VR. So it's still worth taking seriously.

Voice commands are pretty easy to set up in Unity⁶ if you're working on a Windows machine. You can create a new script in Unity and include the Windows Phrase Recognizer by adding using UnityEngine.Windows.Speech; to the top.

When trying to build a project with the phrase recognizer you may encounter an error if you don't have Windows' core speech recognition enabled -- you can fix that by opening the "Speech, inking & typing" settings in the Windows Control panel and turning on speech services. Take a look at Microsoft's documentation if you still have trouble setting things up.

Using the speech services opens the door to two types of voice recognition: the phrase recognizer, and the dictation recognizer.

The more basic of the two is the phrase recognizer which can be further broken down into two categories -- a keyword recognizer, and a grammar recognizer. With the phrase recognizer you can set up specific words, or rules (grammars) that the project listens for and acts on as soon as it hears. The phrase recognizer is great because it triggers as soon as it recognizes one of its key phrases and runs really quickly.

Implementing a phrase recognizer is pretty easy. Suppose I had a sphere and wanted it to turn blue when I said 'blue', and 'red' when I said red -- the code might look something like this:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Windows.Speech;

public class PhraseVoiceControl : MonoBehaviour {
    // references to the red and blue materials we want to use -- set in the inspector
    public Material red;
    public Material blue; 

    private KeywordRecognizer keywordRecognizer;
    private string[] keywords;
    private GameObject sphere;

    void Start () {
        // set up a list of keywords we can recognize
        keywords = new string[] { "red", "blue" };
        keywordRecognizer = new KeywordRecognizer(keywords);

        // triggers an event when a phrase is recognized
        keywordRecognizer.OnPhraseRecognized += OnPhraseRecognized;
        keywordRecognizer.Start();
    }

    // event triggered when a phrase is recognized
    void OnPhraseRecognized(PhraseRecognizedEventArgs args){
        // the text of the argument is the word the recognizer thinks you have said
        if(args.text == "red"){
            GetComponent<Renderer>().material = red;
        } else if(args.text == "blue"){
            GetComponent<Renderer>().material = blue;
        }
    }
}

A grammar recognizer is similar but instead of words the phrase recognizer listens for patterns outlined in a formal grammar -- you can learn about setting up a grammar here.

The more complicated system is the dictation recognizer. It transcribes the whole phrase you say into a string that you can use, and waits until you have completed a sentence by looking for a significant pause. This is why the dictation recognizer is a lot slower than the phrase recognizer, so if you have single word specific commands you should always use phrases instead. That said, with the entire text of what is spoken you can start to do interesting stuff with AI by sending the text off to an NLP API⁷.

Getting started with dictation recognition in Unity is very similar, we'll use the same example as above:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Windows.Speech;

public class DictationVoiceControl : MonoBehaviour {
    private DictationRecognizer dictationRecognizer;

    public Material red;
    public Material blue;

    void Start () {
        dictationRecognizer = new DictationRecognizer();

        dictationRecognizer.DictationResult += Sentences;
        dictationRecognizer.Start();
    }

    private void Sentences(string text, ConfidenceLevel confidence){
        // the arguments are the full text of what the recognizer things you said, and how confident it is in that assessment
        if(text.Contains("red")){
            GetComponent<Renderer>().material = red;
        } else if(text.Contains("blue")){
            GetComponent<Renderer>().material = blue;
        }
    }
}

This is not a great use case for the dictation recognizer because we have particular words that we know we want to switch on. However, now that you have the full text of what is spoken it can then be processed by an NLP library or hooked into some kind of bot to generate responses to queries.

While getting started with voice in Unity is pretty easy, it's quite powerful and I've really only been able to scrape the surface here⁸. Natural language processing, and bot writing, are textbook length material though something I am starting to dig into. I would absolutely recommend talking to computer and only worry about it talking back if you haven't built a bot⁹

I still cannot touch type, AMA. ↩
"Fuck this goddamn machine to hell." ↩
Hey, extroverts, good to see you here. ↩
Yeah, I did make an American Psycho joke there. ↩
Or, my nerdy readers, like trying to type your code in vim's command mode. ↩
And a dark hellscape in Unreal. ↩
I decided I wanted to consume APIs instead of adding plugins and went with Microsoft's LUIS over IBM's Watson because I didn't really want to make an IBM account. ↩
I've been really trying to lay off the magnum opus blog posts. ↩
I told myself I wasn't going to make this joke but there it is. ↩