Building a realtime React VR app

Esteban Herrera — Fri, 23 Feb 2018 01:30:16 +0000

React VR is a library that allows you to write virtual reality apps for the web using JavaScript and React on top of the WebVR API. This specification is now supported by the latest (or in some cases experimental) versions of browsers such as Chrome, Firefox, and Edge, and you don’t need to have a headset to access VR in a browser.

WebVR Experiments is a site that showcases some projects that show you what’s possible with WebVR. One project that caught my attention was The Musical Forest, made by the awesome people of Google Creative Lab using A-Frame, a web framework for WebVR developed by the Mozilla VR team.

In the Musical Forest, hitting a shape triggers a note, and using WebSockets, people can play music together in realtime. However, due to all the features and technologies used, the app is somewhat complicated (you can find the source code here). So, why not use React VR and Pusher to create a simplified version of it?

Here’s how the React VR/Pusher version looks:

A user can enter a channel identifier in the URL. When a 3D shape is hit, a sound will play and a Pusher event will be published so other users in the same channel can receive the event and the app can play that sound too.

We’ll have a Node.js backend to publish the events, so you should be familiar with JavaScript and React. If you’re not very familiar with some VR concepts or React VR, here’s a guide to get you started.

For reference (or if you just want to try the app), you can find the React VR project here and the Node.js backend here.

Setting up the React VR project

Let’s start by installing (or updating) the React VR CLI tool:

npm install -g react-vr-cli

Next, create a new React VR project:

react-vr init musical-exp-react-vr-pusher

Now go to the directory it created and execute the following command to start the development server:

cd musical-exp-react-vr-pusher
npm start

In your browser, go to http://localhost:8081/vr/. Something like the following will show up:

If you’re using a compatible browser, you should also see the View in VR button to view the app with a headset:

Now let’s start coding our app.

Creating the background

We’re going to use an equirectangular image as the image background. The main characteristic of this type of images is that the width must be exactly twice the height, so open your favorite image editing software and create an image of size 4096x2048 with a gradient or color of your choice:

Create a new folder called images inside the static_assets directory in the root of your app and save your image there. Now open the file index.vr.js and replace the content of the render method with the following:

render() {
  return (
    <View>
      <Pano source={asset('images/background.jpg')} />
    </View>
  );
}

When you reload the page (or if you enable hot reloading), you should see something like this:

Now, to simulate a tree, we’re going to use a Cylinder. In fact, we’ll need a hundred of this to create a forest around the user. In the original Musical Forest, we can find the algorithm to generate the trees around the users in the js/components/background-objects.js file. If we adapt the code into a React component for our project, we can get this:

import React from 'react';
import {
  View,
  Cylinder,
} from 'react-vr';

export default ({trees, perimeter, colors}) => {
  const DEG2RAD = Math.PI / 180;

  return (
    <View>
      {Array.apply(null, {length: trees}).map((obj, index) => {
          const theta = DEG2RAD * (index / trees) * 360;
          const randomSeed = Math.random();
          const treeDistance = randomSeed * 5 + perimeter;
          const treeColor = Math.floor(randomSeed * 3);
          const x = Math.cos(theta) * treeDistance;
          const z = Math.sin(theta) * treeDistance;

          return (
            <Cylinder
              key={index}
              radiusTop={0.3}
              radiusBottom={0.3}
              dimHeight={10}
              segments={10}
              style={{
                color: colors[treeColor],
                opacity: randomSeed,
                transform: [{scaleY : 2 + Math.random()}, {translate: [x, 3, z]},],
              }}
            />
          );
      })}
    </View>
  );
}

This functional component takes three parameters:

trees, which indicates the number of trees the forest will have
perimeter, a value to control how far the trees will be rendered from the user
colors, an array with values of colors for the trees.

Using Array.apply(null, {length: trees}), we can create an array of empty values to which we can apply the map function to render an array of cylinders with random colors, opacities and positions inside a View component.

We can save this code in a file called Forest.js inside a components directory and use it inside of index.vr.js in the following way:

...
import Forest from './components/Forest';

export default class musical_exp_react_vr_pusher extends React.Component {
  render() {
    return (
      <View>
        <Pano source={asset('images/background.jpg')} />

        <Forest trees={100} perimeter={15} colors={['#016549', '#87b926', '#b1c96b']} 
        />
      </View>
    );
  }
};

...

In the browser, you should see something like this:

Great, our background is complete, now let’s add the 3D objects to play the sounds.

Creating the 3D shapes

We are going to have six 3D shapes and each will play six different sounds when clicked. Also, a little animation when the cursor enters and exits the shape will come in handy.

To do that, we’ll need a VrButton, an Animated.View, and a Box, a Cylinder, and a Sphere for the shapes. However, as each shape is going to be different, let’s just encapsulate in a component what is the same. Save the following code in the file components/SoundShape.js:

import React from 'react';
import {
  VrButton,
  Animated,
} from 'react-vr';

export default class SoundShape extends React.Component {

  constructor(props) {
    super(props);
    this.state = {
      bounceValue: new Animated.Value(0),
    };
  }

  animateEnter() {
    Animated.spring(  
      this.state.bounceValue, 
      {
        toValue: 1, 
        friction: 4, 
      }
    ).start(); 
  }

  animateExit() {
    Animated.timing(
      this.state.bounceValue,
      {
        toValue: 0,
        duration: 50,
      }
    ).start();
  }

  render() {
    return (
      <Animated.View
        style={{
          transform: [
            {rotateX: this.state.bounceValue},
          ],
        }}
      >
        <VrButton
          onEnter={()=>this.animateEnter()}
          onExit={()=>this.animateExit()}
        >
          {this.props.children}
        </VrButton>
      </Animated.View>
    );
  }
};

When the cursor enters the button area, Animated.spring will change the value of this.state.bounceValue from 0 to 1 and show a bouncy effect. When the cursor exits the button area, Animated.timing will change the value of this.state.bounceValue from 1 to 0 in 50 milliseconds. For this to work, we wrap the VrButton with an Animated.View component that will change the rotateX transform of the View on each state change.

In index.vr.js, we can add a SpotLight (you can add any other type of light you want or change its properties) and use the SoundShape component to add a cylinder this way:

...
import {
  AppRegistry,
  asset,
  Pano,
  SpotLight,
  View,
  Cylinder,
} from 'react-vr';
import Forest from './components/Forest';
import SoundShape from './components/SoundShape';

export default class musical_exp_react_vr_pusher extends React.Component {
  render() {
    return (
      <View>
        ...

        <SpotLight intensity={1} style={{transform: [{translate: [1, 4, 4]}],}} />

        <SoundShape>
          <Cylinder
            radiusTop={0.2}
            radiusBottom={0.2}
            dimHeight={0.3}
            segments={8}
            lit={true}
            style={{
              color: '#96ff00', 
              transform: [{translate: [-1.5,-0.2,-2]}, {rotateX: 30}],
            }}
          />
        </SoundShape>
      </View>
    );
  }
};
...

Of course, you can change the properties of the shapes or even replace them with 3D models.

Let’s also add a pyramid (which is a cylinder with a zero op radius and four segments):

<SoundShape>
  <Cylinder
    radiusTop={0}
    radiusBottom={0.2}
    dimHeight={0.3}
    segments={4}
    lit={true}
    style={{
      color: '#96de4e',
      transform: [{translate: [-1,-0.5,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>

A cube:

<SoundShape>
  <Box
    dimWidth={0.2}
    dimDepth={0.2}
    dimHeight={0.2}
    lit={true}
    style={{
      color: '#a0da90', 
      transform: [{translate: [-0.5,-0.5,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>

A box:

<SoundShape>
  <Box
    dimWidth={0.4}
    dimDepth={0.2}
    dimHeight={0.2}
    lit={true}
    style={{
      color: '#b7dd60',
      transform: [{translate: [0,-0.5,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>

A sphere:

<SoundShape>
  <Sphere
    radius={0.15}
    widthSegments={20}
    heightSegments={12}
    lit={true}
    style={{
      color: '#cee030',
      transform: [{translate: [0.5,-0.5,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>

And a triangular prism:

<SoundShape>
  <Cylinder
    radiusTop={0.2}
    radiusBottom={0.2}
    dimHeight={0.3}
    segments={3}
    lit={true}
    style={{
      color: '#e6e200',
      transform: [{translate: [1,-0.2,-2]}, {rotateX: 30}],
    }}
  />
</SoundShape>

After you add the necessary imports, save the file and refresh your browser. You should see something like this:

Now let’s add some sounds!

Adding sound

For audio, React VR supports wav, mp3, and ogg files, among others. You can find the complete list here.

You can go to Freesound or other similar sites to get some sound files. Download the ones you like and place them in the directory static_assets/sounds. For this project, we’re going to use six animal sounds, something like a bird, another bird, another bird, a cat, a dog, and a cricket (as a quick note, I had to re-save this file lowering its bitrate so it can be played by React VR).

For our purposes, React VR give us three options to play a sound:

VrSoundEffects
The onClickSound event of a VrButton
A Sound component

However, only the Sound component supports 3D/positional audio so the left and right balance of the sound will change as the listener moves around the scene or turns their head. So let’s add it to our SoundShape component along with an onClick event to the VrButton:

...
import {
  ...
  Sound,
} from 'react-vr';

export default class SoundShape extends React.Component {
  ...
  render() {
    return (
      <Animated.View
        ...
      >
        <VrButton
          onClick={() => this.props.onClick()}
          ...
        >
          ...
        </VrButton>
        <Sound playerState={this.props.playerState} source={this.props.sound} />
      </Animated.View>
    );
  }
}

We’re going to use a MediaPlayerState to control the playing of the sound. Both will be passed as properties of the component.

This way, let’s define an array with this information in index.vr.js:

...
import {
  ...
  MediaPlayerState,
} from 'react-vr';
...

export default class musical_exp_react_vr_pusher extends React.Component {

  constructor(props) {
    super(props);

        this.config = [
          {sound: asset('sounds/bird.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/bird2.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/bird3.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/cat.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/cricket.wav'), playerState: new MediaPlayerState({})},
          {sound: asset('sounds/dog.wav'), playerState: new MediaPlayerState({})},
        ];
  }

  ...
}

And a method to play a sound using the MediaPlayerState object when the right index is passed:


...

export default class musical_exp_react_vr_pusher extends React.Component {

  ...

  onShapeClicked(index) {
    this.config[index].playerState.play();
  }

  ...
}

Now, we only need to pass all this information to our SoundShape components. So let’s group our shapes in an array and use a map function to generate the components:

...

export default class musical_exp_react_vr_pusher extends React.Component {

  ...

  render() {
    const shapes = [
      <Cylinder
        ...
      />,
      <Cylinder
        ...
      />,
      <Box
        ...
      />,
      <Box
        ...
      />,
      <Sphere
        ...
      />,
      <Cylinder
        ...
      />
    ];

    return (
      <View>
        ...

        {shapes.map((shape, index) => {
          return (       
            <SoundShape 
              onClick={() => this.onShapeClicked(index)} 
              sound={this.config[index].sound} 
              playerState={this.config[index].playerState}>
                {shape}
            </SoundShape>
          );
        })}

      </View>
    );
  }

  ...
}

If you restart your browser and try, you should hear the sounds as you click on the shapes.

Now let’s add to our app multi-user support in realtime with Pusher.

Setting up Pusher

Create a free account at https://pusher.com/signup.
When you create an app, you'll be asked to enter some configuration options:

Enter a name, choose React as your front-end tech, and Node.js as the back-end tech. This will give you some sample code to get you started:

But don't worry, this won't lock you into this specific set of technologies as you can always change them. With Pusher, you can use any combination of libraries.

Next, copy your cluster ID (next to the app title, in this example mt1), App ID, Key, and Secret information as we'll need them next. You can also find them in the App Keys tab.

Publishing the event

React VR acts as a Web Worker (you can know more about React VR architecture in this video) so we need to include Pusher’s worker script in index.vr.js this way:

...
importScripts('https://js.pusher.com/4.1/pusher.worker.min.js');

export default class musical_exp_react_vr_pusher extends React.Component {
  ...
}

We have two requirements that need to be taken care of. First, we need to be able to pass an identifier through the URL (like http://localhost:8081/vr/?channel=1234) so users can choose which channel they want to be in and share it with their friends.

To address this, we need to read the URL. Luckily, React VR comes with the native module Location, which makes available to the React context the properties of the object window.location.

Next, we need to make a call to a server that will publish the Pusher event so all the connected clients can also play the event. However, we don’t want the client that broadcasts the event to receive it too, because in that case, the sound will be played twice, and there’s no point in waiting to receive the event to play the sound when you can play it immediately when the user clicks the shape.

Each Pusher connection is assigned a unique socket ID. To exclude recipients from receiving events in Pusher, we just need to pass to the server the socket ID of the client we want to be excluded a socket_id when this is triggering an event. (You can find more information here.)

This way, adapting a little bit a function (getParameterByName) to read the parameters of the URL, and saving the socketId when a successful connection is made to Pusher, we can address both requirements with this:

...
import {
  ...
  NativeModules,
} from 'react-vr';
...
const Location = NativeModules.Location;

export default class musical_exp_react_vr_pusher extends React.Component {
  componentWillMount() {
    const pusher = new Pusher('<INSERT_PUSHER_APP_KEY>', {
      cluster: '<INSERT_PUSHER_APP_CLUSTER>',
      encrypted: true,
    });
    this.socketId = null;
    pusher.connection.bind('connected', () => {
      this.socketId = pusher.connection.socket_id;
    });
    this.channelName = 'channel-' + this.getChannelId();
    const channel = pusher.subscribe(this.channelName);
    channel.bind('sound_played',  (data) => {
      this.config[data.index].playerState.play();
    });
  }

  getChannelId() {
    let channel = this.getParameterByName('channel', Location.href);
    if(!channel) {
      channel = 0;
    }

    return channel;
  }

  getParameterByName(name, url) {
    const regex = new RegExp("[?&]" + name + "(=([^&#]*)|&|#|$)");
    const results = regex.exec(url);
    if (!results) return null;
    if (!results[2]) return '';
    return decodeURIComponent(results[2].replace(/\+/g, " "));
  }

  ...
}

If there isn’t a channel parameter in the URL, by default we assign the ID 0. This ID will be added to the Pusher channel to make it unique.

Finally, we just need to call an endpoint on the server side that will publish the event, passing the socket ID of the client and the channel where we’ll be publishing events:

...
export default class musical_exp_react_vr_pusher extends React.Component {
  ...
  onShapeClicked(index) {
    this.config[index].playerState.play();
    fetch('http://<INSERT_YOUR_SERVER_URL>/pusher/trigger', {
      method: 'POST',
      headers: {
        'Accept': 'application/json',
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        index: index,
        socketId: this.socketId,
        channelName: this.channelName,
      })
    });
  }
  ...
}

And that’s all the code of the React part. Now let’s take a look at the server.

Creating the Node.js backend

Execute the following command to generate a package.json file:

npm init -y

Add the following dependencies:

npm install --save body-parser express pusher

And save the following code in a file:

const express = require('express');
const bodyParser = require('body-parser');
const Pusher = require('pusher');

const app = express();
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
/*
  The following headers are needed because the development server of React VR
  is started on a different port than this server. 
  When the final project is published, you may not need this middleware
*/
app.use((req, res, next) => {
  res.header("Access-Control-Allow-Origin", "*")
  res.header("Access-Control-Allow-Headers", 
             "Origin, X-Requested-With, Content-Type, Accept")
  next();
});

const pusher = new Pusher({
  appId: '<INSERT_PUSHER_APP_ID>',
  key: '<INSERT_PUSHER_APP_KEY>',
  secret: '<INSERT_PUSHER_APP_SECRET>',
  cluster: '<INSERT_PUSHER_APP_CLUSTER>',
  encrypted: true,
});

app.post('/pusher/trigger', function(req, res) {
  pusher.trigger(req.body.channelName, 
                 'sound_played', 
                 { index: req.body.index },
                 req.body.socketId );
  res.send('ok');
});

const port = process.env.PORT || 5000;
app.listen(port, () => console.log(`Running on port ${port}`));

As you can see, here we set up an Express server, the Pusher object, and the route /pusher/trigger, which just triggers an event with the index of the sound to be played and the socketID to exclude the recipient of the event.

And we’re done. Let’s test it.

Testing

Execute the Node.js backend with:

node server.js

Update your server URL in index.vr.js (with your IP instead of localhost) and enter in your browser an address like http://localhost:8081/vr/?channel=1234 in two browser windows. When you click on a shape, you should hear the sound played twice (of course, it’s more fun doing this with a friend in another computer):

Conclusion

React VR is a great library to create virtual reality experiences in an easy way, especially if you already know React/React Native. Pair it with Pusher and you’ll have powerful tools to program the next generation of web applications.

You can build a production release of this project to deploy it in any web server by following the steps on this page.

Also, you can customize this code by changing the colors, the shapes, the sounds, or add more functionality from the original Musical Forest.

Finally, remember that you can find the code of the app in this GitHub repository.

This post first appeared on Pusher's blog.

DEV Community: Esteban Herrera