Creating a HUD in react-three-fiber

Joe Hsu — Thu, 30 Apr 2020 16:31:02 +0000

Follow along as I try building a virtual reality application using react-three-fiber. See my fist post on building the scene.

After creating a basic scene with react-three-fiber, I wanted to add a HUD for the user.

Single Camera

On my first several attempts, I tried to render multiple cameras, a PerspectiveCamera for the player and an OrthographicCamera for the HUD. After thinking about targeting a virtual-reality context, I realized I needed a spatial UI.

use the mouse to drag the camera and look around

Having the HUD follow the camera.

Instead of sticking the HUD directly to the camera, I used react-spring/three to animate the HUD's group with a spring to bounce to the camera orientation.

Inside of the useFrame hook, I grabbed the camera's position and orientation. Using the set provided by useSpring, I update the HUD's target position and rotation.

Next Steps

HUD positioning

I need to figure out how to get the headset position and place the HUD based on that. Simply using the PerspectiveCamera position doesn't seem to place it HUD based on the headset position, but maybe the camera's position is the play area?

HUD interaction

Now that I have a HUD, I will need to to allow the player to interact with the HUD and figure out a layout.

Hello WebVR with react-three-fiber

Joe Hsu — Thu, 23 Apr 2020 19:06:07 +0000

I have been experimenting with react-three-fiber and also trying to use it to build virtual reality scenes. Starting form somewhere simple, I decided to rebuild the "Hello WebVR" example found in A-Frame.

The first thing that caught me off guard was that after the scene is setup by react-three-fiber, it calls camera.lookAt(0,0,0), even when passing a rotation to <Canvas camera={{ rotation: [x, y, z] }} />. I created a Camera component that takes the camera from useThree() and reset the camera rotation inside a useEffect() react hook.

I created some wrapper components for the basic shapes (Cylinder, Sphere, Box, Plane) and define a flat set of props that would be passed to <mesh /> and its corresponding geometry component. This caused some interesting TypeScript issues and took me a bit to figure out how to take react-three-fiber's types and merge them with additional props.

Using generics, I created a single type I could use for each of the shape components

type MeshShape<T extends new (...args: any) => any, P = {}> = Overwrite<
  ReactThreeFiber.Object3DNode<THREE.Mesh, typeof THREE.Mesh>,
  {
    args?: ConstructorParameters<T>;
  } & P
>;

// Box Props
type IBox = MeshShape<
  typeof THREE.BoxBufferGeometry,
  {
    color?: ReactThreeFiber.Color;
  }
>;

I ported some of the look-controls component from A-Frame to get drag mouse look working, excluding touch events.

My next exploration will be to create a HUD.