DEV Community: Carlos Ocejo

Do you know what it takes to create a WebRTC video conferencing application?

Carlos Ocejo — Thu, 11 May 2023 02:08:48 +0000

I didn't know either, I had some experience using WebRTC for one-to-one communication, so doing it with more participants seemed easy to me, but I didn't know what I was getting into, because there were several concepts that I didn't know and I had to understand before to be able to start.

My experience of WebRTC vs. video conferencing app

Let's start with the basics What is WebRTC?

Web Real-Time Communication (WebRTC) is an open-source project supported by Google, Mozilla, and Opera among others, and a group of APIs and protocols developed at the World Wide Web Consortium (W3C) and the Internet Engineering Task Force (IETF) that allows us to have real-time communication by integrating video, audio, and data natively through the browser, even on mobile devices.

Web Real Time Communication

How does it work?

The keyword here is “native”, which means that you don't need to install any plugin or software for it to work, all thanks to a set of APIs that make it possible, which are:

getUserMedia (gUM): It is responsible for safely obtaining the audio and video devices, by asking the user for the necessary permissions to share. This API only works on pages served through HTTPS, to take advantage of the security and features it provides, so the connection is secure.
RTCPeerConnection: Allows communication between peers *(users), the connection is made using SRTP, which is a secure variant of RTP. It should be noted that RTP is the network protocol designed to deliver audio and video over IP networks. Therefore, RTCPeerConnection handles media transmission between *peers and provides end-to-end delivery services for real-time data in a secure manner. Another protocol used is RTCP, which provides information about the quality of service provided by RTP.
DataChannel: It helps us to send any type of data like a text chat or file transfer.

Peer-to-peer connection

Signaling

WebRTC cannot create connections alone, a communication channel is needed to exchange information before establishing a connection, this is called a signal channel or signaling service. It is worth mentioning that it is not part of the WebRTC standard, but it is a fundamental piece.

The information that is exchanged is the “offer and response” using SDP (Session Description Protocol) which contains the codec, source address, and audio and video time information.

Peer A initiates the connection, and creates a message type called offer. Then it sends this offer to peer B using the signal channel. Peer B receives the offer on the signal channel and creates a response. Finally it sends it back to peer A using the same signal channel.

Also, the peers must exchange information about the network connection. This is known as an ICE candidate (ICE candidate) and gives us information for the connection such as the IP address and port of each host.

Once each peer knows where the other is, the connection is established directly. Once the connection is started, the server is no longer needed but can be used for participant authentication and session control.

Peer to peer connection using signaling

Suspiciously simple right? But something is not right.

Suspiciously simple

WebRTC uses peer-to-peer connections, where voice, video, and data connections are established directly between browsers. Unfortunately, firewalls and NAT make this difficult, to achieve that there are STUN and TURN servers.

STUN and TURN servers

A STUN server (Session Traversal Utilities for NAT) lets the peer know its public network information and how it can be reached from the Internet.

If the direct connection between the peers fails, or is not possible, a TURN (Traversal Using Relays around NAT) server can retransmit the traffic to the other peer, but this results in higher consumption of resources.

STUN Server

Peer to peer connection via internet

With this, we have the complete flow of how WebRTC works, in this case, the communication is one-to-one, but what happens when we need more participants?

Well, we know that for the connection to be possible each peer must create a connection object, this object contains all the audio, video, and data information, as well as the public network information of the STUN and TURN servers.

When this process is repeated, in each peer an additional object is created for each peer connected, this is called mesh topology.

Mesh topology, each peer must maintain a connection to each connected peer

If the number of peers increases, the browser must maintain a connection with each of the peers, which in turn increases resource consumption and can be problematic for low-end or mobile devices. This is the simplest solution since you don't have to change anything in the infrastructure because all the work is done in the browser, but if you need to support more users, you should take another strategy.

Multipoint Controller Unit (MCU)

The idea behind this is to have a central media server, which is in charge of receiving, decoding, processing, encoding, and sending back the streams to the peers. Therefore, the peers only maintain a single connection with the media server, instead of having one for each peer. So if the number of peers increases, the device resource consumption will not be impacted.

Multipoint Controller Unit (MCU)

With this strategy, the problems of the mesh topology are solved, but a server with enough resources is needed because processing, encoding, and decoding all the streams consume a lot of CPU, and this directly impacts the budget.

Selective Forwarding Unit (SFU)

SFU is similar to MCU, it also uses a media server, the difference is that instead of doing all the processing, the server redirects the streams to the peers so that they do the necessary processing even the server can select whether to send or not a specific stream according to the peer that is going to receive it, for example, the video quality can change depending on the connection speed of the peer, this way the server doesn't need to be so powerful. Therefore, it is a better option since the problems of mesh topology are avoided while taking the advantage of MCU without the high associated costs.

Selective Forwarding Unit (SFU)

We now know that WebRTC works natively in the browser, uses open APIs and protocols to establish secure peer-to-peer connections through STUN and TURN servers and previously coordinated by a signaling server, as well as the different strategies and topologies that can be used.

As you can see, creating a videoconference platform may seem simple, but it starts to get interesting when you apply it to the real world, since you must take into account different aspects and concepts, as well as the advantages or disadvantages of certain topologies according to the use that you want to give it, without leaving costs aside.

I hope this post will serve as a starting point to venture into the development of applications with WebRTC.

3D in React with Three.js

Carlos Ocejo — Sun, 26 Mar 2023 19:44:26 +0000

If you have never worked with any 3D software or three.js before, this article can help you understand the basics of three.js and how to integrate it with React. At first, it may seem a bit overwhelming, but in the end, you will have a base app for playing and experimenting with three.js.

Three.js

Let's start with the basics, three.js is a WebGL engine, ok but what does that mean?

Well, WebGL allows web content to use an API based on OpenGL ES 2.0 to carry out 2D and 3D rendering in an HTML canvas element without using any plug-ins.

The three.js library provides many functions and APIs for drawing 3D scenes in the browser.

This is the minimal structure of an app with three.js.

three.js app setup - threejs.org

Hello cube... no, Loaders!

When we learn a programming language, the first thing we do is a "Hello World!"
the second thing is to add it to our Linkedin profile :P, well what we will do here is make a minimal scene but not with a simple and boring cube, we will use Loaders to load a 3D model in our scene.

So get to work!

Project setup

I'll use this project as a base, it already has typescript, parcel, eslint, emotion and react configured. You can use create-react-app or any other base if you want.

Dependencies

We are going to need to install three.js

yarn add three

npm i three

and since we are going to use typescript, we need to install the types

yarn add -D @types/three

npm i -D @types/three

Integrate Three with React

First of all, React is unaware of changes made to the DOM outside of React, if the DOM nodes are manipulated by another library, React gets confused. To make React happy and avoid conflicts we must prevent the component from updating, using elements that React has no reason to update like an empty div or in our case, a canvas.

Our three.js code uses the dom so we need to wait for the first render, for this we will use a hook we could use useEffect, but this is a good case for useLayoutEffect. If you want, take a look at this excellent article where you can read about the differences between the two: https://kentcdodds.com/blog/useeffect-vs-uselayouteffect

That being said, our component would look like this:

export function App() {

  useLayoutEffect(()=>{
    // three-js code here
  },[])

  return (
    <canvas/>
  )
}

Renderer

This is the main object of Three.js, it receives a scene and a camera and renders (draws) the part of the scene that is inside the camera as a 2d image on the canvas.

Now, we need to import three.js

import * as THREE from 'three';

We will create an instance of WebGLRenderer, for this we need to pass it a canvas, if it is not passed the canvas it will create one and you will have to add it to the document later.

To use the canvas of our component we need a reference to the DOM element and for that, we will use another hook, useRef

export function App() {

  const canvasRef = useRef<HTMLCanvasElement>(null)

  useLayoutEffect(()=>{
    // three-js code here
  },[])

  return (
    <canvas ref={canvasRef}/>
  )
}

Now with the reference to the canvas, we can create the instance

const renderer = new THREE.WebGLRenderer({
    canvas: canvasRef.current as HTMLCanvasElement,
    antialias: true,
    alpha: true,
})

Aside from the canvas, we're passing a couple of additional parameters, antialias which makes sure we don't have weird borders on our objects, and alpha which makes our canvas have a transparent background. So we can set the background with CSS.

Camera

So, we need a camera, we'll use a PerspectiveCamera and set its position to x,y,z coordinates.

const camera = new THREE.PerspectiveCamera(
    45, // fov - field of view
    window.innerWidth / window.innerHeight, // aspect
    0.1, // near
    100 // far
)

camera.position.set(2, 1, 2)

the parameters for PerspectiveCamera are fov, aspect, near and far:

fov is the field of view in degrees
aspect is the aspect ratio of the canvas
near and far represent the space in front of the camera that will be rendered, anything outside of that range will not be rendered.

three.js perspective camera - threejs.org

Scene

Whats next? Create a scene. Basically a scene is a container, everything you want to be drawn needs to be added to the scene.

const scene = new THREE.Scene()

Lights

Our scene requires lights so that the objects can be seen better, we will create two lights: a DirectionalLight and a HemisphereLight.

const directionalLight = new THREE.DirectionalLight(0xffffff, 0.2);
directionalLight.castShadow = true;
directionalLight.position.set(-1, 2, 4)
scene.add(directionalLight);

const ambientLight = new THREE.AmbientLight(0xffffff, 0.7)
scene.add(ambientLight)

Loaders

Now that we have our scene lit, we can start adding objects.
We will use a loader to load a model, in fact, there are different loaders for different formats (.gltf, .obj, .fbx, etc.). In this case, I will use an fbx model (model) but you can use any model with fbx format.

Let's import the FBXLoader

import { FBXLoader } from "three/examples/jsm/loaders/FBXLoader"

FBXLoader receives as parameters the path where the .fbx file is and onLoad, onProgress and onError callbacks.

I will create a 3D object as a container for my model, this is useful to use it as a pivot, in the case the model is not at the origin, so we can center it easily.

In the onLoad we get the object, we can scale, set his position, and add it to the container as a child, in my case the model was a bit up in the Y coordinate. Finally, we add the container to the scene.

const container = new Object3D()
container.position.set(0, 0, 0)

const fbxLoader = new FBXLoader()
fbxLoader.load(
    "/mush.fbx",
    (object) => {
        object.scale.set(0.05, 0.05, 0.05)
        object.position.set(0, -0.3, 0)
        container.add(object)
    },
    (xhr) => {
        console.log((xhr.loaded / xhr.total) * 100 + "% loaded")
    },
    (error) => {
        console.log(error)
    }
)
scene.add(container)

Render and animation

We are missing an important step, we need to tell the renderer that we want to render, so that it can update the next frame, for this we can use setAnimationLoop or requestanimationFrame, for projects with WebXR we must use setAnimationLoop.

renderer.setAnimationLoop(() => {
    renderer.render(scene, camera)
})

Now we can see our object, for example; if we wanted to rotate the object
we can use the container, since the model is a child of the container, we can animate it and the model will follow the movement of the container.

renderer.setAnimationLoop(() => {
    container.rotation.y -= 0.01
    renderer.render(scene, camera)
})

Responsive

We will focus on the canvas occupying 100% of its container, so we add some CSS. Note that I am using emotion.
Do you remember that our canvas has a transparent background? That allows us to set the background color with CSS.

import { css, Global } from "@emotion/react"
...
<>
    <Global
        styles={css`
            html,
            body {
                margin: 0;
                padding: 0;
                height: 100%;
            }
        `}
    />
    <canvas
        css={css`
            width: 100%;
            height: 100%;
            display: block;
            background-image: linear-gradient(
                to top,
                #0a264f,
                #15487a,
                #1b6ea6,
                #1b96d2,
                #16c1fd
            );
        `}
        ref={canvasRef}
    />
</>

It's still not fully responsive, we need to update the size of the renderer and the camera aspect when the window resizes. For this, we add an event, and we also remove the event when the component is unmounted at the end of our hook.

useLayoutEffect(() => {
    ...
    const onResize = () => {
      camera.aspect = window.innerWidth / window.innerHeight
      camera.updateProjectionMatrix()
      renderer.setSize(window.innerWidth, window.innerHeight)
    }

    window.addEventListener("resize", onResize, false)

    return () => {
      window.removeEventListener("resize", onResize)
    }
  }, [])

Controls

We already have a 3d scene, but we can't move through it. So, let's add controls to make it more interactive.

import { OrbitControls } from "three/examples/jsm/controls/OrbitControls"

useLayoutEffect(() => {
    ...
    new OrbitControls(camera, renderer.domElement)

With this, we can zoom, pan, and move around the scene using the mouse.
And voila! We have a 3d scene in react with three.js.

You can play with it here.

Code

If you got lost, I put the complete code.

import { useLayoutEffect, useRef } from "react"
import {
  DirectionalLight,
  Object3D,
  PerspectiveCamera,
  Scene,
  WebGLRenderer,
  AmbientLight,
} from "three"
import { OrbitControls } from "three/examples/jsm/controls/OrbitControls"
import { FBXLoader } from "three/examples/jsm/loaders/FBXLoader"

export function App() {
  const canvasRef = useRef<HTMLCanvasElement>(null)

  useLayoutEffect(() => {
    const renderer = new WebGLRenderer({
      canvas: canvasRef.current as HTMLCanvasElement,
      antialias: true,
      alpha: true,
    })

    const camera = new PerspectiveCamera(
      45, // fov
      window.innerWidth / window.innerHeight, // aspect
      0.1, // near
      100 // far
    )

    camera.position.set(2, 1, 2)

    const scene = new Scene()

    // lights
    const directionalLight = new DirectionalLight(0xffffff, 0.2)
    directionalLight.castShadow = true
    directionalLight.position.set(-1, 2, 4)
    scene.add(directionalLight)

    const ambientLight = new AmbientLight(0xffffff, 0.7)
    scene.add(ambientLight)

    //fbx model
    const container = new Object3D()
    container.position.set(0, 0, 0)

    const fbxLoader = new FBXLoader()
    fbxLoader.load(
      "/mush.fbx",
      (object) => {
        object.scale.set(0.05, 0.05, 0.05)
        object.position.set(0, -0.3, 0)
        container.add(object)
      },
      (xhr) => {
        console.log((xhr.loaded / xhr.total) * 100 + "% loaded")
      },
      (error) => {
        console.log(error)
      }
    )
    scene.add(container)

    renderer.setAnimationLoop(() => {
      //container.rotation.y -= 0.01
      renderer.render(scene, camera)
    })

    renderer.setSize(window.innerWidth, window.innerHeight)

    const onResize = () => {
      camera.aspect = window.innerWidth / window.innerHeight
      camera.updateProjectionMatrix()
      renderer.setSize(window.innerWidth, window.innerHeight)
    }

    window.addEventListener("resize", onResize, false)

    new OrbitControls(camera, renderer.domElement)

    return () => {
      window.removeEventListener("resize", onResize)
    }
  }, [])

  return (
    <>
      <Global
        styles={css`
          html,
          body {
            margin: 0;
            padding: 0;
            height: 100%;
          }
        `}
      />
      <canvas
        css={css`
          width: 100%;
          height: 100%;
          display: block;
          background-image: linear-gradient(
            to top,
            #0a264f,
            #15487a,
            #1b6ea6,
            #1b96d2,
            #16c1fd
          );
        `}
        ref={canvasRef}
      />
    </>
  )
}

If you want to be lazy, you can clone the repo here.

Conclusion

If you got this far, you already have a sample react app with three.js, I recommend checking out the examples on the three.js page, taking a look at the docs, and exploring your own ideas.

DEV Community: Carlos Ocejo

Do you know what it takes to create a WebRTC video conferencing application?

Let's start with the basics What is WebRTC?

How ​​does it work?

Signaling

STUN and TURN servers

Multipoint Controller Unit (MCU)

Selective Forwarding Unit (SFU)

3D in React with Three.js

Three.js

Hello cube... no, Loaders!

Project setup

Dependencies

Integrate Three with React

Renderer

Camera

Scene

Lights

Loaders

Render and animation

Responsive

Controls

Code

Conclusion

How does it work?