DEV Community: Hermes Frangoudis

Building a 1-to-many iOS video app with Agora 4.x SDK Preview

Hermes Frangoudis — Fri, 17 Sep 2021 15:35:09 +0000

Ever wonder how to build a video chat app? This is a guide to how to quickly and easily create a video chat app on iOS that can support multiple participants using the Agora Video SDK.

Requirements

Xcode 10.0+
A physical iOS device. The iOS simulator lacks camera functionality.
CocoaPods (If you don’t have CocoaPods installed already, you can find instructions here).
An Agora account (see How To Get Started with Agora)
An understanding of how to build iOS layouts with a Storyboard. If you need a refresher, there’s a great tutorial here.

Setting up the Agora library with CocoaPods

Create a new Xcode project (or use an existing one).
In Terminal, navigate to the root directory of your project and run pod init to initialize CocoaPods.
Open the Podfile that was created and add the following code to import the Agora library:

target 'Your App' do 
  pod 'AgoraRtcEngine_iOS', '3.4.0' 
end

Run pod install in Terminal to install the library.
From now on, open YourApp.xcworkspace to edit and run your app.

Add Camera and Microphone permissions

In order to use the microphone and camera, we’ll need to ask the user for permission to do so. In your Info.plist add the following keys:

Privacy - Microphone Usage Description 
Privacy - Camera Usage Description

Make sure you add a value for each. These values are user-facing, and will be displayed when the app asks for these permissions from the user.

Setting up the scene

In our Main.storyboard we'll need to add the views Agora will use to display the video feeds. For our demo, we'll be using a single large view to display our local feed, and a collection view to show an arbitrary number of remote users, but feel free to adjust as necessary for your own needs.

The local view is in green, and the remote view template is in red, for ease of identification. Add a View object for the local stream, a UIButton to mute and hang up the call, and a UICollectionView to hold the remote streams. Your UICollectionViewCells can be as simple as a single view to hold the stream — in the example above, I’ve added an overlay to show the remote user’s name if we know it.

Make sure you hook up the views in your main View Controller, and set the View Controller as the UICollectionView’s delegate and dataSource:

class AgoraVideoViewController: UIViewController, UICollectionViewDelegate, UICollectionViewDataSource {

    @IBOutlet weak var collectionView: UICollectionView!
    @IBOutlet weak var localVideoView: UIView!
    @IBOutlet weak var muteButton: UIButton!
    @IBOutlet weak var hangUpButton: UIButton!

And connect up your custom collection view cell:

class VideoCollectionViewCell: UICollectionViewCell {
    @IBOutlet weak var videoView: UIView!
}

Tip: If you want to add overlays to your video streams, make sure you don’t add them as subviews of the view objects you’re going to use as video screens. The video canvas will be drawn on top of them. Add them as sibling views instead.

Initialize the Agora engine

In order to use the Agora engine, we need to create an instance of AgoraRtcEngineKit with our app ID.

First, we will need to retrieve our app ID by going to the Agora Dashboard. If you haven’t created an Agora project yet, do so now by clicking “New Project.”

Once you have a project, click the “Edit” button (or open the Project Management pane) to view that project’s details. Copy the app ID and add it to your project. If you enabled the App Certificate, you’ll also need a Token to join channels — you can generate a temporary one by clicking “Generate Temp Token.” You can also read our tutorial on generating your own tokens here.

Note: Any Agora projects made for production should require a token from the client to connect. See Connecting to Agora with Tokens — Using Swift for more information on how to work with tokens in an iOS project.

The first call you make to Agora must be to initialize a shared Agora engine. We’ll make sure to do this by creating a helper function that initializes the engine if we haven’t done so yet, and just returns it if we have. That way we can just call it whenever we need a reference to the engine without having to worry about who does it first.

import AgoraRtcKit

let appID = "YourAppIDHere"
var agoraKit: AgoraRtcEngineKit?
let tempToken: String? = nil //If you have a token, put it here.
var userID: UInt = 0 //This tells Agora to generate an id for you. If you have unique user IDs already, you can use those.

...

private func getAgoraEngine() -> AgoraRtcEngineKit {
    if agoraKit == nil {
        agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: appID, delegate: self)
    }
    return agoraKit!
}

Tip: This is a quick way to ensure the engine is only initialized once when you need it, but for a larger app you may want to consider wrapping it in a Singleton instead, so you can easily access it from anywhere.

We’ll also need to implement the AgoraRtcEngineDelegate protocol so we can respond to relevant callbacks:

extension AgoraVideoViewController: AgoraRtcEngineDelegate {

}

Enable Video

The next step is to tell Agora we want video enabled, and to tell it where to put the local video stream. We can then call this function from our viewDidLoad().

func setUpVideo() {
    getAgoraEngine().enableVideo()

    let videoCanvas = AgoraRtcVideoCanvas()
    videoCanvas.uid = userID
    videoCanvas.view = localVideoView
    videoCanvas.renderMode = .fit
    getAgoraEngine().setupLocalVideo(videoCanvas)
}

Tip: If you want to customize how the video is displayed, this is a good place to configure the video profile.

Join a channel

Once the engine is initialized, joining a call is as easy as calling joinChannel() on the Agora engine.

var channelName = "default"

func joinChannel() {
  localVideoView.isHidden = false

  let engine = getAgoraEngine()
  engine.startPreview()
  engine.setClientRole(.broadcaster)

  let joinCHRtn = engine.joinChannel(byToken: tempToken, channelId: channelName, info: nil, uid: userID) { [weak self] (sid, uid, elapsed) in
    self?.userID = uid
  }
}

Note: Any Agora projects made for production should require a token from the client to connect. See Connecting to Agora with Tokens — Using Swift for more information on how to work with tokens in an iOS project.

Setting up Remote Video

Now is the time to put our UICollectionView to good use. We’ll keep a list of remote user IDs, and for each one, set up a remote video canvas within our collection.

var remoteUserIDs: [UInt] = []

func collectionView(_ collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {
    return remoteUserIDs.count
}

func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
    let cell = collectionView.dequeueReusableCell(withReuseIdentifier: "videoCell", for: indexPath)

    let remoteID = remoteUserIDs[indexPath.row]
    if let videoCell = cell as? VideoCollectionViewCell {
        let videoCanvas = AgoraRtcVideoCanvas()
        videoCanvas.uid = remoteID
        videoCanvas.view = videoCell.videoView
        videoCanvas.renderMode = .fit
        getAgoraEngine().setupRemoteVideo(videoCanvas)
    }

    return cell
}

Tip: Remember to set your custom cell’s reuse identifier in your Main.Storyboard!

To get this list of userIDs (and maintain it), we’ll utilize the rtcEngine(didJoinedOfUid:) and rtcEngine(didOfflineOfUid:) callbacks. Inside your AgoraRtcEngineDelegate extension, add the following functions:

extension AgoraVideoViewController: AgoraRtcEngineDelegate {
    func rtcEngine(_ engine: AgoraRtcEngineKit, didJoinedOfUid uid: UInt, elapsed: Int) {
        remoteUserIDs.append(uid)
        collectionView.reloadData()
    }

    func rtcEngine(_ engine: AgoraRtcEngineKit, didOfflineOfUid uid: UInt, reason: AgoraUserOfflineReason) {
        if let index = remoteUserIDs.firstIndex(where: { $0 == uid }) {
            remoteUserIDs.remove(at: index)
            collectionView.reloadData()
        }
    }
}

And with that, you have a working video chat app. Beware of audio feedback if testing on multiple devices at once.

Polish

There are a few more pieces that we should add in to make our app a little nicer. For one, our buttons don’t do anything. Lets’s fix that first. Enabling the mute button is a simple call to muteLocalAudioStream():

var muted = false {
    didSet {
        if muted {
            muteButton.setTitle("Unmute", for: .normal)
        } else {
            muteButton.setTitle("Mute", for: .normal)
        }
    }
}

@IBAction func didToggleMute(_ sender: Any) {
    muted.toggle()
    getAgoraEngine().muteLocalAudioStream(muted)
}

We can also hang up by calling leaveChannel():

@IBAction func didTapHangUp(_ sender: Any) {
    leaveChannel()
}

func leaveChannel() {
    getAgoraEngine().leaveChannel(nil)
    localVideoView.isHidden = true
    remoteUserIDs.removeAll()
    collectionView.reloadData()
}

Tip: If you don’t hide the local video view (or pop the view controller) you’ll end up with a static view of the last frame recorded.

And we’re done! You can find the completed app here. Thanks for following along, and happy coding!

Other Resources

For more information about building applications using Agora.io SDKs, take a look at Agora Video Call Quickstart Guide and Agora API Reference.

I also invite you to join the Agora.io Developer Slack community.

How to Build a VR Video Chat App Using Unity’s XR Framework

Hermes Frangoudis — Fri, 28 May 2021 15:35:47 +0000

Virtual reality (VR) has seen many recent gains in popularity, and as headset makers work to keep up with hardware demand, developers are working to keep up with users’ need for engaging content. VR isn’t the only technology that’s seen an increase in popularity. In today’s professional world everyone is using live video streaming to connect and collaborate. This creates an interesting opportunity for developers to develop applications that can leverage Virtual Reality along with video streaming to remove all barriers of distance and create an immersive telepresence experience.

VR developers face two unique problems:

How do you make VR more inclusive and allow users to share their POV with people who aren’t using VR headsets?
How do you bring non-VR participants into a VR environment?

Most VR headsets allow the user to mirror the user’s POV to a nearby screen using a casting technology (Miracast or Chromecast). This creates a limitation of requiring one physical screen per VR headset, and the headset and screen must be in the same room. This type of streaming feels very old-school given that most users today expect to have the freedom to stream video to others whom are remotely located.

In this guide, I’m going to walk through building a Virtual Reality application that allows users to live stream their VR perspective. We’ll also add the ability to have non-VR users live stream themselves into the virtual environment using their web browser.

We’ll build this entire project from within the Unity Editor without writting any code.

For this project, I will use an HTC Vive Focus Plus because it allows me to build using Unity’s XR framework, which makes it relatively easy to setup a VR environment. And Vive builds to an Android target, so I can use the Agora Video for Unity SDK to add live video streaming to the experience.

Project Overview

This project will consist of three parts. The first part will walk through how to set up the project, implementing the Vive packages along with the Agora SDK and Virtual Camera prefab.

The second part will walk through creating the scene, including setting up the XR Rig with controllers, adding the 3D environment, creating the UI, and implementing the Agora Virtual Camera prefab.

The third section will show how to use a live streaming web app to test the video streaming between VR and non-VR users.

Part 1: Build a Unity XR app

Set Up the Unity Project
Enable teh XR Framework and HTC Modules
Download Agora Video SDK and Prefab

Part 2: Create the Scene

Set Up the 3D Environment and the XR Camera Rig
Add the XR Controllers
Add the Perspective Camera and the Render Texture
Add the UI buttons and Remote Video
Implement the Agora Prefab

Part 3: Testing VR to Web Streams

Set Up a Basic Live Streaming Web app
Test the Streams Between VR and the Web

Prerequisites

Unity Editor
VR headset compatible with Unity’s XR Framework (I’m using the HTC Vive Focus Plus)
An understanding of the Unity Editor and Game Objects
Basic understanding of Unity’s XR Framework
An Agora developer account (see How to Get Started with Agora)
An understanding of HTML, CSS, and JS (minimal knowledge needed)
A simple web server (I like to use Live Server)

Note: While no Unity or web development knowledge is needed to follow along, certain basic concepts from the prerequisites won’t be explained in detail.

Part 1: Build a Unity XR app

The first part of the project will build a Unity app using the XR Framework, walking through how to add the Vive registry to the project, download and install the Vive plug-ins, install the Agora Video for Unity Plug-in, and implement the Agora plug-in using a drag-and-drop prefab.

Set Up the Unity Project

Start by creating a Unity project using the 3D Template. For this demo, I’m using Unity 2019.4.18f1 LTS. If you wish to use Unity 2020, then you will need to use the Wave 4.0 SDK, which is currently in beta.

Note: At the time of writing, I had access to the beta SDK for the new HTC headset. I chose to use the Wave 3.0 SDK. This project can be set up in exactly the same way using the Wave 4.0 SDK.

Enable the XR Framework and HTC Modules

Once the new project has loaded in Unity, open the Project Settings and navigate to the Package Manager tab. In the Scoped Registries list, click the plus sign and add the Vive registry details.

Once the Vive “scopedRegistries” object and its keys have been added, you’ll see the various loaders importing the files. Next open Window > Package Manager and select Packages: My Registries. You will see the VIVE Wave. If no package is shown, click Refresh at the bottom-left corner.

Click through and install each of the Vive packages. Once the packages finish installing, import the PureUnity and XR samples from the Vive Wave XR Plug-in and the Samples from the Essense Wave XR Plug-in into the project.

After the packages have finished installing and the sample has finished importing, the WaveXRPlayerSettingsConfigDialog window will appear. HTC recommends to Accept All to apply the recommended Player Settings.

Next, open the Project Settings, click in the XR Plug-in Management section, and make sure Wave XR is selected.

Now that we have the Wave SDK configured, we need to add Unity’s XR Interaction Toolkit package. This will allow us to use Unity’s XR components for making it possible to interact with Unity inputs or other elements in the scene. In the Package Manager, click the Advanced button (to the left of the search input) and enable the option to “Show preview packages”. Once the preview packages are visible, scroll down to theXR Interaction Toolkit and click install.

Note: If you are using Unity 2018, you will need to configure the input Manager using the presets provided by HTC. The inputs can be defined manually, or you can download this InputManager **preset**. Once you've downloaded the preset, drag it into your Unity Assets, navigate to the InputManager tab in the Project Settings, and apply the presets.

InputManager **preset**from HTC Wave Documentation

When working with the Wave XR Plug-in, you can change the quality level by using QualitySettings.SetQualityLevel. HTC recommends setting the Anti Aliasing levels to 4x Multi Sampling in all quality levels. You can download this QualitySettings **preset** from HTC’s Samples documentation page.

For more information about Input and Quality Settings, see the HTC Wave Documentation.

Download the Agora Video SDK and Prefab

Open the Unity Asset store, navigate to the Agora Video SDK for Unity page, and download the plug-in. Once the plug-in is downloaded, import it into your project.

Note: If you are using Unity 2020, the Asset store is accessible through the web browser. Once you import the asset into Unity, it will import through the Package Manager UI.

The last step in the setup process is to download the Agora Virtual Camera Prefab package and import it into the project.

AgoraIO-Community/Unity-Virtual-Camera-Prefab
*An open source prefab that makes it easy to live stream from a Unity virtual camera into an Agora stream. Drag the…*github.com

When you import the Agora Virtual Camera Prefab package you’ll see that it contains a few scripts, a prefab, and a renderTexture. The two main scripts to note are AgoraInterface.cs (which contains a basic implementation of the Agora Video SDK) and AgoraVirtualCamera.cs (which implements and extends the AgoraInterface specifically to handle the virtual camera stream). The tools folder contains a few helper scripts: one for logging, another to request camera/mic permissions, and one to handle token requests.

Also included in the package is the AgoraVirtualCamera.prefab, an empty GameObject with the AgoraVirtualCamera.cs attached to it. This will make it easy for us to configure the Agora settings directly from the Unity Editor without having to write any code. The last file in the list is the AgoraVirtualCamRT.renderTexture, which we’ll use for rendering the virtual camera stream. The tools folder contains the scripts Logger.cs, PermissionHelper.cs, and RequetstToken.cs.

We are now done with installing all of our dependencies and can move on to building our XR video streaming app.

Part 2 Create the Scene

In this section, we will walk through how to set up our 3D environment using a 3D model, create an XR Camera Rig Game Object, create the buttons and other UI elements, and implement the Agora VR Prefab.

Set Up the 3D Environment and the XR Camera Rig

Create a scene or open the sample scene and import the environment model. You can download the model below from Sketchfab.

3D model hosted on Sketchfab: https://skfb.ly/onQoO

Once the model is imported into the project, drag it into the Scene Hierarchy. This will scale and place the model at (0,0,0) with the correct orientation.

You may have noticed that the scene looks really dark. Let’s fix that by generating lighting. Select Window > Rendering > Lighting Settings and then select the Auto Generate option at the bottom of the window and watch as Unity adjusts and updates the Lighting settings. The lit scene should look like this:

Auto Generate the lights within Lighting Settings

Next we’ll add a room scale XR rig to the scene. Start by deleting the Main Camera Game Object from the scene, and then right-click in the Scene Hierarchy, navigate to the XR options, and click “Room Scale XR Rig (Action Based)”.

Add the XR Controllers

The last step for setting up our XR Rig is to set the HTC prefabs for theLeftHand Controller and RightHand Controller Game Objects. To do this, we’ll need to import the HTC Essence Controller package. Open the Project Settings, navigate to the Wave XR > Essence tab, and import the “Controller” package.

Once Unity has finished importing the “Controller” package, in the Assets panel navigate to the Wave > Essence > Prefabs folder. The Prefabs folder contains prefabs for the Left and Right Controllers.

Select the LeftController Game Object from the Scene Hierarchy, and drag the WaveLeftController prefab into the Model Prefab input for the “XR Controller (Device Based)” script that is attached to the Game Object.

Next, navigate to the Wave > Essence > Scripts folder and drag the CustomizeWaveController script onto the LeftController Game Object.

Repeat these steps for the RightController Game Object, adding the WaveRightController prefab and the CustomizeWaveController script.

That’s it for setting up the controllers to automatically show the appropriate 3D model for the controllers associated with the HMD and the controllers the end user is running.

One thing to keep in mind: Since this is VR, there will be a desire to add some 3D models for the user to interact with. If you plan to add XR Grab Interactable models to the scene, add 3D cubes with collider boxes just below and whatever stationary or solid Game Objects you’d have in your virtual environment, to avoid the XR Grab Interactable models falling through these 3D environment objects when the scene loads. Later in this guides, we’ll add a floor with a collider and teleportation system.

Note: XR Grab Interactable concepts are beyond the scope of this guide, so I won’t go into detail, but for more information take a look at the Unity Guide on using interactables with the XR Interaction toolkit.

Add the Perspective Camera and the Render Texture

The final modification we will make to our XR Rig is to add a Camera Game Object with a Render Texture, as a child of the Main Camera. This will allow us to stream the VR user’s perspective into the channel. For this demo, I named the Camera Game Object as PerspectiveCamera and added the AgoraVirtualCamRT (from the Agora Virtual Camera package) as the Camera’s Render Texture.

Add the UI buttons and Remote Video

Now that the XR Rig is set up and the scene is properly lit, we are ready to start adding our UI elements.

We need to add a few different UI game objects: a join button, a leave button, a button to toggle the mic, a button to quit the app, and a text box to display our console logs in the Virtual Reality scene.

Untiy’s XR Interaction framework has XR-enabled UICanvas Game Objects that we can use to quickly place our buttons. Right click in the Scene Hierarchy, and in XR select the UI Canvas.

We’ll repeat this process two more times so at the end there will be three canvas Game Objects in our scene.

The first canvas will be used for our console logs. We can rename this Game Object to DebugLoggerCanvas. Then set the width to 1000, the height to 500, and scale to (0.025,0.025,0.025) so it fits within our 3D environment. Add a Panel Game Object as a child of the DebugLoggerText, and then add a Text Game Object as a child of the Panel, and name it DebugLoggerText. The DebugLoggerText will display the output from the logger. We want the logs to be legible, so we’ll set DebugLoggerText width to 975 and height to 475. Last, position the DebugLoggerCanvas in the scene. I’ve chosen to place it on the wall with the four window panes.

The next UI Canvas will contain the Join, Leave, and Microphone buttons. Right-click with in the Scene Hierarchy, add another XR UI Canvas to the scene, rename it AgoraBtrnCanvas, set the width to 200, the height to 100, and the scale to (0.075,0.075,0.075). Add three Button Game Objects as children of the AgoraBtnCanvas. Scale and position the buttons in the canvas however you want. Below is how I positioned the buttons on the shelves:

The third UI Canvas will contain the Quit button to allow the users to exit our application. Right-click in the Scene Hierarchy, add another XR UI Canvas to the scene, rename it AgoraBtrnCanvas, set the width to 100, the height to 50, and the scale to (0.075,0.075,0.075). Add a Button Game Object as a child of the XR UI Canvas. Scale and position the button in the canvas however you want. Below is how I positioned the button:

This is a pretty big space, so let’s add a teleportation system so the user can move around in the virtual space. To do this, select the XR Rig in the Scene Hierarchy and add the Teleportation Provider script and the Locomotion System script to the Game Object. Make sure to set the script references to the XR Rig. While it’s not required, setting the reference helps reduce the load time. The last component to enabling teleportation is to add a teleportation area and scale it as large as the floor. The teleportation system allows the user to point and click to move anywhere on the floor plane.

The last two Game Objects we need to add to the scene are the Remote Video Root, and the Screen Video Root. Add an empty Game Object and rename it RemoteVideoRoot. This will be the root node to which new video planes will spawn as remote users connect to the channel. I’ve chosen to place the RemoteVideoRoot on the wall of just windows.

Last, add an empty Game Object and rename it RemoteScreenRoot. This will be the root node to which the video plane will spawn whenever the screen is shared into the channel. I’ve chosen to place the RemoteScreenRoot on the glass wall opposite the RemoteVideoRoot.

Implement the Agora Prefab

Now that we have all of our environment and UI elements added to the scene, we are ready to drag the Agora Virtual Camera Prefab onto the Scene Hierarchy.

First, we’ll add our Agora AppID and either a temp token or the URL for a token server. The Agora prefab assumes the token server returns the same JSON response as the Agora Token Server in Golang guide.

We also need to select a channel name. For this example I will use AGORAVIVE. It’s worth noting that for later when we connect from the web.

Note: For users to be correctly matched, the AppID and Channel name must match exactly because channel names within Agora’s platform are case-sensitive.

The next steps are as simple as they sound. Drag the various Game Objects that we just created into the Prefab input fields.

There is one last step before we can test what we’ve built. Add an On Click handler to the Join, Leave, Microphone, and Quit buttons. For each of these, we’ll use the Agora Virtual Camera prefab as the On Click reference and select the corresponding function for each button. For example, the JoinBtn On Click would call AgoraVirtualCamera > JoinChannel.

At this point, we can test the work we’ve done by building the project to our VR HMD. If you are using a Vive headset, the build settings should look like this:

Make sure that the Platform is set to Android and that the Run Device is visible.

After you initiate the build, the project will take a few minutes to build and deploy to the device. Once the app is deployed, you’ll see a loader screen before you are dropped into the virtual environment.

Click the Join button to join the channel. Once you have joined the channel, the microphone button will turn green and some logs will appear in the debug panel. When you click the Mic button it will turn red to indicate that the mic is muted.

Part 3: Testing VR to Web Streams

In the last part of this guide we’ll use a live streaming web app to test the video streaming between VR and non-VR users.

Set Up a Basic Live Streaming Web App

The Agora platform makes it easy for developers to build a live video streaming application on many different platforms. Using the Agora Web SDK, we can build a group video streaming web app that allows users to join as either active participants or passive audience members. Active participants broadcast their camera stream into the channel and are visible in the VR scene. Users can also join as passive audience members and view the video streams from the VR headset along with the streams of the active participants.

In the interest of brevity, I won’t dive into the minutiae of building a live streaming web app in this guide, but you can check out my guide How To: Build a Live Broadcasting Web App.

For this example, we’ll use a prebuilt example similar to the web app built in my guide How To: Build a Live Broadcasting Web App. If you are interested in running the web client locally or customizing the code, you can download the source code.

Test the Streams Between VR and the Web

Use the links below to join as either an active or a passive participant.

*Active Participant*: https://agoraio-community.github.io/agora-advanced-web-broadcast-demo/broadcast.html

*Passive Participant:* https://agoraio-community.github.io/agora-advanced-web-broadcast-demo/index.html

Once you’ve launched the web view and connected to the channel, launch the VR app and click the Join button. Once the VR headset joins the channel, you will be able to see the VR user’s perspective in the web browser.

Done!

Wow, that was intense! Thanks for following and coding along with me. Below is a link to the completed project. Feel free to fork and make pull requests with any feature enhancements. Now it’s your turn to take this knowledge and go build something cool.

AgoraIO-Community/Unity-Virtual-Camera-Prefab
*An open source prefab that makes it easy to live stream from a Unity virtual camera into an Agora stream. Drag the…*github.com

AgoraIO-Community/agora-advanced-web-broadcast-demo
*A demo project using the Agora Web RTC and RTM SDKs, leveraging advance RTC features like dual stream with stream…*github.com

Other Resources

For more information about the Agora Video SDK, see the Agora.io Video Web SDK API Reference or Agora.io Unity SDK API Reference. For more information about the Agora.io RTM SDK, see the Agora.io RTM Web SDK API Reference.

For more information about the Vive Wave SDK, see the Vive Wave Unity Reference.

For technical support with Agora, I invite you to join the Agora Developer Slack community.

How to Build a Token Server for Agora Applications using GoLang

Hermes Frangoudis — Mon, 31 Aug 2020 16:26:19 +0000

Security within video chat applications is a hot topic at the moment. As remote working and virtual events become more prolific, the need for security will increase.

Within the Agora Platform, one layer of security comes in the form of token authentication. A token, for those of you that don’t know, is a dynamic key that is generated using a set of given inputs. Agora’s Platform uses tokens to authenticate users.

Agora offers token security for both its RTC and RTM SDKs. This guide will explain how to build a simple microservice using Golang and the Gin framework to generate an Agora RTC and RTM tokens.

Pre Requisites

A basic understanding of Golang (minimal knowledge needed)
An understanding of how web servers function (minimal knowledge needed)
An Agora Developer Account (see: How To Get Started with Agora)

Project Setup

To start, let’s open our terminal and create a new folder for our project and cd into it.

mkdir agora-token-server
cd agora-token-server

Now that the project has been created, let’s initialize the project’s Go module.

go mod init agora-token-server

Lastly, we’ll use go get to add our Gin and Agora dependencies.

go get github.com/gin-gonic/gin
go get github.com/AgoraIO-Community/go-tokenbuilder

Build the Gin web server

Now that the project is set up, open the folder in your favorite code editor and create the main.go file.

Within the main.go we’ll start by declaring our package and adding the main function.

package main

func main() {

}

Next we’ll import the Gin framework, create our Gin app, set up a simple GET endpoint and set it to listen and serve on localhost port 8080. For the simple endpoint, we’ll set it to take the request context and return a JSON response with a 200 status header.

package main

import (
  "github.com/gin-gonic/gin"
)

func main() {

  api := gin.Default()

  api.GET("/ping", func(c *gin.Context) {
    c.JSON(200, gin.H{
      "message": "pong",
    })
  })

  api.Run(":8080") // listen and serve on localhost:8080
}

We are ready to test our server. Go back to the terminal window and run:

go run main.go

To test the endpoint open your web browser and visit:

localhost:8080/ping

You’ll see the server respond as expected.

{"message":"pong"}

After we confirm that our endpoint is working, return to the terminal window and use the keyboard command ctrl c to terminate the process.

Generate the Agora Tokens

Now that we have our Gin server setup, we are ready to add the functionality to generate the RTC and RTM tokens.

Before we can generate our tokens we need to add our AppID and AppCertificate. We’ll declare the appID and appCertificate as Strings in the global scope. For this guide, use environment variables to store the project credentials, so we’ll need to retrieve them. Within main() we’ll use os.LookupEnv to retrieve the environment variables. The os.LookupEnv returns a String for the environment variable along with a boolean for whether the variable existed. We’ll use the latter return values to check if the environment is configured correctly. If so we can assign the environment variable values to our global appID and AppCertificate variables, respectively.

package main

import (
  "log"
  "os"

  "github.com/gin-gonic/gin"
)

var appID, appCertificate string

func main() {

  appIDEnv, appIDExists := os.LookupEnv("APP_ID")
  appCertEnv, appCertExists := os.LookupEnv("APP_CERTIFICATE")

  if !appIDExists || !appCertExists {
    log.Fatal("FATAL ERROR: ENV not properly configured, check APP_ID and APP_CERTIFICATE")
  } else {
    appID = appIDEnv
    appCertificate = appCertEnv
  }

  api := gin.Default()

  api.GET("/ping", func(c *gin.Context) {
    c.JSON(200, gin.H{
      "message": "pong",
    })
  })

  api.Run(":8080")
}

Next we will add 3 endpoints, one for RTC tokens, another for RTM tokens, and one that returns both tokens.

The RTC token will require a channel name, the UID, the user role, tokentype to distinguish between string and integer based UIDs, and lastly an expiration time. The RTM endpoint only requires a UID and an expiration time. The dual token endpoint will need to accept the same structure as the RTC token endpoint.

api.GET("rtc/:channelName/:role/:tokentype/:uid/", getRtcToken)
api.GET("rtm/:uid/", getRtmToken)
api.GET("rte/:channelName/:role/:tokentype/:uid/", getBothTokens)

To minimize the amount of repeated code, the three functions getRtcToken, getRtmToken, and getBothTokens will call separate functions (parseRtcParams/parseRtmParams) to validate and extract the values passed to each endpoint. Then each function will use the returned values to generate the tokens and return them as JSON in the response body.

RTC tokens can be generated using two types of UIDs (uint/string), so we’ll use a function (generateRtcToken) to wrap the Agora RTC Token Builder functions BuildTokenWithUserAccount/BuildTokenWithUID.

Below is the base template for our token server. We’ll walk through each function and fill in the blanks.

package main

import (
  "log"
  "os"

  "github.com/AgoraIO-Community/go-tokenbuilder/rtctokenbuilder"
  "github.com/gin-gonic/gin"
)

var appID, appCertificate string

func main() {

  appIDEnv, appIDExists := os.LookupEnv("APP_ID")
  appCertEnv, appCertExists := os.LookupEnv("APP_CERTIFICATE")

  if !appIDExists || !appCertExists {
    log.Fatal("FATAL ERROR: ENV not properly configured, check APP_ID and APP_CERTIFICATE")
  } else {
    appID = appIDEnv
    appCertificate = appCertEnv
  }

  api := gin.Default()

  api.GET("/ping", func(c *gin.Context) {
    c.JSON(200, gin.H{
      "message": "pong",
    })
  })

  api.GET("rtc/:channelName/:role/:tokenType/:uid/", getRtcToken)
  api.GET("rtm/:uid/", getRtmToken)
  api.GET("rte/:channelName/:role/:tokenType/:uid/", getBothTokens)

  api.Run(":8080")
}

func getRtcToken(c *gin.Context) {

}

func getRtmToken(c *gin.Context) {

}

func getBothTokens(c *gin.Context) {

}

func parseRtcParams(c *gin.Context) (channelName, tokentype, uidStr string, role rtctokenbuilder.Role, expireTimestamp uint32, err error) {

}

func parseRtmParams(c *gin.Context) (uidStr string, expireTimestamp uint32, err error) {

}

func generateRtcToken(channelName, uidStr, tokentype string, role rtctokenbuilder.Role, expireTimestamp uint32) (rtcToken string, err error) {

}

Build the RTC Token

We’ll start with getRtcToken. This function takes a reference to the gin.Context, using it to call parseRtcParams which will extract the required values. Then using the returned values to call generateRtcToken to generate the token String. We’ll also include a few checks for errors to make sure there weren’t any issues along the way. Lastly we’ll build the response.

func getRtcToken(c *gin.Context) {
  log.Printf("rtc token\n")
  // get param values
  channelName, tokentype, uidStr, role, expireTimestamp, err := parseRtcParams(c)

  if err != nil {
    c.Error(err)
    c.AbortWithStatusJSON(400, gin.H{
      "message": "Error Generating RTC token: " + err.Error(),
      "status":  400,
    })
    return
  }

  rtcToken, tokenErr := generateRtcToken(channelName, uidStr, tokentype, role, expireTimestamp)

  if tokenErr != nil {
    log.Println(tokenErr) // token failed to generate
    c.Error(tokenErr)
    errMsg := "Error Generating RTC token - " + tokenErr.Error()
    c.AbortWithStatusJSON(400, gin.H{
      "status": 400,
      "error":  errMsg,
    })
  } else {
    log.Println("RTC Token generated")
    c.JSON(200, gin.H{
      "rtcToken": rtcToken,
    })
  }
}

Next let’s fill in parseRtcParams. This function also takes a reference to the gin.Context, which we’ll use to extract the parameters and return them. You’ll notice parseRtcParams also returns an error in case we run into any issues we can return an error message.


func parseRtcParams(c *gin.Context) (channelName, tokentype, uidStr string, role rtctokenbuilder.Role, expireTimestamp uint32, err error) {
  // get param values
  channelName = c.Param("channelName")
  roleStr := c.Param("role")
  tokentype = c.Param("tokentype")
  uidStr = c.Param("uid")
  expireTime := c.DefaultQuery("expiry", "3600")

  if roleStr == "publisher" {
    role = rtctokenbuilder.RolePublisher
  } else {
    role = rtctokenbuilder.RoleSubscriber
  }

  expireTime64, parseErr := strconv.ParseUint(expireTime, 10, 64)
  if parseErr != nil {
    // if string conversion fails return an error
    err = fmt.Errorf("failed to parse expireTime: %s, causing error: %s", expireTime, parseErr)
  }

  // set timestamps
  expireTimeInSeconds := uint32(expireTime64)
  currentTimestamp := uint32(time.Now().UTC().Unix())
  expireTimestamp = currentTimestamp + expireTimeInSeconds

  return channelName, tokentype, uidStr, role, expireTimestamp, err
}

Lastly, we’ll fill in the generateRtcToken function. This function takes the channel name, the UID as a String, the type of token (uid or userAccount), the role, and the expire time.

Using these values, the function calls the appropriate Agora RTC Token Builder function (BuildTokenWithUserAccount/BuildTokenWithUID) to generate a token String. Once the token builder function returns we’ll first check for errors and if there aren’t any we’ll return the token String value.

func generateRtcToken(channelName, uidStr, tokentype string, role rtctokenbuilder.Role, expireTimestamp uint32) (rtcToken string, err error) {

  if tokentype == "userAccount" {
    log.Printf("Building Token with userAccount: %s\n", uidStr)
    rtcToken, err = rtctokenbuilder.BuildTokenWithUserAccount(appID, appCertificate, channelName, uidStr, role, expireTimestamp)
    return rtcToken, err

  } else if tokentype == "uid" {
    uid64, parseErr := strconv.ParseUint(uidStr, 10, 64)
    // check if conversion fails
    if parseErr != nil {
      err = fmt.Errorf("failed to parse uidStr: %s, to uint causing error: %s", uidStr, parseErr)
      return "", err
    }

    uid := uint32(uid64) // convert uid from uint64 to uint 32
    log.Printf("Building Token with uid: %d\n", uid)
    rtcToken, err = rtctokenbuilder.BuildTokenWithUID(appID, appCertificate, channelName, uid, role, expireTimestamp)
    return rtcToken, err

  } else {
    err = fmt.Errorf("failed to generate RTC token for Unknown Tokentype: %s", tokentype)
    log.Println(err)
    return "", err
  }
}

Build the RTM token

Next, let’s move on to getRtmToken. Just like the code above, getRtmToken takes a reference to the gin.Context, uses it to call parseRtmParams to extract the required values, and uses the returned values to generate an RTM token. The difference here is that we call the Agora RTM Token builder directly to generate the token, String. We’ll include the error checks to make sure there weren’t any issues, and lastly we’ll build the response.

func getRtmToken(c *gin.Context) {
  log.Printf("rtm token\n")
  // get param values
  uidStr, expireTimestamp, err := parseRtmParams(c)

  if err != nil {
    c.Error(err)
    c.AbortWithStatusJSON(400, gin.H{
      "message": "Error Generating RTC token: " + err.Error(),
      "status":  400,
    })
    return
  }

  rtmToken, tokenErr := rtmtokenbuilder.BuildToken(appID, appCertificate, uidStr, rtmtokenbuilder.RoleRtmUser, expireTimestamp)

  if tokenErr != nil {
    log.Println(err) // token failed to generate
    c.Error(err)
    errMsg := "Error Generating RTM token: " + tokenErr.Error()
    c.AbortWithStatusJSON(400, gin.H{
      "error":  errMsg,
      "status": 400,
    })
  } else {
    log.Println("RTM Token generated")
    c.JSON(200, gin.H{
      "rtmToken": rtmToken,
    })
  }
}

Next let’s fill in parseRtmParams. This function also takes a reference to the gin.Context, then extracts and returns the parameters.

func parseRtmParams(c *gin.Context) (uidStr string, expireTimestamp uint32, err error) {
  // get param values
  uidStr = c.Param("uid")
  expireTime := c.DefaultQuery("expiry", "3600")

  expireTime64, parseErr := strconv.ParseUint(expireTime, 10, 64)
  if parseErr != nil {
    // if string conversion fails return an error
    err = fmt.Errorf("failed to parse expireTime: %s, causing error: %s", expireTime, parseErr)
  }

  // set timestamps
  expireTimeInSeconds := uint32(expireTime64)
  currentTimestamp := uint32(time.Now().UTC().Unix())
  expireTimestamp = currentTimestamp + expireTimeInSeconds

  // check if string conversion fails
  return uidStr, expireTimestamp, err
}

Build both RTC and RTM tokens

Now that we are able to generate both RTC and RTM tokens with individual server requests, we are going to fill in getBothTokens to allow for generating both tokens from a single request. We’ll use code very similar to the getRtcToken, except this time we’ll include the RTM token.

func getBothTokens(c *gin.Context) {
  log.Printf("dual token\n")
  // get rtc param values
  channelName, tokentype, uidStr, role, expireTimestamp, rtcParamErr := parseRtcParams(c)

  if rtcParamErr != nil {
    c.Error(rtcParamErr)
    c.AbortWithStatusJSON(400, gin.H{
      "message": "Error Generating RTC token: " + rtcParamErr.Error(),
      "status":  400,
    })
    return
  }
  // generate the rtcToken
  rtcToken, rtcTokenErr := generateRtcToken(channelName, uidStr, tokentype, role, expireTimestamp)
  // generate rtmToken
  rtmToken, rtmTokenErr := rtmtokenbuilder.BuildToken(appID, appCertificate, uidStr, rtmtokenbuilder.RoleRtmUser, expireTimestamp)

  if rtcTokenErr != nil {
    log.Println(rtcTokenErr) // token failed to generate
    c.Error(rtcTokenErr)
    errMsg := "Error Generating RTC token - " + rtcTokenErr.Error()
    c.AbortWithStatusJSON(400, gin.H{
      "status": 400,
      "error":  errMsg,
    })
  } else if rtmTokenErr != nil {
    log.Println(rtmTokenErr) // token failed to generate
    c.Error(rtmTokenErr)
    errMsg := "Error Generating RTC token - " + rtmTokenErr.Error()
    c.AbortWithStatusJSON(400, gin.H{
      "status": 400,
      "error":  errMsg,
    })
  } else {
    log.Println("RTC Token generated")
    c.JSON(200, gin.H{
      "rtcToken": rtcToken,
      "rtmToken": rtmToken,
    })
  }

}

Test the Token Server

Let’s go back to our terminal window and run our token server.

run main.go

Once the server instance is running we’ll see the list of endpoints and the message: Listening and serving HTTP on :8080.

Now that our server instance is running, let’s open our web browser and test. For these tests we’ll try a few variations that omit various query params.

Test the RTC endpoint

We’ll start with the RTC token:

http://localhost:8080/rtc/testing/publisher/userAccount/1234/
http://localhost:8080/rtc/testing/publisher/uid/1234/

The endpoints will generate a token that can be used in the channel: testing by a user with the role of publisher and the UID (String and uint) of 1234.

{
  "rtcToken": "0062ec0d84c41c4442d88ba6f5a2beb828bIADJRwbbO8J93uIDi4J305xNXA0A+pVDTPLPavzwsLW3uAZa8+ij4OObIgDqFTEDoOMyXwQAAQAwoDFfAgAwoDFfAwAwoDFfBAAwoDFf"
}

To test this token we can use the Agora 1:1 Web Demo.

Test the RTM endpoint

Next, we’ll test the RTM token:

http://localhost:8080/rtm/1234/

The endpoints will generate a token that can be used by a user with the UID of 1234 to log into RTM with the given AppID.

{
  "rtmToken": "0062ec0d84c41c4442d88ba6f5a2beb828bIABSMH0fzaqy7sa0erk8u4Bp6FJ4sO1kQ/o6HCRECBRrzKPg45sAAAAAEAAjAkAEO+cyXwEA6APLozFf"
}

To test this token we can use the Agora RTM Tutorial Demo.

Test the Dual token endpoint

We’ll finish our testing with the Dual token endpoint:

http://localhost:8080/rte/testing/publisher/userAccount/1234/
http://localhost:8080/rte/testing/publisher/uid/1234/

The endpoints will generate both RTC and RTM tokens that can be used by a user with the UID (String or uint) of 1234 and for the Video channel: testing with the role of publisher.

{
  "rtcToken": "0062ec0d84c41c4442d88ba6f5a2beb828bIAD33wY6pO+xp6iBY8mbYz2YtOIiRoTTrzdIPF9DEFlSIwZa8+ij4OObIgAQ6e0EX+UyXwQAAQDvoTFfAgDvoTFfAwDvoTFfBADvoTFf",
  "rtmToken": "0062ec0d84c41c4442d88ba6f5a2beb828bIABbCwQgl2te3rk0MEDZ2xrPoalb37fFhTqmTIbGeWErWaPg45sAAAAAEAD1WwYBX+UyXwEA6APvoTFf"
}

To test the tokens we can use the Agora 1:1 Web Demo for the RTC token and Agora RTM Tutorial Demo for the RTM token.

After testing the endpoints, your terminal window will display all the requests.

Done!

And just like that we are done! In-case you weren’t coding along or want to see the finished product all together, I have uploaded all the code to GitHub: Agora Token Service.

Thanks for taking the time to read my tutorial and if you have any questions please let me know. If you see any room for improvement feel free to fork the repo and make a pull request!

Other Resources

For more information about the Tokens for Agora.io applications, please take a look at the Set up Authentication guide and [Agora Advanced Guide: How to build a Token]f(https://docs.agora.io/en/Video/token_server_go?platform=Go)(Go).

I also invite you to join the Agoira.io Developer Slack community.

How to Build a Token Server for Agora Applications using NodeJS

Hermes Frangoudis — Fri, 31 Jul 2020 22:00:01 +0000

Security within video chat applications is a hot topic at the moment. As remote working and virtual events become more prolific, the need for security will increase.

Within the Agora Platform, one layer of security comes in the form of token authentication. A token, for those of you that don't know, is a dynamic key that is generated using a set of given inputs. Agora's Platform uses tokens to authenticate users.

Agora offers token security for both its RTC and RTM SDKs. This guide will explain how to build a simple microservice using NodeJS and Express to generate an Agora RTC token. The example can easily be adapted to include an RTM token as it follows a similar pattern.

Prerequisites

A basic understanding of JavaScript ES6, NodeJS, and NPM (minimal knowledge needed)
An understanding of how express web servers function
(minimal knowledge needed)
An Agora Developer Account (see: How To Get Started with Agora)

Project Setup

To start our project we’ll create a new folder and open a terminal window at this folder.

In the terminal, we’ll run npm init to set up the node project. The create project prompt will appear. I used the default settings, but feel free to customize this portion.

Now that the project has been created, we can add our NPM dependencies (express and agora-access-token) using:

npm install express
npm install agora-access-token

Build the Express server

Now that the project is set up, open the folder in your favorite code editor. Looking at the package.json, you’ll notice that the entry file is index.js but this file doesn’t exist in our project so we’ll have to create a new file and name it index.js.

Within the index.js we’ll start by requiring our modules. From express, we’ll need the express object and from agora-access-token we’ll leverage ES6's destructuring assignments to extract references to the RtcTokenBuilder and RtcRole objects.

const express = require('express');
const {RtcTokenBuilder, RtcRole} = require('agora-access-token');

Let’s define constants for the PORT number we’re gonna use to listen for requests, I like to use 8080. We’ll also define constants for our Agora AppID and AppCertificate, I like to use environment variables so we aren’t exposing these values in our code but you can also set these values as Strings containing your respective Agora keys.

const PORT = 8080;
const APP_ID = process.env.APP_ID;
const APP_CERTIFICATE = process.env.APP_CERTIFICATE;

Next we’ll define our app constant that will instantiate our Express object and allow us to set up our server.

const app = express();

Before we can set up the GET endpoint for our Express server, we’ll need to define the functions that are invoked when the endpoint is accessed. The first function (nocache) will apply the response headers, that force the browser to never cache the response so we ensure we always get a fresh token. You’ll notice we call the next() method at the end because this function is a middleware function which is the first in the series so we need to call next() let Express know to continue to the next middleware function in the series.

const nocache = (req, resp, next) => {
  resp.header('Cache-Control', 'private, no-cache, no-store, must-revalidate');
  resp.header('Expires', '-1');
  resp.header('Pragma', 'no-cache');
  next();
};

The second function (generateAccessToken) will handle the request and return the JSON response. We’ll define the function for now and add the body once we finish setting up the Express server. This is the last function in the series so we don’t need the next parameter/function.

const generateAccessToken = (req, resp) => { };

Let’s define a GET endpoint, passing in the nochache and generateAccessToken functions.

app.get('/access_token', nocache, generateAccessToken);

As the last step for creating our Express server, we’ll implement the .listen() method and pass in the PORT and a callback once the server is ready and listening on the given port.

app.listen(PORT, () => {
  console.log(`Listening on port: ${PORT}`);
});

Generate the Agora Token

Now that we have our Express server setup, we are ready to add the functionality to the generateAccessToken function. We’ll start by setting the response header to ensure we don’t run into any CORS issues.

resp.header('Access-Control-Allow-Origin', '*');

Get the Query Parameters ###

Next we’ll check for the channelName query parameter. This is a required parameter so if channelName is undefined we need to return an error with a 500 response code and a JSON object with the error.

const channelName = req.query.channelName;
if (!channelName) {
  return resp.status(500).json({ 'error': 'channel is required' });
}

The next couple of parameters (uid, role, expirationTime) are not required so we’ll assign default values as needed.

For the uid we’ll set the default value to 0, which allows us to generate a “wildcard” token that can be used to join the given channel with any uid. This is only appropriate for use in low security situations (or during development), where it’s ok for all the users to share a single token.

An example of a low-security situation is live broadcast where anyone can join and watch as an audience member.

For the role, we’ll default each user to be a SUBSCRIBER and only check if the request passes a value of publisher, otherwise any other value can be ignored.

Please Note: Only privilege is enforced by the Agora Platform by default. This is the join channel privilege, to enable the enforcement of other privileges you will need to make a request through Agora Support.

For the expirationTime we’ll default to 3600 seconds which gives the user an hour to join the channel before the privilege expires. One thing to note about expiration time is the token’s privilege lifetime needs to be an integer that represents the time since Jan 1, 1970. We’ll use the current time and add our expiration time to it.

  // get uid 
  let uid = req.query.uid;
  if(!uid || uid == '') {
    uid = 0;
  }
  // get role
  let role = RtcRole.SUBSCRIBER;
  if (req.query.role == 'publisher') {
    role = RtcRole.PUBLISHER;
  }
  // get the expire time
  let expireTime = req.query.expireTime;
  if (!expireTime || expireTime == '') {
    expireTime = 3600;
  } else {
    expireTime = parseInt(expireTime, 10);
  }
  // calculate privilege expire time
  const currentTime = Math.floor(Date.now() / 1000);
  const privilegeExpireTime = currentTime + expireTime;

Build the Token ###

Now that we have all our elements for our token we are ready to use the RtcTokenBuilder object’s buildTokenWithUid to generate our token.

const token = RtcTokenBuilder.buildTokenWithUid(APP_ID, APP_CERTIFICATE, channelName, uid, role, privilegeExpireTime);

Return the Response ###

The last step in generating our token is returning the JSON response that contains the token.

return resp.json({ 'token': token });

Test the Token Server

Let’s go back to our package.json and add a “start” command within the “scripts” object. The start command will execute the “node index.js” command so we can run our server instance.

"scripts": {
  "test": "echo \"Error: no test specified\" && exit 1",
  "start": "node index.js"
},

Start the server ###

Let’s go back to our command prompt window and use our new command:

npm start

Once the server instance is listening we’ll see our “Listening on port: 8080” within our terminal window.

Test the endpoint ##

Now that our server instance is running, let’s open our web browser and test. For these tests we’ll try a few variations that omit various query params.

We’ll start by omitting all query params:

localhost:8080/access_token

This will display:

{"error":"channel is required"}

Next we’ll pass “test” as the channelName:

localhost:8080/access_token?channelName=test

This will output a token that can be used by any user.

{"token":"0062ec0d84c41c4442d88ba6f5a2beb828bIAD9qg4N4hd04MvaY6A72m4BjYmO/7+xnRMinaI0ncLzkAx+f9gAAAAAEACS0zcn9gASXwEAAQCGvRBf"}

We can continue testing with the rest of the query params and we’ll get a similar response as above.

localhost:8080/access_token?channelName=test&role=subscriber
localhost:8080/access_token?channelName=test&role=subscriber&uid=1234
localhost:8080/access_token?channelName=test&role=subscriber&uid=1234&expireTime=6400

Done!

And just like that we are done! In-case you weren’t coding along or want to see the finished product all together, I have uploaded all the code to GitHub:

https://github.com/digitallysavvy/Agora-Node-TokenServer

Thanks for taking the time to read my tutorial and if you have any questions please let me know with a comment. If you see any room for improvement feel free to fork the repo and make a pull request!

Other Resources

For more information about the Tokens for Agora.io applications, please take look at the Set up Authentication guide and Agora Advanced Guide: How to build a Token(NodeJS).

I also invite you to join the Agoira.io Developer Slack community.

How To Build Hot Dog — Not Hot Dog

Hermes Frangoudis — Sun, 19 Jul 2020 02:56:10 +0000

Are you a big TV show binge-watcher? I sure am! As a dev, one thing I really enjoy is when a particular show highlights how technology interacts with the real world in believable ways, impacting us with unexpected—and often unintentionally hilarious—results.

For instance, back in 2017, HBO’s Silicon Valley aired an episode with the “Hot dog — Not hot dog” scene, where Jin Yang creates an app that recognizes hot dogs and everything else as “not hot dog.” The scene depicts a classic first step in training an AI for visual recognition.

For this tutorial, I will train a custom AI model using IBM Watson, and then use that model to detect “hot dog” or “not hot dog” within a live camera view. I’ll use augmented reality to display the result to the user. Since everything is more fun with friends, we’ll add a live streaming component to it!

Prerequisites

Basic understanding of Swift
Basic understanding of ARKit
has Basic understanding of CoreML and AI
Agora.io Developer Account
IBM Cloud Account with Watson Studio

Please Note: While no CoreML/AI knowledge is needed to follow along, certain basic concepts won't be explained along the way.

Device Requirements

In this project, we’ll be using ARKit, so we have some device requirements:

iPhone 6S or newer
iPhone SE
iPad (2017)
All iPad Pro models

Training the AI

Before we can build our iOS app, we first need to train the computer vision AI model. I chose IBM’s Watson Studio because they provide a very simple, drag-n-drop interface for training a computer vision model.

Create the Watson Project

Once you’ve created and logged into your Watson Studio account, click Create Project button. Give your project a name/description, add the storage, and then click to create the project.

Next, click Add Project and select Visual Recognition. Make sure to follow the prompts to add a Watson Visual Recognition service to the project. On the Custom Models screen, we’re going to Classify Images, so click the Create Model button. Now let’s name our model. I chose the name HotDog!

Source Training Images

Now that we've set up our Watson instance, we need images to train our model. Sourcing images for AI training may seem like a daunting undertaking. While it is quite a heavy lift, there are tools that help make this task easier.

I chose to use the Google Images Download python script. The script makes it easy to scrape images from Google while still respecting the original owner's copyrights.

Once you have set up the Google Images Download script, let's open up the command line and run it using:

googleimagesdownload --keywords "hotdog" --usage_rights labeled-for-reuse

We need to remove any photos like these so we are left with only pictures of real hot dogs (images can include toppings). Once all images are removed, you’ll notice that we’re left with about 50 photos. This isn’t very many considering that you’d usually want thousands of photos to train your model. While Watson could probably work with only 50 or so photos, let’s run the script a few more times with other keywords. These are the commands I used:

googleimagesdownload --keywords "hotdog" --usage_rights labeled-for-reuse
googleimagesdownload --keywords "plain hotdog" --usage_rights labeled-for-reuse
googleimagesdownload --keywords "real hotdog" --usage_rights labeled-for-reuse
googleimagesdownload --keywords "hotdog no toppings" --usage_rights labeled-for-reuse

After running the script a few times and removing any non-real hot dog images from each set of results, I was able to source 170 images for my model.

Let's put all our hot dog images into a single folder and name it hotdog. Now that we have our hot dog images, we need to find some not-hotdog images. Again, use the Google Images Download script but this time with a batch of keywords. I used:

googleimagesdownload --keywords "cake, pizza, hamburger, french fries, cup, plate, fork, glasses, computer, sandwich, table, dinner, meal, person, hand, keyboard"  --usage_rights labeled-for-reuse

IBM Watson's free tier imposes file size limits (250mb - per round) for training models, so once we've downloaded all of our non-hotdog images, we need to remove any images with large file sizes. Let's move all the images into a single folder and name it nothotdog. Next, zip each folder so you have hotdog.zip and nothotdog.zip.

Now, go back to the Watson Studio project, and upload the hotdog.zip file to our computer vision model. Once our zip finishes uploading, you'll notice that a new class hotdog has been created for us.

Next, upload your nothotdog.zip file. After it finishes uploading, you'll have two classes: hotdog and nothotdog. For this example, we only need one class, hotdog; the other class needs to be migrated into the existing Negative class. To do this, we need to open up the nothotdog class and select all the images. To do so, select the list view from the top, then scroll to the bottom and set the list length to 200, then scroll back to the top and click the select all button.

With all your images selected, click the Reclassify button, select the Negative class, and click Submit.

Once all the images have been reclassified, click back to the list of models to select and delete the nothotdog class. Now, we are ready to click the Train Model button to get Watson trained on our images.

Note: if you want to use a large data set you’ll have to break it up into pieces and repeat the training process (above) for each batch.

That's about it for collecting training images, all in all it wasn't too bad.

Building the iOS App

Now that Watson is training the visual recognition model, we are ready to build our iOS app.

In this example we’ll build an app that allows users to create or join a channel. Users that create a channel are then able to live stream themselves while they use IBM Watson custom model to infer hotdog or not hotdog.

Let’s start by creating a new single view app in Xcode.

Remove Scene Delegate

Since we are using the Storyboard interface, we can remove the SceneDelegate.swift and the Scene Manifest entry from the info.plist. Then we need to open the AppDelegate.swift and remove the Scene Delegate methods, and add the window property. Your AppDelegate.swift should look like this:

Since this project will implement ARKit and Agora, we’ll use the AgoraARKit library to simplify the implementation and UI for us.

Create a Podfile, open it and add the AgoraARKit pod.

platform :ios, '12.2'

target 'Agora Watson ARKit Demo' do
  use_frameworks!

  # Pods for Agora Watson ARKit Demo
  pod 'AgoraARKit'

  target 'Agora Watson ARKit DemoTests' do
    inherit! :search_paths
    # Pods for testing
  end

  target 'Agora Watson ARKit DemoUITests' do
    # Pods for testing
  end

end

Then run the install:

pod install

Permissions

Add NSCameraUsageDescription, NSMicrophoneUsageDescription, NSPhotoLibraryAddUsageDescription, and NSPhotoLibraryUsageDescription to the info.plist with a brief description for each. AgoraARKit uses the popular ARVideoKit framework and the last two permissions are required by ARVideoKit because of its ability to store photos/videos.

Note: I'm not implementing on-device recording so we don't need any of the library permissions; but if you plan to use this in production you will need to include them because they are requirements for ARVideoKit. For more information review Apple's guidelines on permissions.

Building the UI

We are ready to start building the UI. For this app we will have to build two views, the initial view and the AR view.

Within any live streaming and communication apps, you (as the developer) have two options for setting a channel name, do it for the user or allow users to input their own. The latter is more flexible, so we're going to extend our initial view to need to inherit from AgoraLobbyVC and allow users to input a channel name. Open your ViewController.swift, add import AgoraARKit just below the import UIKit line and set your ViewController class to inherit from AgoraLobbyVC.

Next, set your Agora App Id within the loadViewmethod and also set a custom image for the bannerImage property.

Next let's override the joinSession and createSession methods within our ViewController to set the images for the audience and broadcaster views.

import UIKit
import AgoraARKit

class ViewController: AgoraLobbyVC {

    override func loadView() {
        super.loadView()

        AgoraARKit.agoraAppId = ""

        // set the banner image within the initial view
        if let agoraLogo = UIImage(named: "watson_live_banner") {
            self.bannerImage = agoraLogo
        }
    }

    override func viewDidLoad() {
        super.viewDidLoad()
        // Do any additional setup after loading the view.
    }

    // MARK: Button Actions
    @IBAction override func joinSession() {
        if let channelName = self.userInput.text {
            if channelName != "" {
                let arAudienceVC = ARAudience()
                if let exitBtnImage = UIImage(named: "exit") {
                    arAudienceVC.backBtnImage = exitBtnImage
                }
                arAudienceVC.channelName = channelName
                arAudienceVC.modalPresentationStyle = .fullScreen
                self.present(arAudienceVC, animated: true, completion: nil)
            } else {
               // TODO: add visible msg to user
               print("unable to join a broadcast without a channel name")
            }
        }
    }

    @IBAction override func createSession() {
        if let channelName = self.userInput.text {
            if channelName != "" {
                let arBroadcastVC = ARBroadcaster()
                if let exitBtnImage = UIImage(named: "exit") {
                    arBroadcastVC.backBtnImage = exitBtnImage
                }
                if let micBtnImage = UIImage(named: "mic"),
                    let muteBtnImage = UIImage(named: "mute") {
                    arBroadcastVC.micBtnImage = micBtnImage
                    arBroadcastVC.muteBtnImage = muteBtnImage
                }

                arBroadcastVC.channelName = channelName
                arBroadcastVC.modalPresentationStyle = .fullScreen
                self.present(arBroadcastVC, animated: true, completion: nil)
            } else {
               // TODO: add visible msg to user
               print("unable to launch a broadcast without a channel name")
            }
        }
    }
}

Adding in the AI

Once Watson has finished training your model, you’ll need to download the CoreML file. Open Watson Studio and select the Hotdog Model. Within the model details, select the Implementation tab, then select the Core ML tab from the sub-menu on the left side of the screen. At the top of the Core ML section is the link to download the *CoreML*app model file.

Once you’ve downloaded the Hotdog.mlmodel file, drag the file into your Xcode project.

The computer vision will be running within our AR view, which is also the camera view being streamed into Agora, so we'll extend the ARBroadcaster class. The ARBoadcaster class is a bare-bones ARSCNView that is set up as a custom video source for Agora's SDK.

Create a new class called arHotDogBroadcaster which inherits from ARBroadcaster. Next we need to add properties for VNRequest and the DispatchQueue. Next extend the viewDidLoad and import the coreML model.

    let mlModel: MLModel = Hotdog().model
    var visionRequests = [VNRequest]()
    let dispatchQueueML = DispatchQueue(label: "io.agora.dispatchqueue.ml") // A Serial Queue

    override func viewDidLoad() {
        super.viewDidLoad()
        // Set up Vision Model
        guard let hotDogModel = try? VNCoreMLModel(for: mlModel) else {
            fatalError("Could not load model. Ensure Coreml model is in your XCode Project and part of a target (see: https://stackoverflow.com/questions/45884085/model-is-not-part-of-any-target-add-the-model-to-a-target-to-enable-generation ")
        }

        // Set up Vision-CoreML Request
        let classificationRequest = VNCoreMLRequest(model: hotDogModel, completionHandler: classificationCompleteHandler)
        classificationRequest.imageCropAndScaleOption = VNImageCropAndScaleOption.centerCrop // Crop from centre of images and scale to appropriate size.
        visionRequests = [classificationRequest]

    }

We'll use the currentFrame from the ARKit scene as our input for our computer vision. Use the currentFrame.capturedImage to create a CIImage that will be used as input for our VNImageRequestHandler.

func runCoreML() {
    // Get Camera Image as RGB
    guard let sceneView = self.sceneView else { return }
    guard let currentFrame = sceneView.session.currentFrame else { return }
    let pixbuff : CVPixelBuffer = currentFrame.capturedImage
    let ciImage = CIImage(cvPixelBuffer: pixbuff)

    // Prepare CoreML/Vision Request
    let imageRequestHandler = VNImageRequestHandler(ciImage: ciImage, options: [:])

    // Run Image Request
    do {
        try imageRequestHandler.perform(self.visionRequests)
    } catch {
        print(error)
    }

}

How do we know what the results are? If you look at the viewDidLoad snippet above, you'll notice we set classificationCompleteHandler as the completion block for any classification requests.

func classificationCompleteHandler(request: VNRequest, error: Error?) {
    // Catch Errors
    if error != nil {
        print("Error: " + (error?.localizedDescription)!)
        return
    }
    guard let observations = request.results else {
        print("No results")
        return
    }

    // Get Classifications
    let classification: VNClassificationObservation = observations.first as! VNClassificationObservation

    DispatchQueue.main.async {
        // Print Classifications
        print("--")

        // Display Debug Text on screen
        let debugText: String = "- \(classification.identifier) : \(classification.confidence)"
        print(debugText)

        // Display prediction
        var objectName: String = "Not Hotdog"
        if classification.confidence > 0.4 {
            objectName = "Hotdog"
        }

        // show the result
        self.showResult(objectName)
    }
}

Every time the CoreML engine returns a response we need to parse it and display to the user, “Hot Dog” or “Not Hot Dog.” You’ll notice in the snippet, that once we have a result, we parse it and then check the confidence level. I set the bar fairly low with a 40% confidence. That means that the AI model only has to be 40% confident that it sees a hotdog.

During testing, 40% confidence proved adequate for the intents of this project, but you may want to adjust that value depending on how sensitive you want your AI to be.

All that's left now is to display the result to the user using augmented reality. You'll notice in the classificationCompleteHandler, we call the function self.showResult and pass in a string with the value of either "Hot Dog" or "Not Hot Dog." Within showResult, we need to get the estimated position of the object and add an AR text label.

func showResult(_ result: String) {
    // HIT TEST : REAL WORLD
    // Get Screen Centre
    let screenCentre : CGPoint = CGPoint(x: self.sceneView.bounds.midX, y: self.sceneView.bounds.midY)

    let arHitTestResults : [ARHitTestResult] = sceneView.hitTest(screenCentre, types: [.featurePoint]) // Alternatively, we could use '.existingPlaneUsingExtent' for more grounded hit-test-points.

    if let closestResult = arHitTestResults.first {
        // Get Coordinates of HitTest
        let transform : matrix_float4x4 = closestResult.worldTransform
        let worldCoord : SCNVector3 = SCNVector3Make(transform.columns.3.x, transform.columns.3.y, transform.columns.3.z)

        // Create 3D Text
        let node : SCNNode = createNewResultsNode(result)
        resultsRootNode.addChildNode(node)
        node.position = worldCoord
    }
}

func createNewResultsNode(_ text : String) -> SCNNode {
    // Warning: Programmatically generating 3D Text is susceptible to crashing. To reduce chances of crashing; reduce number of polygons, letters, smoothness, etc.
    print("shwo result: \(text)")
    // Billboard contraint to force text to always face the user
    let billboardConstraint = SCNBillboardConstraint()
    billboardConstraint.freeAxes = SCNBillboardAxis.Y

    // SCN Text
    let scnText = SCNText(string: text, extrusionDepth: CGFloat(textDepth))
    var font = UIFont(name: "Helvetica", size: 0.15)
    font = font?.withTraits(traits: .traitBold)
    scnText.font = font
    scnText.alignmentMode = CATextLayerAlignmentMode.center.rawValue
    scnText.firstMaterial?.diffuse.contents = UIColor.orange
    scnText.firstMaterial?.specular.contents = UIColor.white
    scnText.firstMaterial?.isDoubleSided = true
    scnText.chamferRadius = CGFloat(textDepth)

    // Text Node
    let (minBound, maxBound) = scnText.boundingBox
    let textNode = SCNNode(geometry: scnText)
    // Centre Node - to Centre-Bottom point
    textNode.pivot = SCNMatrix4MakeTranslation( (maxBound.x - minBound.x)/2, minBound.y, textDepth/2)
    // Reduce default text size
    textNode.scale = SCNVector3Make(0.2, 0.2, 0.2)

    // Sphere Node
    let sphere = SCNSphere(radius: 0.005)
    sphere.firstMaterial?.diffuse.contents = UIColor.cyan
    let sphereNode = SCNNode(geometry: sphere)

    // Text Parent Node
    let textParentNode = SCNNode()
    textParentNode.addChildNode(textNode)
    textParentNode.addChildNode(sphereNode)
    textParentNode.constraints = [billboardConstraint]

    return textParentNode
}

Now that we have our model ready to run, we need to add a way for the user to invoke the computer vision model. Let's use the View's touchesBegan method to call the runCoreML method.

override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
  dispatchQueueML.async {
      self.runCoreML()
  }
}

Implement new broadcaster class

We're almost done. The last step (before we can start testing) is to set the ARBroadcaster in the ViewController.swift by updating line 43 to:

let arBroadcastVC = arHotDogBroadcaster()

This will set the ARbroadcaster to use our new arHotDogBroadcaster and we are ready to start testing!

That's It!

The core application is done, I'll leave it up to you to customize the UI. Thanks for following along. If you have any questions or feedback, please leave a comment.

I've uploaded my complete code with UI customizations, to GitHub so feel free to fork the repo and make PR's for new features.

Other Resources

How To: Build a Live Broadcasting Web App

Hermes Frangoudis — Fri, 05 Jun 2020 15:23:12 +0000

In a previous post we built a group video chat app similar to a Google Hangouts/Meet. Now we want to build something more akin to Instagram Live. When building a live broadcasting web-app, the architecture is similar to that of a communication web-app, except with a slight twist.

Pre Requisites

A simple web server — I like to use Live Server
An SSL cert or way to have an https connection (I use ngrok)
A developer account with Agora.io
An understanding of HTML/CSS/JS
An understanding of how Bootstrap and JQuery function (minimal knowledge needed)
Read: Building a group video chat web-app

Architecture and Structure Design

In “Building a Group Video Chat Web-App,” the build conformed to the communication scenario, where every user in the channel has the ability to broadcast their audio and video streams.

In this project we will be implementing a broadcasting scenario, where there is a mix of users. Some users in the channel are broadcasting (Broadcasters) their camera Broadcasters (Audience).

In this case we need to limit the broadcasting capability to specific users while allowing all other users are in the Audience they will only consume the Broadcaster stream(s).

The graphic above helps us to visualize the flow of our Broadcaster web-app. We can see we have to implement two different clients, one for Broadcasters and one for the Audience. Each client will have a unique interface for each of the user roles (Broadcaster/Audience) and both will connect to the Agora.io Software Defined Real-Time Network (SD-RTN).

Core Structure (HTML)

n our live broadcast web-app we will have two clients (Broadcaster/Audience), each with their own UI.

The Broadcaster client is almost identical to the client we built for our communication web app. The main difference being we won’t need to account for other Broadcasters, since we’re building this as a one-to-many broadcast.

<html lang="en">
    <head>
        <title>Agora.io [HOST] - AllThingsRTC Live Stream</title>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <script src="https://cdn.agora.io/sdk/web/AgoraRTCSDK-2.6.1.js"></script>
        <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.0/css/all.css" integrity="sha384-lZN37f5QGtY3VHgisS14W3ExzMWZxybE1SJSEsQp9S+oqd12jhcu+A56Ebc1zFSJ" crossorigin="anonymous">
        <link href="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/css/bootstrap.min.css" rel="stylesheet">
        <link href="https://cdnjs.cloudflare.com/ajax/libs/animate.css/3.7.0/animate.min.css" rel="stylesheet">
        <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
        <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"></script>
        <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/js/bootstrap.min.js"></script>
        <!-- <link href="https://cdnjs.cloudflare.com/ajax/libs/mdbootstrap/4.8.2/css/mdb.min.css" rel="stylesheet"> -->
        <link rel="stylesheet" type="text/css" href="css/style.css"/>
    </head>
    <body>
            <div class="container-fluid p-0">
                <div id="main-container">
                    <div id="screen-share-btn-container" class="col-2 float-right text-right mt-2">
                        <button id="screen-share-btn"  type="button" class="btn btn-lg">
                            <i id="screen-share-icon" class="fab fa-slideshare"></i>
                        </button>
                    </div>
                    <div id="buttons-container" class="row justify-content-center mt-3">
                        <div id="audio-controls" class="col-md-2 text-center btn-group">
                            <button id="mic-btn" type="button" class="btn btn-block btn-dark btn-lg">
                                <i id="mic-icon" class="fas fa-microphone"></i>
              </button>
              <!-- insert mic selection drop-down -->
                        </div>
                        <div id="video-controls" class="col-md-2 text-center btn-group">
                            <button id="video-btn"  type="button" class="btn btn-block btn-dark btn-lg">
                                <i id="video-icon" class="fas fa-video"></i>
              </button>
              <!-- insert camera selection drop-down -->
                        </div>
                        <div class="col-md-2 text-center">
                            <button id="exit-btn"  type="button" class="btn btn-block btn-danger btn-lg">
                                <i id="exit-icon" class="fas fa-phone-slash"></i>
                            </button>
                        </div>
                    </div>
                    <div id="full-screen-video"></div>
                    <div id="lower-ui-bar" class="row fixed-bottom mb-1">
                        <div id="rtmp-btn-container" class="col ml-3 mb-2">
                            <button id="rtmp-config-btn"  type="button" class="btn btn-primary btn-lg row rtmp-btn" data-toggle="modal" data-target="#addRtmpConfigModal">
                                <i id="rtmp-config-icon" class="fas fa-rotate-270 fa-sign-out-alt"></i>
                            </button>
                            <button id="add-rtmp-btn"  type="button" class="btn btn-secondary btn-lg row rtmp-btn" data-toggle="modal" data-target="#add-external-source-modal">
                                <i id="add-rtmp-icon" class="fas fa-plug"></i>
                            </button>
                        </div>
                        <div id="external-broadcasts-container" class="container col-flex">
                            <div id="rtmp-controlers" class="col">
                                <!-- insert rtmp  controls -->
                            </div>
                        </div>
                    </div>
                </div>
                <!-- insert RTMP Config Modal -->
                <!-- insert External Injest Url Modal -->
            </div>
        </div>
    </body>
    <script>
        $("#mic-btn").prop("disabled", true);
        $("#video-btn").prop("disabled", true);
        $("#screen-share-btn").prop("disabled", true);
        $("#exit-btn").prop("disabled", true);
    </script>
    <script src="js/agora-interface.js"></script>
    <script src="js/agoraBroadcastClient.js"></script>
    <script src="js/ui.js"></script>
</html>

The above code should look very familiar with a few minor differences — we’ve added some comments to add drop down selectors for the camera and microphone buttons. Along with the button comments there are also two rows of comments for some modal windows that we need to add in.

<!-- insert RTMP Config Modal -->
<!-- insert External Injest Url Modal -->

Let’s start with the camera/mic buttons. We can use button groups to create simple containers for each and we’ll use Agora.io’s SDK to get the input devices and populate these containers. This will give the broadcaster the ability to switch their camera and microphone to any media device connected to their computer/tablet/phone.

<div id="buttons-container" class="row justify-content-center mt-3">
    <div id="audio-controls" class="col-md-2 text-center btn-group">
    <button id="mic-btn" type="button" class="btn btn-block btn-dark btn-lg">
        <i id="mic-icon" class="fas fa-microphone"></i>
    </button>
    <!-- insert mic selection drop-down -->
    </div>
    <div id="video-controls" class="col-md-2 text-center btn-group">
    <button id="video-btn"  type="button" class="btn btn-block btn-dark btn-lg">
        <i id="video-icon" class="fas fa-video"></i>
    </button>
    <!-- insert camera selection drop-down -->
    </div>
    <div class="col-md-2 text-center">
    <button id="exit-btn"  type="button" class="btn btn-block btn-danger btn-lg">
        <i id="exit-icon" class="fas fa-phone-slash"></i>
    </button>
    </div>
</div>

Now we are ready to add in our models for configuring the settings for pushing our broadcast stream to RTMP servers. I’ll dive into pushing the stream to external servers a little later in the article, for now lets add the modals and input elements.

<!-- RTMP Config Modal -->
<div class="modal fade slideInLeft animated" id="addRtmpConfigModal" tabindex="-1" role="dialog" aria-labelledby="rtmpConfigLabel" aria-hidden="true" data-keyboard=true>
  <div class="modal-dialog">
    <div class="modal-content">
      <div class="modal-header">
        <h5 class="modal-title" id="rtmpConfigLabel"><i class="fas fa-sliders-h"></i></h5>
        <button type="button" class="close" data-dismiss="modal" data-reset="modal" aria-label="Close">
          <span aria-hidden="true">&times;</span>
        </button>
      </div>
      <div class="modal-body">
        <form id="rtmp-config">
            <div class="form-group">
              <input id="rtmp-url" type="text" class="form-control" placeholder="URL *"/>
            </div>
            <div class="form-group">
                <label for="window-scale">Video Scale</label>
                <input id="window-scale-width" type="number" value="640" min="1" max="1000" step="1"/> (w)&nbsp;
                <input id="window-scale-height" type="number" value="360" min="1" max="1000" step="1"/> (h) 
            </div>
            <div class="form-group row">
                <div class="col-flex">
                  <label for="audio-bitrate">Audio Bitrate</label>  
                  <input id="audio-bitrate" type="number" value="48" min="1" max="128" step="2"/>   
                </div>
                <div class="col-flex ml-3">
                    <label for="video-bitrate">Video Bitrate</label>
                    <input id="video-bitrate" type="number" value="400" min="1" max="1000000" step="2"/>
                </div>
            </div>
            <div class="form-group row">
              <div class="col-flex">
                  <label for="framerate">Frame Rate</label>
                  <input id="framerate" type="number" value="15" min="1" max="10000" step="1"/>
              </div>
              <div class="col-flex ml-3">
                <label for="video-gop">Video GOP</label>
                <input id="video-gop" type="number" value="30" min="1" max="10000" step="1"/>
              </div>
            </div>
            <div class="form-group">
                <label for="video-codec-profile">Video Codec Profile </label>
                <select id="video-codec-profile" class="form-control drop-mini">
                  <option value="66">Baseline</option>
                  <option value="77">Main</option>
                  <option value="100" selected>High (default)</option>
                </select>
            </div>
            <div class="form-group">
                <label for="audio-channels">Audio Channels </label>
                <select id="audio-channels" class="form-control drop-mini">
                  <option value="1" selected>Mono (default)</option>
                  <option value="2">Dual sound channels</option>
                  <option value="3" disabled>Three sound channels</option>
                  <option value="4" disabled>Four sound channels</option>
                  <option value="5" disabled>Five sound channels</option>
                </select>
            </div>
            <div class="form-group">
                <label for="audio-sample-rate">Audio Sample Rate </label>
                <select id="audio-sample-rate" class="form-control drop-mini">
                  <option value="32000">32 kHz</option>
                  <option value="44100" selected>44.1 kHz (default)</option>
                  <option value="48000">48 kHz</option>
                </select>
            </div>
            <div class="form-group">
                <label for="background-color-picker">Background Color </label>
                <input id="background-color-picker" type="text" class="form-control drop-mini" placeholder="(optional)" value="0xFFFFFF" />
            </div>
            <div class="form-group">
                <label for="low-latancy">Low Latency </label>
                <select id="low-latancy" class="form-control drop-mini">
                  <option value="true">Low latency with unassured quality</option>
                  <option value="false" selected>High latency with assured quality (default)</option>
                </select>
            </div>
        </form>
      </div>
      <div class="modal-footer">
        <button type="button" id="start-RTMP-broadcast" class="btn btn-primary">
            <i class="fas fa-satellite-dish"></i>
        </button>
      </div>
    </div>
  </div>
</div>
<!-- end Modal -->
<!-- External Injest Url Modal -->
<div class="modal fade slideInLeft animated" id="add-external-source-modal" tabindex="-1" role="dialog" aria-labelledby="add-external-source-url-label" aria-hidden="true" data-keyboard=true>
    <div class="modal-dialog">
      <div class="modal-content">
        <div class="modal-header">
          <h5 class="modal-title" id="add-external-source-url-label"><i class="fas fa-broadcast-tower"></i> [add external url]</i></h5>
          <button id="hide-external-url-modal" type="button" class="close" data-dismiss="modal" data-reset="modal" aria-label="Close">
            <span aria-hidden="true">&times;</span>
          </button>
        </div>
        <div class="modal-body">
            <form id="external-injest-config">
                <div class="form-group">
                  <input id="external-url" type="text" class="form-control" placeholder="URL *"/>
                </div>
                <div class="form-group">
                    <label for="external-window-scale">Video Scale</label>
                    <input id="external-window-scale-width" type="number" value="640" min="1" max="1000" step="1"/> (w)&nbsp;
                    <input id="external-window-scale-height" type="number" value="360" min="1" max="1000" step="1"/> (h) 
                </div>
                <div class="form-group row">
                    <div class="col-flex">
                      <label for="external-audio-bitrate">Audio Bitrate</label> 
                      <input id="external-audio-bitrate" type="number" value="48" min="1" max="128" step="2"/>  
                    </div>
                    <div class="col-flex ml-3">
                        <label for="external-video-bitrate">Video Bitrate</label>
                        <input id="external-video-bitrate" type="number" value="400" min="1" max="1000000" step="2"/>
                    </div>
                </div>
                <div class="form-group row">
                  <div class="col-flex">
                      <label for="external-framerate">Frame Rate</label>
                      <input id="external-framerate" type="number" value="15" min="1" max="10000" step="1"/>
                  </div>
                  <div class="col-flex ml-3">
                    <label for="external-video-gop">Video GOP</label>
                    <input id="external-video-gop" type="number" value="30" min="1" max="10000" step="1"/>
                  </div>
                </div>
                <div class="form-group">
                    <label for="external-audio-channels">Audio Channels </label>
                    <select id="external-audio-channels" class="form-control drop-mini">
                      <option value="1" selected>Mono (default)</option>
                      <option value="2">Dual sound channels</option>
                    </select>
                </div>
                <div class="form-group">
                    <label for="external-audio-sample-rate">Audio Sample Rate </label>
                    <select id="external-audio-sample-rate" class="form-control drop-mini">
                      <option value="32000">32 kHz</option>
                      <option value="44100" selected>44.1 kHz (default)</option>
                      <option value="48000">48 kHz</option>
                    </select>
                </div>
            </form>
        </div>
        <div class="modal-footer">
          <button type="button" id="add-external-stream" class="btn btn-primary">
              <i id="add-rtmp-icon" class="fas fa-plug"></i>    
          </button>
        </div>
      </div>
    </div>
  </div>
  <!-- end Modal -->

Now that we have our Broadcaster client we need our Audience client. The Audience client is very straight-forward. We have some wrapper div along with the fullscreen div that we can set to 100% of the browser window height and width to play the broadcast video stream.

<html lang="en">
    <head>
        <title>Agora.io - AllThingsRTC Live Stream</title>
        <meta charset="utf-8">
        <meta name="viewport" content="width=device-width, initial-scale=1">
        <script src="js/AgoraRTCSDK-3.1.1.js"></script>
        <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.0/css/all.css" integrity="sha384-lZN37f5QGtY3VHgisS14W3ExzMWZxybE1SJSEsQp9S+oqd12jhcu+A56Ebc1zFSJ" crossorigin="anonymous">
        <link href="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/css/bootstrap.min.css" rel="stylesheet">
        <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
        <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/js/bootstrap.min.js"></script>
        <link rel="stylesheet" type="text/css" href="css/style.css"/>
    </head>
    <body>
        <div class="container-fluid p-0">
            <div id="full-screen-video"></div>
            <div id="watch-live-overlay">
                <div id="overlay-container">
                    <div class="col-md text-center">
                        <button id="watch-live-btn" type="button" class="btn btn-block btn-primary btn-xlg">
                        <i id="watch-live-icon" class="fas fa-broadcast-tower"></i><span>Watch the Live Stream</span>
                        </button>
                    </div>
                </div>
            </div>
        </div>
    </body>
    <script src="js/agoraAudienceClient.js"></script>
</html>

Giving it some style with CSS

As with our last project, we’ll use a mix of Bootstrap and some custom CSS. For both clients we’ll use the same css file. The broadcaster client we can re-use most of the CSS from our communications web-app, with some adjustments/updates. For the audience, we can keep the CSS very simple.


body {
  margin: 0;
  padding: 0;
  background-image: url('../images/rtc-logo.png');
  background-repeat: no-repeat;
  background-size: contain;
  background-position: center;
}

body .btn:focus{ 
  outline: none !important;
  box-shadow:none !important;
}

#buttons-container {
  position: absolute;
  z-index: 2;  
  width: 100vw;
}

#buttons-container div {
  max-width: 250px;
  min-width: 150px;
  margin-bottom: 10px;
}

.btn-group button i {
  padding-left: 25px;
}

#full-screen-video {
  position: absolute;
  width: 100vw;
  height: 100vh;
}

#full-screen-video-iframe {
  position: absolute;
  width: 100vw;
  height: 100vh;
  background-image: url('../images/AllThingsRTC_Live-bg.jpg');
  background-repeat: no-repeat;
  background-size: cover;
  background-position: center;
}

#rtmp-btn-container { 
  position: relative; 
  display: inline-block;
  margin-top: auto;
  z-index: 99;
}

.rtmp-btn {
  bottom: 5vh;
  right: 5vw;
  display: block;
  margin: 0 0 5px 0;
}

#add-rtmp-btn {
  padding: 0.5rem 1.15rem;
}

.remote-stream-container { 
  display: inline-block;
}

#rtmp-controlers {
  height: 100%;
  margin: 0;
}

#local-video {
  position: absolute;
  z-index: 1;
  height: 20vh;
  max-width: 100%;
}

.remote-video {
  position: absolute;
  z-index: 1;
  height: 100% !important;
  width: 80%;
  max-width: 500px;
}

#mute-overlay {
  position: absolute;
  z-index: 2;
  bottom: 0;
  left: 0;
  color: #d9d9d9;
  font-size: 2em;
  padding: 0 0 3px 3px;
  display: none;
} 

.mute-overlay {
  position: absolute;
  z-index: 2;
  top: 2px;
  color: #d9d9d9;
  font-size: 1.5em;
  padding: 2px 0 0 2px;
  display: none;
}

#no-local-video, .no-video-overlay {
  position: absolute;
  z-index: 3;
  width: 100%;
  top: 40%;
  color: #cccccc;
  font-size: 2.5em;
  margin: 0 auto;
  display: none;
}

.no-video-overlay {
  width: 80%;
}

#screen-share-btn-container {
  z-index: 99;
}

#watch-live-overlay {
  position: fixed;
  top: 0;
  left: 0;
  width: 100%;
  height: 100%;
  text-align: center;
  background-image: url('../images/AllThingsRTC_Live-bg.jpg');
  background-repeat: no-repeat;
  background-size: cover;
  background-position: center;
}

#external-broadcasts-container {
  max-width: 70%;
  margin: auto 0 5px;
}

#external-broadcasts-container input {
  width: 50%;
}

#external-broadcasts-container button {
  color: #fff;
}

#watch-live-overlay #overlay-container {
  padding: 25px;
  border-radius: 5px;
  position:relative;
  margin: 0 auto;
  top: 65%;
  width: 70%;
}

#watch-live-overlay button {
  display: block;
  /* margin: -50px auto; */
  color: #0096e6;
  background: #fff;
}

#watch-live-overlay img {
  height: auto;
  width: 100%;
  object-fit: cover;
  object-position: center;
}

#watch-live-overlay button i {
  padding: 0 10px;
}

.btn-xlg {
  padding: 20px 35px;
  font-size: 30px;
  line-height: normal;
  -webkit-border-radius: 8px;
     -moz-border-radius: 8px;
          border-radius: 8px;
}

.drop-mini {
  width: inherit;
  display: inline-block;
}

#external-injest-config label, #rtmp-config label {
  margin: 0 .5rem .5rem 0;
}

#external-injest-config .row,#rtmp-config .row {
  margin-left: inherit;
  margin-right: inherit;
}


#addRtmpConfigModal .modal-header, 
#external-injest-config .modal-header {
  padding: 0.5rem 1rem 0;
  border-bottom: none;
}

#addRtmpConfigModal .modal-header .close, 
#external-injest-config .modal-header .close {
  padding: 0.5rem;
  margin: -.025rem;
}

#addRtmpConfigModal .modal-body, 
#external-injest-config .modal-body {
  padding: 1rem 1rem 0.25rem;
}

#addRtmpConfigModal .modal-footer, 
#external-injest-config .modal-footer {
  padding: 0 1rem 0.5rem;
  border-top: none;
}

#pushToRtmpBtn {
  padding: 10px 15px;
}

.close .fa-xs {
  font-size: .65em;
}

/* pulsating broadcast button */
.pulse-container {
  height: 100%;
  margin: 5px 10px 0;
}

.pulse-button {
  position: relative;
  /* width: 32px; */
  /* height: 32px; */
  border: none;
  box-shadow: 0 0 0 0 rgba(232, 76, 61, 0.7);
  /* border-radius: 50%; */
  background-color: #e84c3d;
  background-size:cover;
  background-repeat: no-repeat;
  cursor: pointer;
}

.pulse-anim {
  -webkit-animation: pulse 2.25s infinite cubic-bezier(0.66, 0, 0, 1);
  -moz-animation: pulse 2.25s infinite cubic-bezier(0.66, 0, 0, 1);
  -ms-animation: pulse 2.25s infinite cubic-bezier(0.66, 0, 0, 1);
  animation: pulse 2.25s infinite cubic-bezier(0.66, 0, 0, 1); 
}

@-webkit-keyframes pulse {to {box-shadow: 0 0 0 15px rgba(232, 76, 61, 0);}}
@-moz-keyframes pulse {to {box-shadow: 0 0 0 15px rgba(232, 76, 61, 0);}}
@-ms-keyframes pulse {to {box-shadow: 0 0 0 15px rgba(232, 76, 61, 0);}}
@keyframes pulse {to {box-shadow: 0 0 0 15px rgba(232, 76, 61, 0);}}

/* Respomnsive design */

@media only screen and (max-width: 795px) {
  #watch-live-overlay #overlay-container {
    width: 100%;
  }
}

@media only screen and (max-height: 350px) {
  #watch-live-overlay img {
    height: auto;
    width: 80%;
  }
  #watch-live-overlay #overlay-container {
    top: 60%;
  }
  .btn-xlg {
    font-size: 1rem;
  }
}

@media only screen and (max-height: 400px){
  .btn-xlg {
    font-size: 1.25rem;
  }
}

@media only screen and (max-width: 400px) {
  .btn-xlg {
    padding: 10px 17px;
  }
}

Building the Javascript Clients

Let’s start with our Broadcaster. I chose to use Agora.io’s WebSDK to simplify the heavy lifting generally associated with writing a WebRTC interface. I wrote a short guide on how to get setup with Agora.io for anyone new to the Agora.io platform.

In our JS code for both clients we start by declaring and initializing the Client object. Once we create the Client object we can join or leave the channel.

var client = AgoraRTC.createClient({mode: 'live', codec: 'vp8'});

Prior to joining the channel we need to set our client’s role. Within Agora’s SDK, any user publishing a stream is by default a broadcaster, but I’d recommend being explicit about the client’s role within the stream to avoid any unwanted streams.

// create broadcaster client and set the client role
var broadcastClient = AgoraRTC.createClient({mode: 'live', codec: 'vp8'}); 

broadcastClient.setClientRole('host', function() {
  console.log('Client role set as host.');
}, function(e) {
  console.log('setClientRole failed', e);
});

It shouldn’t be assumed that a broadcaster will always have a published stream. In that same breath it can’t be assumed that an audience member will never try to publish a stream (consider someone trying to hack and inject their stream into the broadcast).

// create audience client and set the client role
var audienceClient = AgoraRTC.createClient({mode: 'live', codec: 'vp8'}); 

audienceClient.setClientRole('audience', function() {
  console.log('Client role set to audience');
}, function(e) {
  console.log('setClientRole failed', e);
});

We’ll also need to add event listeners for the various engine events that Agora’s SDK provides. Most of the events should look familiar, as the broadcaster has all of the same controls as a video chat web-app, plus a few extras. We have liveStreamingStarted, liveStreamingFailed, liveStreamingStopped, liveTranscodingUpdated, and streamInjectedStatus as the new events. These events are related to the Agora’s ability to push out to external RTMP server or to pull in an external RTMP stream.

client.on('stream-published', function (evt) {
  console.log('Publish local stream successfully');
});

// when a remote stream is added
client.on('stream-added', function (evt) {
  console.log('new stream added: ' + evt.stream.getId());
});

client.on('stream-removed', function (evt) {
  var stream = evt.stream;
  stream.stop(); // stop the stream
  stream.close(); // clean up and close the camera stream
  console.log("Remote stream is removed " + stream.getId());
});

//live transcoding events..
client.on('liveStreamingStarted', function (evt) {
  console.log("Live streaming started");
}); 

client.on('liveStreamingFailed', function (evt) {
  console.log("Live streaming failed");
}); 

client.on('liveStreamingStopped', function (evt) {
  console.log("Live streaming stopped");
});

client.on('liveTranscodingUpdated', function (evt) {
  console.log("Live streaming updated");
}); 

// ingested live stream 
client.on('streamInjectedStatus', function (evt) {
  console.log("Injected Steram Status Updated");
  console.log(JSON.stringify(evt));
}); 

// when a remote stream leaves the channel
client.on('peer-leave', function(evt) {
  console.log('Remote stream has left the channel: ' + evt.stream.getId());
});

// show mute icon whenever a remote has muted their mic
client.on('mute-audio', function (evt) {
  console.log('Mute Audio for: ' + evt.uid);
});

client.on('unmute-audio', function (evt) {
  console.log('Unmute Audio for: ' + evt.uid);
});

// show user icon whenever a remote has disabled their video
client.on('mute-video', function (evt) {
  console.log('Mute Video for: ' + evt.uid);
});

client.on('unmute-video', function (evt) {
  console.log('Unmute Video for: ' + evt.uid);
});

The RTMP Push/Pull features will allow us to push the video broadcast out to third party RTMP servers such as Facebook, YouTube, Vimeo, or any other service that accepts an RTMP stream and similarly pull streams from these same sources into our live stream.

Since RTMP Push/Pull is one of the more important additions to our broadcaster UI, let’s first take a closer look at the methods needed to push our stream out to an external service and then we’ll dive into how to pull an external stream into our broadcast.

The two main methods that enable us to push out are setLiveTranscoding and startLiveStreaming. Before we can push our stream to an RTMP server, we first need to set the transcoding configuration using client.setLiveTranscoding(config) passing an object with the various settings. Then, once we have our configuration set, we can call client.startLiveStreaming(url) passing the url we wish to push out to.

function setTranscodingConfig() {
  console.log("save rtmp config"); 
  var width = parseInt($('#window-scale-width').val(), 10);
  var height = parseInt($('#window-scale-height').val(), 10);
  var configRtmp = {
    width: width,
    height: height,
    videoBitrate: parseInt($('#video-bitrate').val(), 10),
    videoFramerate: parseInt($('#framerate').val(), 10),
    lowLatency: ($('#low-latancy').val() === 'true'),
    audioSampleRate: parseInt($('#audio-sample-rate').val(), 10),
    audioBitrate: parseInt($('#audio-bitrate').val(), 10),
    audioChannels: parseInt($('#audio-channels').val(), 10),
    videoGop: parseInt($('#video-gop').val(), 10),
    videoCodecProfile: parseInt($('#video-codec-profile').val(), 10),
    userCount: 1,
    userConfigExtraInfo: {},
    backgroundColor: parseInt($('#background-color-picker').val(), 16),
    transcodingUsers: [{
      uid: localStreams.uid,
      alpha: 1,
      width: width,
      height: height,
      x: 0,
      y: 0,
      zOrder: 0
    }],
  };

  // set live transcoding config
  client.setLiveTranscoding(configRtmp);

}

We are using the form elements to allow the Broadcaster to control their settings, but since we are building an object for the configuration we need to ensure that we are pulling the values with the correct types.

One area to take note, is the userCount and transcodingUser. In our example, the user count is hardcoded in because we are only broadcasting with a single user, but if you intend to have more than a single Broadcaster in each channel you need to make this number dynamic to the number of streams you want to push to the external server. Along with userCount, the transcodingUser array in our example is a single element array, but you can make that dynamic if you wish to have more Broadcasters in the external stream.

Another point worth mentioning: if you are broadcasting more than one stream to a 3rd party service using RTMP, the streams will be merged into a single stream. To allow developers to control the layout when the videos are merged into a single video stream. The configuration object allows us to set the overall width and height of the stream and then within the transcodingUser array we can also specify the scale and position of the individual streams within the merged video stream.

Earlier we added some dropdown UI elements to our microphone and camera buttons. Let’s now take the time to add in the Agora method to load the list of our devices.

Its best to wait until the user has granted permission to use the devices before we ask for the list of permissions because each browser has varying support as to how much information they return. For example, in Safari the browser will return an empty array but in Chrome the browser returns an array of devices but without any identifiable information. On the other hand, every browser gives consistent access once the user has granted permission.

// The user has granted access to the camera and mic.
localStream.on("accessAllowed", function() {
  if(devices.cameras.length === 0 && devices.mics.length === 0) {
    console.log('[DEBUG] : checking for cameras & mics');
    getCameraDevices();
    getMicDevices();
  }
  console.log("accessAllowed");
});

function getCameraDevices() {
  console.log("Checking for Camera Devices.....")
  client.getCameras (function(cameras) {
    devices.cameras = cameras; // store cameras array
    cameras.forEach(function(camera, i){
      var name = camera.label.split('(')[0];
      var optionId = 'camera_' + i;
      var deviceId = camera.deviceId;
      if(i === 0 && localStreams.camera.camId === ''){
        localStreams.camera.camId = deviceId;
      }
      $('#camera-list').append('<a class="dropdown-item" id="' + optionId + '">' + name + '</a>');
    });
    $('#camera-list a').click(function(event) {
      var index = event.target.id.split('_')[1];
      changeStreamSource ({camIndex: index});
    });
  });
}

function getMicDevices() {
  console.log("Checking for Mic Devices.....")
  client.getRecordingDevices(function(mics) {
    devices.mics = mics; // store mics array
    mics.forEach(function(mic, i){
      var name = mic.label.split('(')[0];
      var optionId = 'mic_' + i;
      var deviceId = mic.deviceId;
      if(i === 0 && localStreams.camera.micId === ''){
        localStreams.camera.micId = deviceId;
      }
      if(name.split('Default - ')[1] != undefined) {
        name = '[Default Device]' // rename the default mic - only appears on Chrome & Opera
      }
      $('#mic-list').append('<a class="dropdown-item" id="' + optionId + '">' + name + '</a>');
    }); 
    $('#mic-list a').click(function(event) {
      var index = event.target.id.split('_')[1];
      changeStreamSource ({micIndex: index});
    });
  });
}

In the snippet above we store the devices in arrays to quickly switch whenever the user selects a different device. The stream.switchDevice() method allows us to pass in the deviceId and quickly switch the input device on our broadcast stream.

// user clicks on an element within the camera list
$('#camera-list a').click(function(event) {
  var index = event.target.id.split('_')[1];
  changeStreamSource (index, "video");
});

// user clicks on an element within the mic list
$('#mic-list a').click(function(event) {
  var index = event.target.id.split('_')[1];
  changeStreamSource (index, "audio");
});

// switch the input device
function changeStreamSource (deviceIndex, deviceType) {
  console.log('Switching stream sources for: ' + deviceType);
  var deviceId;
  var existingStream = false;

  if (deviceType === "video") {
    deviceId = devices.cameras[deviceIndex].deviceId
  }

  if(deviceType === "audio") {
    deviceId = devices.mics[deviceIndex].deviceId;
  }

  localStreams.camera.stream.switchDevice(deviceType, deviceId, function(){
    console.log('successfully switched to new device with id: ' + JSON.stringify(deviceId));
    // set the active device ids
    if(deviceType === "audio") {
      localStreams.camera.micId = deviceId;
    } else if (deviceType === "video") {
      localStreams.camera.camId = deviceId;
    } else {
      console.log("unable to determine deviceType: " + deviceType);
    }
  }, function(){
    console.log('failed to switch to new device with id: ' + JSON.stringify(deviceId));
  });
}

When we are done, our agora-broadcast-client.js should look like the code below.

/**
 * Agora Broadcast Client 
 */

var agoraAppId = ''; // set app id
var channelName = 'AgoraBroadcastDemo'; // set channel name

// create client instance
var client = AgoraRTC.createClient({mode: 'live', codec: 'vp8'}); // h264 better detail at a higher motion
var mainStreamId; // reference to main stream

// set video profile 
// [full list: https://docs.agora.io/en/Interactive%20Broadcast/videoProfile_web?platform=Web#video-profile-table]
var cameraVideoProfile = '720p_6'; // 960 × 720 @ 30fps  & 750kbs

// keep track of streams
var localStreams = {
  uid: '',
  camera: {
    camId: '',
    micId: '',
    stream: {}
  }
};

// keep track of devices
var devices = {
  cameras: [],
  mics: []
}

var externalBroadcastUrl = '';

// default config for rtmp
var defaultConfigRTMP = {
  width: 640,
  height: 360,
  videoBitrate: 400,
  videoFramerate: 15,
  lowLatency: false,
  audioSampleRate: 48000,
  audioBitrate: 48,
  audioChannels: 1,
  videoGop: 30,
  videoCodecProfile: 100,
  userCount: 0,
  userConfigExtraInfo: {},
  backgroundColor: 0x000000,
  transcodingUsers: [],
};

// set log level:
// -- .DEBUG for dev 
// -- .NONE for prod
AgoraRTC.Logger.setLogLevel(AgoraRTC.Logger.DEBUG); 

// init Agora SDK
client.init(agoraAppId, function () {
  console.log('AgoraRTC client initialized');
  joinChannel(); // join channel upon successfull init
}, function (err) {
  console.log('[ERROR] : AgoraRTC client init failed', err);
});

// client callbacks
client.on('stream-published', function (evt) {
  console.log('Publish local stream successfully');
});

// when a remote stream is added
client.on('stream-added', function (evt) {
  console.log('new stream added: ' + evt.stream.getId());
});

client.on('stream-removed', function (evt) {
  var stream = evt.stream;
  stream.stop(); // stop the stream
  stream.close(); // clean up and close the camera stream
  console.log("Remote stream is removed " + stream.getId());
});

//live transcoding events..
client.on('liveStreamingStarted', function (evt) {
  console.log("Live streaming started");
}); 

client.on('liveStreamingFailed', function (evt) {
  console.log("Live streaming failed");
}); 

client.on('liveStreamingStopped', function (evt) {
  console.log("Live streaming stopped");
});

client.on('liveTranscodingUpdated', function (evt) {
  console.log("Live streaming updated");
}); 

// ingested live stream 
client.on('streamInjectedStatus', function (evt) {
  console.log("Injected Steram Status Updated");
  console.log(JSON.stringify(evt));
}); 

// when a remote stream leaves the channel
client.on('peer-leave', function(evt) {
  console.log('Remote stream has left the channel: ' + evt.stream.getId());
});

// show mute icon whenever a remote has muted their mic
client.on('mute-audio', function (evt) {
  console.log('Mute Audio for: ' + evt.uid);
});

client.on('unmute-audio', function (evt) {
  console.log('Unmute Audio for: ' + evt.uid);
});

// show user icon whenever a remote has disabled their video
client.on('mute-video', function (evt) {
  console.log('Mute Video for: ' + evt.uid);
});

client.on('unmute-video', function (evt) {
  console.log('Unmute Video for: ' + evt.uid);
});

// join a channel
function joinChannel() {
  var token = generateToken();
  var userID = 0; // set to null to auto generate uid on successfull connection

  // set the role
  client.setClientRole('host', function() {
    console.log('Client role set as host.');
  }, function(e) {
    console.log('setClientRole failed', e);
  });

  // client.join(token, 'allThingsRTCLiveStream', 0, function(uid) {
  client.join(token, channelName, userID, function(uid) {
      createCameraStream(uid, {});
      localStreams.uid = uid; // keep track of the stream uid  
      console.log('User ' + uid + ' joined channel successfully');
  }, function(err) {
      console.log('[ERROR] : join channel failed', err);
  });
}

// video streams for channel
function createCameraStream(uid, deviceIds) {
  console.log('Creating stream with sources: ' + JSON.stringify(deviceIds));

  var localStream = AgoraRTC.createStream({
    streamID: uid,
    audio: true,
    video: true,
    screen: false
  });
  localStream.setVideoProfile(cameraVideoProfile);

  // The user has granted access to the camera and mic.
  localStream.on("accessAllowed", function() {
    if(devices.cameras.length === 0 && devices.mics.length === 0) {
      console.log('[DEBUG] : checking for cameras & mics');
      getCameraDevices();
      getMicDevices();
    }
    console.log("accessAllowed");
  });
  // The user has denied access to the camera and mic.
  localStream.on("accessDenied", function() {
    console.log("accessDenied");
  });

  localStream.init(function() {
    console.log('getUserMedia successfully');
    localStream.play('full-screen-video'); // play the local stream on the main div
    // publish local stream

    if($.isEmptyObject(localStreams.camera.stream)) {
      enableUiControls(localStream); // move after testing
    } else {
      //reset controls
      $("#mic-btn").prop("disabled", false);
      $("#video-btn").prop("disabled", false);
      $("#exit-btn").prop("disabled", false);
    }

    client.publish(localStream, function (err) {
      console.log('[ERROR] : publish local stream error: ' + err);
    });

    localStreams.camera.stream = localStream; // keep track of the camera stream for later
  }, function (err) {
    console.log('[ERROR] : getUserMedia failed', err);
  });
}

function leaveChannel() {

  client.leave(function() {
    console.log('client leaves channel');
    localStreams.camera.stream.stop() // stop the camera stream playback
    localStreams.camera.stream.close(); // clean up and close the camera stream
    client.unpublish(localStreams.camera.stream); // unpublish the camera stream
    //disable the UI elements
    $('#mic-btn').prop('disabled', true);
    $('#video-btn').prop('disabled', true);
    $('#exit-btn').prop('disabled', true);
    $("#add-rtmp-btn").prop("disabled", true);
    $("#rtmp-config-btn").prop("disabled", true);
  }, function(err) {
    console.log('client leave failed ', err); //error handling
  });
}

// use tokens for added security
function generateToken() {
  return null; // TODO: add a token generation
}

function changeStreamSource (deviceIndex, deviceType) {
  console.log('Switching stream sources for: ' + deviceType);
  var deviceId;
  var existingStream = false;

  if (deviceType === "video") {
    deviceId = devices.cameras[deviceIndex].deviceId
  }

  if(deviceType === "audio") {
    deviceId = devices.mics[deviceIndex].deviceId;
  }

  localStreams.camera.stream.switchDevice(deviceType, deviceId, function(){
    console.log('successfully switched to new device with id: ' + JSON.stringify(deviceId));
    // set the active device ids
    if(deviceType === "audio") {
      localStreams.camera.micId = deviceId;
    } else if (deviceType === "video") {
      localStreams.camera.camId = deviceId;
    } else {
      console.log("unable to determine deviceType: " + deviceType);
    }
  }, function(){
    console.log('failed to switch to new device with id: ' + JSON.stringify(deviceId));
  });
}

// helper methods
function getCameraDevices() {
  console.log("Checking for Camera Devices.....")
  client.getCameras (function(cameras) {
    devices.cameras = cameras; // store cameras array
    cameras.forEach(function(camera, i){
      var name = camera.label.split('(')[0];
      var optionId = 'camera_' + i;
      var deviceId = camera.deviceId;
      if(i === 0 && localStreams.camera.camId === ''){
        localStreams.camera.camId = deviceId;
      }
      $('#camera-list').append('<a class="dropdown-item" id="' + optionId + '">' + name + '</a>');
    });
    $('#camera-list a').click(function(event) {
      var index = event.target.id.split('_')[1];
      changeStreamSource (index, "video");
    });
  });
}

function getMicDevices() {
  console.log("Checking for Mic Devices.....")
  client.getRecordingDevices(function(mics) {
    devices.mics = mics; // store mics array
    mics.forEach(function(mic, i){
      var name = mic.label.split('(')[0];
      var optionId = 'mic_' + i;
      var deviceId = mic.deviceId;
      if(i === 0 && localStreams.camera.micId === ''){
        localStreams.camera.micId = deviceId;
      }
      if(name.split('Default - ')[1] != undefined) {
        name = '[Default Device]' // rename the default mic - only appears on Chrome & Opera
      }
      $('#mic-list').append('<a class="dropdown-item" id="' + optionId + '">' + name + '</a>');
    }); 
    $('#mic-list a').click(function(event) {
      var index = event.target.id.split('_')[1];
      changeStreamSource (index, "audio");
    });
  });
}

function startLiveTranscoding() {
  console.log("start live transcoding"); 
  var rtmpUrl = $('#rtmp-url').val();
  var width = parseInt($('#window-scale-width').val(), 10);
  var height = parseInt($('#window-scale-height').val(), 10);

  var configRtmp = {
    width: width,
    height: height,
    videoBitrate: parseInt($('#video-bitrate').val(), 10),
    videoFramerate: parseInt($('#framerate').val(), 10),
    lowLatency: ($('#low-latancy').val() === 'true'),
    audioSampleRate: parseInt($('#audio-sample-rate').val(), 10),
    audioBitrate: parseInt($('#audio-bitrate').val(), 10),
    audioChannels: parseInt($('#audio-channels').val(), 10),
    videoGop: parseInt($('#video-gop').val(), 10),
    videoCodecProfile: parseInt($('#video-codec-profile').val(), 10),
    userCount: 1,
    userConfigExtraInfo: {},
    backgroundColor: parseInt($('#background-color-picker').val(), 16),
    transcodingUsers: [{
      uid: localStreams.uid,
      alpha: 1,
      width: width,
      height: height,
      x: 0,
      y: 0,
      zOrder: 0
    }],
  };

  // set live transcoding config
  client.setLiveTranscoding(configRtmp);
  if(rtmpUrl !== '') {
    client.startLiveStreaming(rtmpUrl, true)
    externalBroadcastUrl = rtmpUrl;
    addExternalTransmitionMiniView(rtmpUrl)
  }
}

function addExternalSource() {
  var externalUrl = $('#external-url').val();
  var width = parseInt($('#external-window-scale-width').val(), 10);
  var height = parseInt($('#external-window-scale-height').val(), 10);

  var injectStreamConfig = {
    width: width,
    height: height,
    videoBitrate: parseInt($('#external-video-bitrate').val(), 10),
    videoFramerate: parseInt($('#external-framerate').val(), 10),
    audioSampleRate: parseInt($('#external-audio-sample-rate').val(), 10),
    audioBitrate: parseInt($('#external-audio-bitrate').val(), 10),
    audioChannels: parseInt($('#external-audio-channels').val(), 10),
    videoGop: parseInt($('#external-video-gop').val(), 10)
  };

  // set live transcoding config
  client.addInjectStreamUrl(externalUrl, injectStreamConfig)
  injectedStreamURL = externalUrl;
  // TODO: ADD view for external url (similar to rtmp url)
}

// RTMP Connection (UI Component)
function addExternalTransmitionMiniView(rtmpUrl){
  var container = $('#rtmp-controlers');
  // append the remote stream template to #remote-streams
  container.append(
    $('<div/>', {'id': 'rtmp-container',  'class': 'container row justify-content-end mb-2'}).append(
      $('<div/>', {'class': 'pulse-container'}).append(
          $('<button/>', {'id': 'rtmp-toggle', 'class': 'btn btn-lg col-flex pulse-button pulse-anim mt-2'})
      ),
      $('<input/>', {'id': 'rtmp-url', 'val': rtmpUrl, 'class': 'form-control col-flex" value="rtmps://live.facebook.com', 'type': 'text', 'disabled': true}),
      $('<button/>', {'id': 'removeRtmpUrl', 'class': 'btn btn-lg col-flex close-btn'}).append(
        $('<i/>', {'class': 'fas fa-xs fa-trash'})
      )
    )
  );

  $('#rtmp-toggle').click(function() {
    if ($(this).hasClass('pulse-anim')) {
      client.stopLiveStreaming(externalBroadcastUrl)
    } else {
      client.startLiveStreaming(externalBroadcastUrl, true)
    }
    $(this).toggleClass('pulse-anim');
    $(this).blur();
  });

  $('#removeRtmpUrl').click(function() { 
    client.stopLiveStreaming(externalBroadcastUrl);
    externalBroadcastUrl = '';
    $('#rtmp-container').remove();
  });

}

The Audience client is much simpler. We have most of the same event listeners but in our use-case we will only need a handful of them. We’ll include the extra ones in the even that we want to extend the demo at a later date. Below is the full implementation of our agora-audience-client.js.

/**
 * Agora Broadcast Client 
 */

var agoraAppId = ''; // set app id
var channelName = 'AgoraBroadcastDemo'; // set channel name

// create client 
var client = AgoraRTC.createClient({mode: 'live', codec: 'vp8'}); // vp8 to work across mobile devices

// set log level:
// -- .DEBUG for dev 
// -- .NONE for prod
AgoraRTC.Logger.setLogLevel(AgoraRTC.Logger.DEBUG); 

// Due to broswer restrictions on auto-playing video, 
// user must click to init and join channel
$("#watch-live-btn").click(function(){
  // init Agora SDK
  client.init(agoraAppId, function () {
    $("#watch-live-overlay").remove();
    console.log('AgoraRTC client initialized');
    joinChannel(); // join channel upon successfull init
  }, function (err) {
    console.log('[ERROR] : AgoraRTC client init failed', err);
  });
});

client.on('stream-published', function (evt) {
  console.log('Publish local stream successfully');
});

// connect remote streams
client.on('stream-added', function (evt) {
  var stream = evt.stream;
  var streamId = stream.getId();
  console.log('New stream added: ' + streamId);
  console.log('Subscribing to remote stream:' + streamId);
  // Subscribe to the stream.
  client.subscribe(stream, function (err) {
    console.log('[ERROR] : subscribe stream failed', err);
  });
});

client.on('stream-removed', function (evt) {
  var stream = evt.stream;
  stream.stop(); // stop the stream
  stream.close(); // clean up and close the camera stream
  console.log("Remote stream is removed " + stream.getId());
});

client.on('stream-subscribed', function (evt) {
  var remoteStream = evt.stream;
  remoteStream.play('full-screen-video');
  console.log('Successfully subscribed to remote stream: ' + remoteStream.getId());
});

// remove the remote-container when a user leaves the channel
client.on('peer-leave', function(evt) {
  console.log('Remote stream has left the channel: ' + evt.uid);
  evt.stream.stop(); // stop the stream
});

// show mute icon whenever a remote has muted their mic
client.on('mute-audio', function (evt) {
  var remoteId = evt.uid;
});

client.on('unmute-audio', function (evt) {
  var remoteId = evt.uid;
});

// show user icon whenever a remote has disabled their video
client.on('mute-video', function (evt) {
  var remoteId = evt.uid;
});

client.on('unmute-video', function (evt) {
  var remoteId = evt.uid;
});

// ingested live stream 
client.on('streamInjectedStatus', function (evt) {
  console.log("Injected Steram Status Updated");
  // evt.stream.play('full-screen-video');
  console.log(JSON.stringify(evt));
}); 

// join a channel
function joinChannel() {
  var token = generateToken();

  // set the role
  client.setClientRole('audience', function() {
    console.log('Client role set to audience');
  }, function(e) {
    console.log('setClientRole failed', e);
  });

  client.join(token, channelName, 0, function(uid) {
      console.log('User ' + uid + ' join channel successfully');
  }, function(err) {
      console.log('[ERROR] : join channel failed', err);
  });
}

function leaveChannel() {
  client.leave(function() {
    console.log('client leaves channel');
  }, function(err) {
    console.log('client leave failed ', err); //error handling
  });
}

// use tokens for added security
function generateToken() {
  return null; // TODO: add a token generation
}

The last piece of Javascript that we need to implement is the Ui.js to add enable the UI elements to interact with the agora-broadcast-client.js.


// UI buttons
function enableUiControls() {

  $("#mic-btn").prop("disabled", false);
  $("#video-btn").prop("disabled", false);
  $("#exit-btn").prop("disabled", false);
  $("#add-rtmp-btn").prop("disabled", false);

  $("#mic-btn").click(function(){
    toggleMic();
  });

  $("#video-btn").click(function(){
    toggleVideo();
  });

  $("#exit-btn").click(function(){
    console.log("so sad to see you leave the channel");
    leaveChannel(); 
  });

  $("#start-RTMP-broadcast").click(function(){
    startLiveTranscoding();
    $('#addRtmpConfigModal').modal('toggle');
    $('#rtmp-url').val('');
  });

  $("#add-external-stream").click(function(){  
    addExternalSource();
    $('#add-external-source-modal').modal('toggle');
  });

  // keyboard listeners 
  $(document).keypress(function(e) {
    // ignore keyboard events when the modals are open
    if (($("#addRtmpUrlModal").data('bs.modal') || {})._isShown ||
        ($("#addRtmpConfigModal").data('bs.modal') || {})._isShown){
      return;
    }

    switch (e.key) {
      case "m":
        console.log("squick toggle the mic");
        toggleMic();
        break;
      case "v":
        console.log("quick toggle the video");
        toggleVideo();
        break; 
      case "q":
        console.log("so sad to see you quit the channel");
        leaveChannel(); 
        break;   
      default:  // do nothing
    }
  });
}

function toggleBtn(btn){
  btn.toggleClass('btn-dark').toggleClass('btn-danger');
}

function toggleVisibility(elementID, visible) {
  if (visible) {
    $(elementID).attr("style", "display:block");
  } else {
    $(elementID).attr("style", "display:none");
  }
}

function toggleMic() {
  toggleBtn($("#mic-btn")); // toggle button colors
  toggleBtn($("#mic-dropdown"));
  $("#mic-icon").toggleClass('fa-microphone').toggleClass('fa-microphone-slash'); // toggle the mic icon
  if ($("#mic-icon").hasClass('fa-microphone')) {
    localStreams.camera.stream.unmuteAudio(); // enable the local mic
  } else {
    localStreams.camera.stream.muteAudio(); // mute the local mic
  }
}

function toggleVideo() {
  toggleBtn($("#video-btn")); // toggle button colors
  toggleBtn($("#cam-dropdown"));
  if ($("#video-icon").hasClass('fa-video')) {
    localStreams.camera.stream.muteVideo(); // enable the local video
    console.log("muteVideo");
  } else {
    localStreams.camera.stream.unmuteVideo(); // disable the local video
    console.log("unMuteVideo");
  }
  $("#video-icon").toggleClass('fa-video').toggleClass('fa-video-slash'); // toggle the video icon
}

// keep the spinners honest
$("input[type='number']").change(event, function() {
  var maxValue = $(this).attr("max");
  var minValue = $(this).attr("min");
  if($(this).val() > maxValue) {
    $(this).val(maxValue);
  } else if($(this).val() < minValue) {
    $(this).val(minValue);
  }
});

// keep the background color as a proper hex
$("#background-color-picker").change(event, function() {
  // check the background color
  var backgroundColorPicker = $(this).val();
  if (backgroundColorPicker.split('#').length > 1){
    backgroundColorPicker = '0x' + backgroundColorPicker.split('#')[1];
    $('#background-color-picker').val(backgroundColorPicker);
  } 
});

Testing Setup (webserver/https)

There are a few different ways to test and deploy our broadcast web-app. We can use localhost but if we want to share with our friends we must spin up a simple web server with a https connection. This is due to browser restricts when accessing userMedia resources like the camera and microphone.

To keep things simple, I like to use Live Server to enable simple web server locally in conjunction with ngrok, a service that creates a tunnel out from your local machine and provides an https url for use. In my experience, this is one of the simplest ways to run an https secured web server on your local machine.

Once the server is ready we can share our ngrok link with a friend or two and run our test broadcast.

NOTE: For testing I am using an iPad and iPhone as the audience, and my laptop is the broadcaster.

Thats a wrap

Thanks for following along. Now it’s your turn to go and build something amazing!

Thanks for taking the time to read my tutorial and if you have any questions please let me know with a comment. If you see any room for improvement feel free to fork the repo and make a pull request!

agora-web-broadcast-demo repo: https://github.com/digitallysavvy/agora-web-broadcast-demo

Build a group video chat web-app

Hermes Frangoudis — Thu, 04 Jun 2020 18:19:10 +0000

Hey everyone, today I want to walk through how to build a simple group video chat web app, very similar to Google Hangouts, Skype or whichever other video chat platform you prefer. Given today’s fragmented JS landscape, I wanted to write this tutorial using the most basic versions of HTML, CSS and JS. Before you say it, I know I know, JQuery isn’t vanilla JS but Vanilla JS can be a bit verbose for certain DOM tasks, I chose to use JQuery to simplify a few things. We are going to cut a few corners and use Bootstrap so we don’t have to worry about writing too much custom CSS.

For the TLDR crowd: Check out the demo of the code in action on GitHub Pages .

Pre Requisites

A simple web server — I like to use Live Server
An SSL cert or way to have an https connection (I use ngrok)
A developer account with Agora.io
An understanding of HTML/CSS/JS
An understanding of how Bootstrap and JQuery function (minimal knowledge needed)

Core Structure (HTML)

Let’s start by laying out our basic html structure. There are a few UI elements we must have, such as the local video stream, the remote video streams, a toolbar that will contain buttons for toggling audio/video streams, a button to share our screen with the group, and lastly a way to leave the chat (we’ll add the buttons a little later).

<html lang="en">
  <head>
    <title>Agora Group Video Chat Demo</title>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
  </head>
  <body>
    <div id="container">
      <div id="main-container">
        <div id="screen-share-btn-container">
          <!-- insert button to share screen -->
        </div>
        <div id="buttons-container">
          <!-- insert buttons to toggle audio/video and leave/end call -->
        </div>
        <div id="full-screen-video"></div>
        <div id="lower-video-bar">
          <div id="remote-streams-container">
            <div id="remote-streams">
              <!-- insert remote streams dynamically -->
            </div>
          </div>
          <div id="local-stream-container">                     
            <div id="local-video"></div>
          </div>
        </div>
      </div>
    </div>
  </body>
  <!-- CSS Includes go here -->
  <!-- JS Includes go here -->
</html>

Adding in CSS and JS

Now that we have our base we can start expanding. Using Bootstrap for our CSS we can quickly style our html with a few simple classes. In the above code, let's add the CSS links (shown below) into the code where we see the comment block .

<!-- Bootstrap and Font Awesome CSS Libraries -->
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.7.0/css/all.css" integrity="sha384-lZN37f5QGtY3VHgisS14W3ExzMWZxybE1SJSEsQp9S+oqd12jhcu+A56Ebc1zFSJ" crossorigin="anonymous">
<link href="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/css/bootstrap.min.css" rel="stylesheet">
<link rel="stylesheet" type="text/css" href="style.css"/>

While Boostrap is great but it isn’t a holistic solution, so I threw in a few extras CSS blocks within a custom CSS file (we’ll get to this a little later). This will help adjust a few elements that we won’t get perfect out of the box with Bootstrap. I also added the Font Awesome CSS framework because we are going to need to incorporate icons for the various buttons and FA makes it really simple.

As I mentioned, Bootstrap is great, but sometimes you still need a little bit of custom CSS. Here are the styling blocks for the above referenced style.css.

#buttons-container {
  position: absolute;
  z-index: 2;  
  width: 100vw;
}

#full-screen-video {
  position: absolute;
  width: 100vw;
  height: 100vh;
}

#lower-video-bar {
  height: 20vh;
}

#local-stream-container { 
  position: relative; 
  display: inline-block;
}

.remote-stream-container { 
  display: inline-block;
}

#remote-streams {
  height: 100%;
}

#local-video {
  position: absolute;
  z-index: 1;
  height: 20vh;
  max-width: 100%;
}

.remote-video {
  position: absolute;
  z-index: 1;
  height: 100% !important;
  width: 80%;
  max-width: 500px;
}

#mute-overlay {
  position: absolute;
  z-index: 2;
  bottom: 0;
  left: 0;
  color: #d9d9d9;
  font-size: 2em;
  padding: 0 0 3px 3px;
  display: none;
} 

.mute-overlay {
  position: absolute;
  z-index: 2;
  top: 2px;
  color: #d9d9d9;
  font-size: 1.5em;
  padding: 2px 0 0 2px;
  display: none;
}

#no-local-video, .no-video-overlay {
  position: absolute;
  z-index: 3;
  width: 100%;
  top: 40%;
  color: #cccccc;
  font-size: 2.5em;
  margin: 0 auto;
  display: none;
}

.no-video-overlay {
  width: 80%;
}

#screen-share-btn-container {
  z-index: 99;
}

Adding UI elements

Now let’s add some buttons to control toggling the mic, video or leaving the channel and finish off the last remaining bits of our UI. This is where font awesome and bootstrap really make things simple. We will use a <button /> element and some FontAwesome icons.

The sections below fits in with the code above by replacing the comments
 and


<div id="screen-share-btn-container" class="col-2 float-right text-right mt-2">
    <button id="screen-share-btn"  type="button" class="btn btn-lg">
        <i id="screen-share-icon" class="fas fa-share-square"></i>
    </button>
</div>
<div id="buttons-container" class="row justify-content-center mt-3">
    <div class="col-md-2 text-center">
        <button id="mic-btn" type="button" class="btn btn-block btn-dark btn-lg">
            <i id="mic-icon" class="fas fa-microphone"></i>
        </button>
    </div>
    <div class="col-md-2 text-center">
        <button id="video-btn"  type="button" class="btn btn-block btn-dark btn-lg">
            <i id="video-icon" class="fas fa-video"></i>
        </button>
    </div>
    <div class="col-md-2 text-center">
        <button id="exit-btn"  type="button" class="btn btn-block btn-danger btn-lg">
            <i id="exit-icon" class="fas fa-phone-slash"></i>
        </button>
    </div>
</div>

We need to add some JS to control the buttons. JQuery will really help us here by simplifying the code for the various DOM operations which will allow the UI to feel dynamic for the user.

// UI buttons
function enableUiControls(localStream) {

  $("#mic-btn").prop("disabled", false);
  $("#video-btn").prop("disabled", false);
  $("#screen-share-btn").prop("disabled", false);
  $("#exit-btn").prop("disabled", false);

  $("#mic-btn").click(function(){
    toggleMic(localStream);
  });

  $("#video-btn").click(function(){
    toggleVideo(localStream);
  });

  $("#screen-share-btn").click(function(){
    toggleScreenShareBtn(); // set screen share button icon
    $("#screen-share-btn").prop("disabled",true); // disable the button on click
    if(screenShareActive){
      stopScreenShare();
    } else {
      initScreenShare(); 
    }
  });

  $("#exit-btn").click(function(){
    console.log("so sad to see you leave the channel");
    leaveChannel(); 
  });

  // keyboard listeners 
  $(document).keypress(function(e) {
    switch (e.key) {
      case "m":
        console.log("squick toggle the mic");
        toggleMic(localStream);
        break;
      case "v":
        console.log("quick toggle the video");
        toggleVideo(localStream);
        break; 
      case "s":
        console.log("initializing screen share");
        toggleScreenShareBtn(); // set screen share button icon
        $("#screen-share-btn").prop("disabled",true); // disable the button on click
        if(screenShareActive){
          stopScreenShare();
        } else {
          initScreenShare(); 
        }
        break;  
      case "q":
        console.log("so sad to see you quit the channel");
        leaveChannel(); 
        break;   
      default:  // do nothing
    }

    // (for testing) 
    if(e.key === "r") { 
      window.history.back(); // quick reset
    }
  });
}

function toggleBtn(btn){
  btn.toggleClass('btn-dark').toggleClass('btn-danger');
}

function toggleScreenShareBtn() {
  $('#screen-share-btn').toggleClass('btn-danger');
  $('#screen-share-icon').toggleClass('fa-share-square').toggleClass('fa-times-circle');
}

function toggleVisibility(elementID, visible) {
  if (visible) {
    $(elementID).attr("style", "display:block");
  } else {
    $(elementID).attr("style", "display:none");
  }
}

function toggleMic(localStream) {
  toggleBtn($("#mic-btn")); // toggle button colors
  $("#mic-icon").toggleClass('fa-microphone').toggleClass('fa-microphone-slash'); // toggle the mic icon
  if ($("#mic-icon").hasClass('fa-microphone')) {
    localStream.enableAudio(); // enable the local mic
    toggleVisibility("#mute-overlay", false); // hide the muted mic icon
  } else {
    localStream.disableAudio(); // mute the local mic
    toggleVisibility("#mute-overlay", true); // show the muted mic icon
  }
}

function toggleVideo(localStream) {
  toggleBtn($("#video-btn")); // toggle button colors
  $("#video-icon").toggleClass('fa-video').toggleClass('fa-video-slash'); // toggle the video icon
  if ($("#video-icon").hasClass('fa-video')) {
    localStream.enableVideo(); // enable the local video
    toggleVisibility("#no-local-video", false); // hide the user icon when video is enabled
  } else {
    localStream.disableVideo(); // disable the local video
    toggleVisibility("#no-local-video", true); // show the user icon when video is disabled
  }
}

As you can see there is some added logic for keyboard controls. During testing I found having keyboard shortcuts made things move quicker. In the snippet above we have support for m, v, s, q to toggle mic, video, and screen-share and to leave the call (respectively).

I saved the above code into a file ui.js to keep it separate from the core video chat logic that we will write. Also let’s make sure to include the ui.js file within our html file (using the snippet below).

<script src="ui.js"></script>

Core Structure (JS)

Now that we the HTML/DOM structure laid out we can add in the JS. I chose to use Agora.io to simplify the heavy task of the WebRTC interface. I wrote a short post on how to get setup with Agora.io for anyone new to the Agora.io platform. In the code below we start by declaring and initializing the Client object. Once we have the Client object we can join/leave the channel but also we will add listeners for the various engine events.

Below I included some of the initial object declarations for the screen sharing. I will expand upon that implementation later on, as we add in the rest of the logic.

// app / channel settings
var agoraAppId = ""; // Set your Agora App ID
var channelName = 'agora-web-docs-demo';

// video profile settings
var cameraVideoProfile = '480_4'; // 640 × 480 @ 30fps  & 750kbs
var screenVideoProfile = '480_2'; // 640 × 480 @ 30fps

// create client instances for camera (client) and screen share (screenClient)
var client = AgoraRTC.createClient({mode: 'rtc', codec: "h264"}); // h264 better detail at a higher motion
var screenClient = AgoraRTC.createClient({mode: 'rtc', codec: 'vp8'}); // use the vp8 for better detail in low motion

// stream references (keep track of active streams) 
var remoteStreams = {}; // remote streams obj struct [id : stream] 

var localStreams = {
  camera: {
    id: "",
    stream: {}
  },
  screen: {
    id: "",
    stream: {}
  }
};

var mainStreamId; // reference to main stream
var screenShareActive = false; // flag for screen share 

// init Agora SDK
client.init(agoraAppId, function () {
  console.log("AgoraRTC client initialized");
  joinChannel(); // join channel upon successfull init
}, function (err) {
  console.log("[ERROR] : AgoraRTC client init failed", err);
});

client.on('stream-published', function (evt) {
  console.log("Publish local stream successfully");
});

// connect remote streams
client.on('stream-added', function (evt) {
  console.log("new stream added: " + streamId);
  // Check if the stream is local
  if (streamId != localStreams.screen.id) {
    console.log('subscribe to remote stream:' + streamId);
    // Subscribe to the stream.
    client.subscribe(stream, function (err) {
      console.log("[ERROR] : subscribe stream failed", err);
    });
  }
});

client.on('stream-subscribed', function (evt) {
  console.log("Subscribe remote stream successfully: " + evt.stream.getId());
});

// remove the remote-container when a user leaves the channel
client.on("peer-leave", function(evt) {
  console.log("Remote stream: " + evt.stream.getId() + "has left");
});

// show mute icon whenever a remote has muted their mic
client.on("mute-audio", function (evt) {
  console.log("Remote stream: " +  evt.uid + "has muted audio");
});

client.on("unmute-audio", function (evt) {
  console.log("Remote stream: " +  evt.uid + "has muted audio");
});

// show user icon whenever a remote has disabled their video
client.on("mute-video", function (evt) {
  console.log("Remote stream: " +  evt.uid + "has muted video");
});

client.on("unmute-video", function (evt) {
  console.log("Remote stream: " +  evt.uid + "has un-muted video");
});

// join a channel
function joinChannel() {
  var token = generateToken();
  var userID = null; // set to null to auto generate uid on successfull connection
  client.join(token, channelName, userID, function(uid) {
      console.log("User " + uid + " join channel successfully");
      createCameraStream(uid);
      localStreams.camera.id = uid; // keep track of the stream uid 
  }, function(err) {
      console.log("[ERROR] : join channel failed", err);
  });
}

// video streams for channel
function createCameraStream(uid) {
  var localStream = AgoraRTC.createStream({
    streamID: uid,
    audio: true,
    video: true,
    screen: false
  });
  localStream.setVideoProfile(cameraVideoProfile);
  localStream.init(function() {
    console.log("getUserMedia successfully");
    // TODO: add check for other streams. play local stream full size if alone in channel
    localStream.play('local-video'); // play the given stream within the local-video div
    // publish local stream
    client.publish(localStream, function (err) {
      console.log("[ERROR] : publish local stream error: " + err);
    });

    enableUiControls(localStream); // move after testing
    localStreams.camera.stream = localStream; // keep track of the camera stream for later
  }, function (err) {
    console.log("[ERROR] : getUserMedia failed", err);
  });
}

function leaveChannel() {
  client.leave(function() {
    console.log("client leaves channel");
  }, function(err) {
    console.log("client leave failed ", err); //error handling
  });
}

// use tokens for added security
function generateToken() {
  return null; // TODO: add a token generation
}

One thing to note, all the Agora.io SDK event listeners should be at the top level, please don’t make the mistake of nesting them into the channel join callback. I made this mistake and it caused me to only have access to streams that joined the channel after me.

As you can see within the code above we have the 'stream-added' callback, this is where we will add logic to handle setting the first remote stream to the full screen video and every subsequent stream into a new div container within the remote-streams div which will give us the group functionality beyond just 1 to 1 video. Below is the function we would call every time a new remote stream is added and we want to have it add itself dynamically to the DOM.

// REMOTE STREAMS UI
function addRemoteStreamMiniView(remoteStream){
  var streamId = remoteStream.getId();
  // append the remote stream template to #remote-streams
  $('#remote-streams').append(
    $('<div/>', {'id': streamId + '_container',  'class': 'remote-stream-container col'}).append(
      $('<div/>', {'id': streamId + '_mute', 'class': 'mute-overlay'}).append(
          $('<i/>', {'class': 'fas fa-microphone-slash'})
      ),
      $('<div/>', {'id': streamId + '_no-video', 'class': 'no-video-overlay text-center'}).append(
        $('<i/>', {'class': 'fas fa-user'})
      ),
      $('<div/>', {'id': 'agora_remote_' + streamId, 'class': 'remote-video'})
    )
  );
  remoteStream.play('agora_remote_' + streamId); 
}

One last note for this section, we have buttons that toggle the mic and video streams but we need to provide feedback to the remote users subscribed to the muted streams. Don’t worry Agora’s SDK provides some callbacks specially for these situations. Above you can see these cases are handled by the events such as mute-audio or mute-video (as well as their inverses for enabling the respective streams).

Enhancing the UI by handling remote stream actions

First let’s start by adding some extra divs with icons for a muted mic and a user icon when the video feed is disabled. I will use the local container as a reference as the remote stream containers will have similar structure.

<div id="local-stream-container" class="col p-0">
  <div id="mute-overlay" class="col">
    <i id="mic-icon" class="fas fa-microphone-slash"></i>
  </div>
  <div id="no-local-video" class="col text-center">
    <i id="user-icon" class="fas fa-user"></i>
  </div>                            
  <div id="local-video" class="col p-0"></div>
</div>

The new divs will hold some FontAwesome icons that we can hide/show whenever the event callbacks are executed for on the local and corresponding remote streams. Now that we have some names for our elements we can easily control them within our event listeners.

// show mute icon whenever a remote has muted their mic
client.on("mute-audio", function (evt) {
  toggleVisibility('#' + evt.uid + '_mute', true);
});

client.on("unmute-audio", function (evt) {
  toggleVisibility('#' + evt.uid + '_mute', false);
});

// show user icon whenever a remote has disabled their video
client.on("mute-video", function (evt) {
  var remoteId = evt.uid;
  // if the main user stops their video select a random user from the list
  if (remoteId != mainStreamId) {
    // if not the main vidiel then show the user icon
    toggleVisibility('#' + remoteId + '_no-video', true);
  }
});

client.on("unmute-video", function (evt) {
  toggleVisibility('#' + evt.uid + '_no-video', false);
});

More frills

There’s a few effects that we can add to really enhance the user experience. First let’s consider what happens when the user wants a different stream to be the full screen. We’ll add a double click listener to each remote stream so when the user double clicks a remote stream it swaps the mini view with the full screen view.

var containerId = '#' + streamId + '_container';
$(containerId).dblclick(function() {
  // play selected container as full screen - swap out current full screen stream
  remoteStreams[mainStreamId].stop(); // stop the main video stream playback
  addRemoteStreamMiniView(remoteStreams[mainStreamId]); // send the main video stream to a container
  $(containerId).empty().remove(); // remove the stream's miniView container
  remoteStreams[streamId].stop() // stop the container's video stream playback
  remoteStreams[streamId].play('full-screen-video'); // play the remote stream as the full screen video
  mainStreamId = streamId; // set the container stream id as the new main stream id
});

Lastly let’s make sure that there is always a full screen stream as long as at least one stream is connected. We can use some similar methods as we did above.

// remove the remote-container when a user leaves the channel
client.on("peer-leave", function(evt) {
  var streamId = evt.stream.getId(); // the the stream id
  if(remoteStreams[streamId] != undefined) {
    remoteStreams[streamId].stop(); // stop playing the feed
    delete remoteStreams[streamId]; // remove stream from list
    if (streamId == mainStreamId) {
      var streamIds = Object.keys(remoteStreams);
      var randomId = streamIds[Math.floor(Math.random()*streamIds.length)]; // select from the remaining streams
      remoteStreams[randomId].stop(); // stop the stream's existing playback
      var remoteContainerID = '#' + randomId + '_container';
      $(remoteContainerID).empty().remove(); // remove the stream's miniView container
      remoteStreams[randomId].play('full-screen-video'); // play the random stream as the main stream
      mainStreamId = randomId; // set the new main remote stream
    } else {
      var remoteContainerID = '#' + streamId + '_container';
      $(remoteContainerID).empty().remove(); // 
    }
  }
});

I added some randomization so when the the full screen remote stream leaves the channel, one of the other remote streams is randomly selected and set to play in the full screen div.

Putting it all together

Now that we have all these snippets lets put them together and fill in the rest of the logic for how the web app should react to each event.

// simple JS interface for Agora.io web SDK

// app / channel settings
var agoraAppId = " "; // Set your Agora App ID
var channelName = 'agora-web-docs-demo';

// video profile settings
var cameraVideoProfile = '480_4'; // 640 × 480 @ 30fps  & 750kbs
var screenVideoProfile = '480_2'; // 640 × 480 @ 30fps

// create client instances for camera (client) and screen share (screenClient)
var client = AgoraRTC.createClient({mode: 'rtc', codec: "h264"}); // h264 better detail at a higher motion
var screenClient = AgoraRTC.createClient({mode: 'rtc', codec: 'vp8'}); // use the vp8 for better detail in low motion

// stream references (keep track of active streams) 
var remoteStreams = {}; // remote streams obj struct [id : stream] 

var localStreams = {
  camera: {
    id: "",
    stream: {}
  },
  screen: {
    id: "",
    stream: {}
  }
};

var mainStreamId; // reference to main stream
var screenShareActive = false; // flag for screen share 

// init Agora SDK
client.init(agoraAppId, function () {
  console.log("AgoraRTC client initialized");
  joinChannel(); // join channel upon successfull init
}, function (err) {
  console.log("[ERROR] : AgoraRTC client init failed", err);
});

client.on('stream-published', function (evt) {
  console.log("Publish local stream successfully");
});

// connect remote streams
client.on('stream-added', function (evt) {
  var stream = evt.stream;
  var streamId = stream.getId();
  console.log("new stream added: " + streamId);
  // Check if the stream is local
  if (streamId != localStreams.screen.id) {
    console.log('subscribe to remote stream:' + streamId);
    // Subscribe to the stream.
    client.subscribe(stream, function (err) {
      console.log("[ERROR] : subscribe stream failed", err);
    });
  }
});

client.on('stream-subscribed', function (evt) {
  var remoteStream = evt.stream;
  var remoteId = remoteStream.getId();
  remoteStreams[remoteId] = remoteStream;
  console.log("Subscribe remote stream successfully: " + remoteId);
  if( $('#full-screen-video').is(':empty') ) { 
    mainStreamId = remoteId;
    remoteStream.play('full-screen-video');
  } else {
    addRemoteStreamMiniView(remoteStream);
  }
});

// remove the remote-container when a user leaves the channel
client.on("peer-leave", function(evt) {
  var streamId = evt.stream.getId(); // the the stream id
  if(remoteStreams[streamId] != undefined) {
    remoteStreams[streamId].stop(); // stop playing the feed
    delete remoteStreams[streamId]; // remove stream from list
    if (streamId == mainStreamId) {
      var streamIds = Object.keys(remoteStreams);
      var randomId = streamIds[Math.floor(Math.random()*streamIds.length)]; // select from the remaining streams
      remoteStreams[randomId].stop(); // stop the stream's existing playback
      var remoteContainerID = '#' + randomId + '_container';
      $(remoteContainerID).empty().remove(); // remove the stream's miniView container
      remoteStreams[randomId].play('full-screen-video'); // play the random stream as the main stream
      mainStreamId = randomId; // set the new main remote stream
    } else {
      var remoteContainerID = '#' + streamId + '_container';
      $(remoteContainerID).empty().remove(); // 
    }
  }
});

// show mute icon whenever a remote has muted their mic
client.on("mute-audio", function (evt) {
  toggleVisibility('#' + evt.uid + '_mute', true);
});

client.on("unmute-audio", function (evt) {
  toggleVisibility('#' + evt.uid + '_mute', false);
});

// show user icon whenever a remote has disabled their video
client.on("mute-video", function (evt) {
  var remoteId = evt.uid;
  // if the main user stops their video select a random user from the list
  if (remoteId != mainStreamId) {
    // if not the main vidiel then show the user icon
    toggleVisibility('#' + remoteId + '_no-video', true);
  }
});

client.on("unmute-video", function (evt) {
  toggleVisibility('#' + evt.uid + '_no-video', false);
});

// join a channel
function joinChannel() {
  var token = generateToken();
  var userID = null; // set to null to auto generate uid on successfull connection
  client.join(token, channelName, userID, function(uid) {
      console.log("User " + uid + " join channel successfully");
      createCameraStream(uid);
      localStreams.camera.id = uid; // keep track of the stream uid 
  }, function(err) {
      console.log("[ERROR] : join channel failed", err);
  });
}

// video streams for channel
function createCameraStream(uid) {
  var localStream = AgoraRTC.createStream({
    streamID: uid,
    audio: true,
    video: true,
    screen: false
  });
  localStream.setVideoProfile(cameraVideoProfile);
  localStream.init(function() {
    console.log("getUserMedia successfully");
    // TODO: add check for other streams. play local stream full size if alone in channel
    localStream.play('local-video'); // play the given stream within the local-video div

    // publish local stream
    client.publish(localStream, function (err) {
      console.log("[ERROR] : publish local stream error: " + err);
    });

    enableUiControls(localStream); // move after testing
    localStreams.camera.stream = localStream; // keep track of the camera stream for later
  }, function (err) {
    console.log("[ERROR] : getUserMedia failed", err);
  });
}

// SCREEN SHARING
function initScreenShare() {
  screenClient.init(agoraAppId, function () {
    console.log("AgoraRTC screenClient initialized");
    joinChannelAsScreenShare();
    screenShareActive = true;
    // TODO: add logic to swap button
  }, function (err) {
    console.log("[ERROR] : AgoraRTC screenClient init failed", err);
  });  
}

function joinChannelAsScreenShare() {
  var token = generateToken();
  var userID = null; // set to null to auto generate uid on successfull connection
  screenClient.join(token, channelName, userID, function(uid) { 
    localStreams.screen.id = uid;  // keep track of the uid of the screen stream.

    // Create the stream for screen sharing.
    var screenStream = AgoraRTC.createStream({
      streamID: uid,
      audio: false, // Set the audio attribute as false to avoid any echo during the call.
      video: false,
      screen: true, // screen stream
      extensionId: 'minllpmhdgpndnkomcoccfekfegnlikg', // Google Chrome:
      mediaSource:  'screen', // Firefox: 'screen', 'application', 'window' (select one)
    });
    screenStream.setScreenProfile(screenVideoProfile); // set the profile of the screen
    screenStream.init(function(){
      console.log("getScreen successful");
      localStreams.screen.stream = screenStream; // keep track of the screen stream
      $("#screen-share-btn").prop("disabled",false); // enable button
      screenClient.publish(screenStream, function (err) {
        console.log("[ERROR] : publish screen stream error: " + err);
      });
    }, function (err) {
      console.log("[ERROR] : getScreen failed", err);
      localStreams.screen.id = ""; // reset screen stream id
      localStreams.screen.stream = {}; // reset the screen stream
      screenShareActive = false; // resest screenShare
      toggleScreenShareBtn(); // toggle the button icon back (will appear disabled)
    });
  }, function(err) {
    console.log("[ERROR] : join channel as screen-share failed", err);
  });

  screenClient.on('stream-published', function (evt) {
    console.log("Publish screen stream successfully");
    localStreams.camera.stream.disableVideo(); // disable the local video stream (will send a mute signal)
    localStreams.camera.stream.stop(); // stop playing the local stream
    // TODO: add logic to swap main video feed back from container
    remoteStreams[mainStreamId].stop(); // stop the main video stream playback
    addRemoteStreamMiniView(remoteStreams[mainStreamId]); // send the main video stream to a container
    // localStreams.screen.stream.play('full-screen-video'); // play the screen share as full-screen-video (vortext effect?)
    $("#video-btn").prop("disabled",true); // disable the video button (as cameara video stream is disabled)
  });

  screenClient.on('stopScreenSharing', function (evt) {
    console.log("screen sharing stopped", err);
  });
}

function stopScreenShare() {
  localStreams.screen.stream.disableVideo(); // disable the local video stream (will send a mute signal)
  localStreams.screen.stream.stop(); // stop playing the local stream
  localStreams.camera.stream.enableVideo(); // enable the camera feed
  localStreams.camera.stream.play('local-video'); // play the camera within the full-screen-video div
  $("#video-btn").prop("disabled",false);
  screenClient.leave(function() {
    screenShareActive = false; 
    console.log("screen client leaves channel");
    $("#screen-share-btn").prop("disabled",false); // enable button
    screenClient.unpublish(localStreams.screen.stream); // unpublish the screen client
    localStreams.screen.stream.close(); // close the screen client stream
    localStreams.screen.id = ""; // reset the screen id
    localStreams.screen.stream = {}; // reset the stream obj
  }, function(err) {
    console.log("client leave failed ", err); //error handling
  }); 
}

// REMOTE STREAMS UI
function addRemoteStreamMiniView(remoteStream){
  var streamId = remoteStream.getId();
  // append the remote stream template to #remote-streams
  $('#remote-streams').append(
    $('<div/>', {'id': streamId + '_container',  'class': 'remote-stream-container col'}).append(
      $('<div/>', {'id': streamId + '_mute', 'class': 'mute-overlay'}).append(
          $('<i/>', {'class': 'fas fa-microphone-slash'})
      ),
      $('<div/>', {'id': streamId + '_no-video', 'class': 'no-video-overlay text-center'}).append(
        $('<i/>', {'class': 'fas fa-user'})
      ),
      $('<div/>', {'id': 'agora_remote_' + streamId, 'class': 'remote-video'})
    )
  );
  remoteStream.play('agora_remote_' + streamId); 

  var containerId = '#' + streamId + '_container';
  $(containerId).dblclick(function() {
    // play selected container as full screen - swap out current full screen stream
    remoteStreams[mainStreamId].stop(); // stop the main video stream playback
    addRemoteStreamMiniView(remoteStreams[mainStreamId]); // send the main video stream to a container
    $(containerId).empty().remove(); // remove the stream's miniView container
    remoteStreams[streamId].stop() // stop the container's video stream playback
    remoteStreams[streamId].play('full-screen-video'); // play the remote stream as the full screen video
    mainStreamId = streamId; // set the container stream id as the new main stream id
  });
}

function leaveChannel() {

  if(screenShareActive) {
    stopScreenShare();
  }

  client.leave(function() {
    console.log("client leaves channel");
    localStreams.camera.stream.stop() // stop the camera stream playback
    client.unpublish(localStreams.camera.stream); // unpublish the camera stream
    localStreams.camera.stream.close(); // clean up and close the camera stream
    $("#remote-streams").empty() // clean up the remote feeds
    //disable the UI elements
    $("#mic-btn").prop("disabled", true);
    $("#video-btn").prop("disabled", true);
    $("#screen-share-btn").prop("disabled", true);
    $("#exit-btn").prop("disabled", true);
    // hide the mute/no-video overlays
    toggleVisibility("#mute-overlay", false); 
    toggleVisibility("#no-local-video", false); 
  }, function(err) {
    console.log("client leave failed ", err); //error handling
  });
}

// use tokens for added security
function generateToken() {
  return null; // TODO: add a token generation
}

Please note: video sharing is not supported by Safari. Thankfully Chrome and FireFox don't have such restrictions.

Let’s drop our JS includes into our html page to make the final connections. The below snippet fits into the main html (above) by replacing the comment  with the snippet below.

<script src="AgoraRTCSDK-3.1.1.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<script src="https://stackpath.bootstrapcdn.com/bootstrap/4.2.1/js/bootstrap.min.js"></script>
<script>
    $("#mic-btn").prop("disabled", true);
    $("#video-btn").prop("disabled", true);
    $("#screen-share-btn").prop("disabled", true);
    $("#exit-btn").prop("disabled", true);
</script>
<script src="agora-interface.js"></script>

Testing Setup (webserver/https)

Since the camera permissions requires a secure (https) connection, before we can test our video chat app we must spin up a simple web server with a https connection. Browsers have Whitelisted the localhost url so you can use that to test.

To keep things simple I like to use Live Server but you can use any method.

If you want to test this out with friends you can run it locally in conjunction with ngrok, a service that creates a tunnel out from your local machine and provides an https url for use. In my experience this is one of the simplest ways to run a publicly accessible https secured webserver on your local machine.

Once the server is ready we can run our test.

NOTE: use two (or more) browser tabs to simulate a local host and a single/multiple remote host(s).

Fin.

And just like that we are done! In-case you weren’t coding along or want to see the finished product all together, I have posted the project code on GitHub.

If you would like to see the demo in action, check out the demo of the code in action on GitHub Pages.

Please note: that due to high demand, I have updated the build so you will need to register for a free Agora.io account to get AppId to test the demo. If you need help, I have a quick 3 step guide

Thanks for taking the time to read my tutorial and if you have any questions please let me know with a comment. If you see any room for improvement feel free to fork the repo and make a pull request!

How to: Build a Video Chat App on Android

Hermes Frangoudis — Wed, 03 Jun 2020 20:18:28 +0000

In this guide, we’ll build a basic video chat app in 10 easy steps, using the Agora.io Video SDK for Android.

Updated: May 24, 2021
This tutorial and project have been updated to use v3.4.1 of the Agora Video for Android SDK

Prerequisites

Android Studio
Basic knowledge of Java and the Android SDK
An Agora Developer Account (see: How To Get Started with Agora)[https://www.agora.io/en/blog/how-to-get-started-with-agora?utm_source=devto&utm_medium=blog&&utm_campaign=How_To_Build_a_Video_Chat_App_on_Android]

Step 1: Agora.io Account

Once you finish the sign-up process, you will be redirected to the Dashboard. Open the Projects tab on the left-hand nav to see your default project’s App ID.

Step 2: Create an Android App

Within Android Studio, create a new Single Activity app.

Step 3: Integrate the Agora SDK

There are two ways to add the Agora Video SDK into your project. You can use JCenter or you can manually add the SDK. For this project we'll add the project using JCenter.

Add the following line in your project level build.gradle:

allprojects {
    repositories {
        ...
        maven { url 'https://www.jitpack.io' }
        ...
    }
}

Add the following line in the /app/build.gradle file of your project:

dependencies {
    ...
    //Agora RTC SDK for video call
    implementation 'com.github.agorabuilder:native-full-sdk:3.4.1'
}

Set up Agora APP ID ###

Next, its time to add your Agora.io App ID (see Step-1) to the Android project’s Strings.xml (app/src/main/res/values/Strings.xml).

<resources>
    <string name="app_name">Agora-Android-Video-Tutorial</string>
    <string name="agora_app_id"><#YOUR APP ID#></string>
</resources>

The next step is to add the appropriate permissions within Manifest.xml

<uses-permission android:name="android.permission.READ_PHONE_STATE" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />

The final step is to prevent obfuscation of the Agora classes, while this might sound complex it’s really simple. In the proguard-rules.pro file, add:

-keep class io.agora.**{*;}

NOTE: Ensure that the Android NDK plugin is installed and setup for this project

Step 4: Setup views

Now that we have the Agora.io SDK integrated, let’s set up our UI. I will breeze through this portion as we will be using standard UI elements.

In the example, I chose to use ImageView instead of Button for the various UI elements. Either works, the important part is to note that there are functions that we link to using the onClick property.

Step 5: Checking Permissions

I know what you must be thinking… “didn’t we already set up the Permissions?” Earlier we let the applications Manifest know which permissions our app plans to use, but we still have to explicitly request the user grant these permissions. Don’t worry this is the final step in getting the boilerplate project running and it’s painless.

First, let's declare which permissions we want to request.

// Permissions
private static final int PERMISSION_REQ_ID = 22;
private static final String[] REQUESTED_PERMISSIONS = {Manifest.permission.RECORD_AUDIO, Manifest.permission.CAMERA};

Next, we set up a couple of functions to help us. First we'll add a method that will request permissions for a given permission string and code.

public boolean checkSelfPermission(String permission, int requestCode) {
    if (ContextCompat.checkSelfPermission(this,
            permission)
            != PackageManager.PERMISSION_GRANTED) {

        ActivityCompat.requestPermissions(this,
                REQUESTED_PERMISSIONS,
                requestCode);
        return false;
    }
    return true;
}

Next, we have a callback method that will get called after the user has responded to the permissions request prompt.

@Override
public void onRequestPermissionsResult(int requestCode,
                                       @NonNull String permissions[], @NonNull int[] grantResults) {
    Log.i(LOG_TAG, "onRequestPermissionsResult " + grantResults[0] + " " + requestCode);

    switch (requestCode) {
        case PERMISSION_REQ_ID: {
            if (grantResults[0] != PackageManager.PERMISSION_GRANTED || grantResults[1] != PackageManager.PERMISSION_GRANTED) {
                Log.i(LOG_TAG, "Need permissions " + Manifest.permission.RECORD_AUDIO + "/" + Manifest.permission.CAMERA);
                break;
            }
            // if permission granted, initialize the engine
            initAgoraEngine();
            break;
        }
    }
}

Last, within our Class’s onCreate we check if our permissions have been granted and if not the above methods will handle the requests.

Step 6: Initializing the Agora.io SDK

Now that we have our view, we are ready to initialize the Agora.io SDK, set up the user profile and set the video quality settings.

In the previous step, you may have noticed there are a couple places that I call initAgoraEngine(). Before we can dive into the initialization we need to make sure that our Activity has access to an instance of the Agora.io RtcEngine.

Within our MainActivity Class, we need to declare a Class property to store our instance of RtcEngine.

private RtcEngine mRtcEngine;

Now its time to initialize! After all the boilerplate setup we are finally at the step where we can start playing with the Agora.io engine!

Go ahead and declare your initAgoraEngine method within your class. Within this function, we will create a new instance of the RtcEngine using the baseContext, the Agora AppID (declared above), and an instance of the RtcEngineEventHandler (we’ll get into this a little later).

private void initAgoraEngine() {
    try {
        mRtcEngine = RtcEngine.create(getBaseContext(), getString(R.string.agora_app_id), mRtcEventHandler);
    } catch (Exception e) {
        Log.e(LOG_TAG, Log.getStackTraceString(e));

        throw new RuntimeException("NEED TO check rtc sdk init fatal error\n" + Log.getStackTraceString(e));
    }
    setupSession();
}

Once we have our new instance it’s time to set up our user’s session. Here we can set the Channel Profile to Communication, as this is a video chat and not a broadcast. This is also where we configure our video encoder settings.

private void setupSession() {
    mRtcEngine.setChannelProfile(Constants.CHANNEL_PROFILE_COMMUNICATION);

    mRtcEngine.enableVideo();

    mRtcEngine.setVideoEncoderConfiguration(new VideoEncoderConfiguration(VideoEncoderConfiguration.VD_1920x1080, VideoEncoderConfiguration.FRAME_RATE.FRAME_RATE_FPS_30,
            VideoEncoderConfiguration.STANDARD_BITRATE,
            VideoEncoderConfiguration.ORIENTATION_MODE.ORIENTATION_MODE_FIXED_PORTRAIT));
}

Step 7: Connecting the Video Streams

Before we can join a view call we need to be able to present the local video stream to the user via the UI elements we setup earlier (Step 4).

In the first line, we get a reference for the UI element will act as our parent view for our video stream. The second step is to use the RtcEngine to create a SurfaceView that will render the stream from the front camera, we also set the new VideoSurface to render on top of its parent view. The next step is to add the VideoSurface as a subview of the UI element. Lastly, we pass the VideoSurface to the engine as part of a VideoCanvas object. We leave the uid parameter blank so the SDK can handle creating a dynamic id for each user.

private void setupLocalVideoFeed() {
  FrameLayout videoContainer = findViewById(R.id.floating_video_container);
  SurfaceView videoSurface = RtcEngine.CreateRendererView(getBaseContext()); 
  videoSurface.setZOrderMediaOverlay(true);
  videoContainer.addView(videoSurface);
  mRtcEngine.setupLocalVideo(new VideoCanvas(videoSurface, VideoCanvas.RENDER_MODE_FIT, 0));
}

Now, that we have our local video feed setup we need to use a similar function to connect our remote video stream.

private void setupRemoteVideoStream(int uid) {
  FrameLayout videoContainer = findViewById(R.id.bg_video_container);
  SurfaceView videoSurface = RtcEngine.CreateRendererView(getBaseContext());
  videoContainer.addView(videoSurface);
  mRtcEngine.setupRemoteVideo(new VideoCanvas(videoSurface, VideoCanvas.RENDER_MODE_FIT, uid));
  mRtcEngine.setRemoteSubscribeFallbackOption(io.agora.rtc.Constants.STREAM_FALLBACK_OPTION_AUDIO_ONLY);
}

The main difference with the remote video from the local, is the user id parameter that gets passed to the engine as part of the VideoCanvas object that gets passed to the engine. The last line sets the fall back option in case the video degrades the engine will revert to audio only.

Step 8: Setup the SDK Event Handler

Earlier, I made a reference to the RtcEngineEventHandler, and now it’s time to declare it as a property of our MainActivity Class. The engine will call these methods from the RtcEngineEventHandler.

// Handle SDK Events
private final IRtcEngineEventHandler mRtcEventHandler = new IRtcEngineEventHandler() {
    @Override
    public void onUserJoined(final int uid, int elapsed) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                // set first remote user to the main bg video container
                setupRemoteVideoStream(uid);
            }
        });
    }

    // remote user has left channel
    @Override
    public void onUserOffline(int uid, int reason) { // Tutorial Step 7
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                onRemoteUserLeft();
            }
        });
    }

    // remote user has toggled their video
     @Override
     public void onRemoteVideoStateChanged(final int uid, final int state, int reason, int elapsed) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                onRemoteUserVideoToggle(uid, state);
            }
        });
    }
};

Each event triggers some fairly straight forward functions, including one we wrote in the previous step. In the interest of keeping this brief, I will provide the code below but I won’t give an in-depth breakdown.

private void onRemoteUserVideoToggle(int uid, int state) {
    FrameLayout videoContainer = findViewById(R.id.bg_video_container);

    SurfaceView videoSurface = (SurfaceView) videoContainer.getChildAt(0);
    videoSurface.setVisibility(state == 0 ? View.GONE : View.VISIBLE);

    // add an icon to let the other user know remote video has been disabled
    if(state == 0){
        ImageView noCamera = new ImageView(this);
        noCamera.setImageResource(R.drawable.video_disabled);
        videoContainer.addView(noCamera);
    } else {
        ImageView noCamera = (ImageView) videoContainer.getChildAt(1);
        if(noCamera != null) {
            videoContainer.removeView(noCamera);
        }
    }
}

private void onRemoteUserLeft() {
    removeVideo(R.id.bg_video_container);
}

private void removeVideo(int containerID) {
    FrameLayout videoContainer = findViewById(containerID);
    videoContainer.removeAllViews();
}

Step 9: Joining and Leaving Channels

I know what you’re thinking, STEP 9 ?!! Don’t sweat it the next two steps are really simple. Let’s start by joining a call…

Below you can see from the first line, Agora SDK makes it simple, the engine calls joinChannel, passing in the channel name followed by the call to set up our local video stream. (Step 7)

public void onjoinChannelClicked(View view) {
    mRtcEngine.joinChannel(null, "test-channel", "Extra Optional Data", 0);
    setupLocalVideoFeed();
    findViewById(R.id.joinBtn).setVisibility(View.GONE); // set the join button hidden
    findViewById(R.id.audioBtn).setVisibility(View.VISIBLE); // set the audio button hidden
    findViewById(R.id.leaveBtn).setVisibility(View.VISIBLE); // set the leave button hidden
    findViewById(R.id.videoBtn).setVisibility(View.VISIBLE); // set the video button hidden
}

Leaving the channel is even simpler, the engine calls leaveChannel. Above you’ll notice there are a few lines to remove the video stream subviews from each UI element.

public void onLeaveChannelClicked(View view) {
    leaveChannel();
    removeVideo(R.id.floating_video_container);
    removeVideo(R.id.bg_video_container);
    findViewById(R.id.joinBtn).setVisibility(View.VISIBLE); // set the join button visible
    findViewById(R.id.audioBtn).setVisibility(View.GONE); // set the audio button hidden
    findViewById(R.id.leaveBtn).setVisibility(View.GONE); // set the leave button hidden
    findViewById(R.id.videoBtn).setVisibility(View.GONE); // set the video button hidden
}

private void leaveChannel() {
    mRtcEngine.leaveChannel();
}

private void removeVideo(int containerID) {
    FrameLayout videoContainer = findViewById(containerID);
    videoContainer.removeAllViews();
}

Step 10: Adding UI Functionality

The last remaining parts are related to connecting the UI elements for toggling the microphone and video stream on the local device. Let’s start with the audio toggle.

First, we get the reference to our button, and then check if it has been toggled on/off using isSelected(). Once we have updated the UI element state, we pass the button’s updated state to the engine.

public void onAudioMuteClicked(View view) {
    ImageView btn = (ImageView) view;
    if (btn.isSelected()) {
        btn.setSelected(false);
        btn.setImageResource(R.drawable.audio_toggle_btn);
    } else {
        btn.setSelected(true);
        btn.setImageResource(R.drawable.audio_toggle_active_btn);
    }

    mRtcEngine.muteLocalAudioStream(btn.isSelected());
}

Moving on to the video toggle, as with the audio toggle we check/update the button’s state using isSelected() and then pass that to the engine. To give a better visual representation of the video being muted, we hide/show the VideoSurface.

public void onVideoMuteClicked(View view) {
    ImageView btn = (ImageView) view;
    if (btn.isSelected()) {
        btn.setSelected(false);
        btn.setImageResource(R.drawable.video_toggle_btn);
    } else {
        btn.setSelected(true);
        btn.setImageResource(R.drawable.video_toggle_active_btn);
    }

    mRtcEngine.muteLocalVideoStream(btn.isSelected());

    FrameLayout container = findViewById(R.id.floating_video_container);
    container.setVisibility(btn.isSelected() ? View.GONE : View.VISIBLE);
    SurfaceView videoSurface = (SurfaceView) container.getChildAt(0);
    videoSurface.setZOrderMediaOverlay(!btn.isSelected());
    videoSurface.setVisibility(btn.isSelected() ? View.GONE : View.VISIBLE);
}

I hope you enjoyed reading along and working together on creating a 1-to-1 Video Chat Android app using the Agora.io Video SDK.

How To: Build an Augmented Reality Remote Assistance App

Hermes Frangoudis — Mon, 06 Apr 2020 20:27:13 +0000

Have you ever been on the phone with customer support and struggled to describe the issue, or had the support person fail to clearly describe the solution or not understand what/where you should be looking?

Most remote assistance today is done through audio or text based chat. These solutions can be frustrating for users who may have a hard time describing their issues or understanding new concepts and terminology associated with troubleshooting whatever they need help with.

Thankfully technology has reached a point where this issue can be easily solved using Video Chat and Augmented Reality. In this guide, we’ll walk through all the steps you need to build an iOS app that leverages ARKit and video chat to create an interactive experience.

Prerequisites

A basic to intermediate understanding of Swift and iOS
Basic understanding of ARKit and Augmented Reality concepts
Agora.io Developer Account
Understand how to use Cocoa Pods
Hardware: a Mac with Xcode and 2 iOS devices
- iPhone: 6S or newer
- iPad: 5th Generation or newer

Please Note: While no Swift/iOS knowledge is needed to follow along, certain basic concepts in Swift/ARKit won’t be explained along the way.

Overview

The app we are going to build is meant to be used by two users who are in separate physical locations. One user will input a channel name and CREATE the channel. This will launch a back-facing AR-enabled camera. The second user will input the same channel name as the first user and JOIN the channel.

Once both users are in the channel, the user that created the channel will broadcast their rear camera into the channel. The second user has the ability to draw on their local screen, and have the touch input displayed in augmented reality in the first user’s world.

Let’s take a moment to review all the steps that we’ll be going through:

Download and build starter project
Project structure overview
Add video chat functionality
Capture and normalize touch data
Add data transmission
Display touch data in augmented reality
Add “Undo” functionality

Getting started with the starter project

I have created a starter project for this tutorial that includes the initial UI elements and buttons, including the bare-bones AR and remote user views.

Let’s start by downloading the starter project repo. Once all the files have finished downloading, open a Terminal window to the project’s directory and run pod install to install all dependencies. Once the dependencies have finished installing, open the AR Remote Support.xcworkspace in Xcode.

Once the project is open in Xcode, let’s build and run the project using the iOS simulator. The project should build and launch without issue.

build the starter project before starting development

Add a channel name, then click Join and Create buttons, to preview the UI’s that we will be working with.

Project Structure Overview

Before we start coding, let’s walk through the starter project files to understand how everything is setup. We’ll start with the dependencies, then go over the required files, and lastly we’ll take a look at the custom classes that we’ll be working with.

Within the Podfile, there are two 3rd-party dependencies: Agora.io’s Real-Time Communications SDK, facilitates in building video chat functionality; ARVideoKit’s open-source renderer, facilitates using the rendered AR view as a video source. The reason we need an off-screen renderer is because ARKit obfuscates the rendered view, so we need a framework to handle the task of exposing the rendered pixelbuffer.

As we move into project files, the AppDelegate.swift has the standard set up with one minor update. The ARVideoKit library is imported and there’s an added delegate function for UIInterfaceOrientationMask to return the ARVideoKit’s orientation. Within the info.plist the required permissions for Camera and Microphone access are included. These permissions are required by ARKit, the Agora.io Video SDK, and ARVideoKit.

Before we jump into the custom ViewController's, let’s take a look at some of the supporting files/classes that we’ll be using. The GetValueFromFile.swift allows us to store any sensitive API credentials in the keys.plist so we don’t have to hard-code them into the project. SCNVector3+Extensions.swift contains some extensions and functions for the SCNVector3 class that make mathematical calculations simpler. The last helper file is ARVideoSource.swift, which contains the implementation of the AgoraVideoSourceProtocol, which we’ll use to pass our rendered AR scene as the video source for one of the users in the video chat.

The ViewController.swift is a simple entry point for the app. It allows users to input a Channel Name and then choose whether they want to: CREATE the channel and receive remote assistance; JOIN the channel and provide remote assistance.

The ARSupportBroadcasterViewController.swift handles the functionality for the user who is receiving remote assistance. This ViewController will broadcast the rendered AR scene to the other user, so it implements the ARSCNViewDelegate, ARSessionDelegate, RenderARDelegate, and AgoraRtcEngineDelegate.

The ARSupportAudienceViewController.swift handles the functionality for the user who is providing remote assistance. This ViewController will broadcast the user’s front-facing camera and will allow the user to draw on their screen and have the touch information displayed in the remote user’s augmented reality scene, so it implements the UIGestureRecognizerDelegate, AgoraRtcEngineDelegate.

For simplicity, let’s refer to ARSupportBroadcasterViewController as BroadcasterVC and ARSupportAudienceViewController as AudienceVC.

Adding Video Chat Functionality

We’ll start by adding our AppID into the keys.plist file. Take a moment to log into your Agora Developer Account, select your project, copy your App ID and paste the hex into the value for AppID within keys.plist.

As an example, you keys.plist would look something like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>AppID</key>
    <string>69d5fd34f*******************5a1d</string>
</dict>
</plist>

Now that we have our AppID set, we will use it to initialize the Agora Engine within the loadView function for both BroadcasterVC and AudienceVC.

There are slight differences in how we set up the video configurations for each class. In the BroadcasterVC we are using an external video source so we can set up the video configuration and the source within the loadView.

override func loadView() {
    super.loadView()
    createUI()                                      // init and add the UI elements to the view
    self.view.backgroundColor = UIColor.black       // set the background color 

    // Agora setup
    guard let appID = getValue(withKey: "AppID", within: "keys") else { return } // get the AppID from keys.plist
    let agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: appID, delegate: self) // - init engine
    agoraKit.setChannelProfile(.communication) // - set channel profile
    let videoConfig = AgoraVideoEncoderConfiguration(size: AgoraVideoDimension1280x720, frameRate: .fps60, bitrate: AgoraVideoBitrateStandard, orientationMode: .fixedPortrait) 
    agoraKit.setVideoEncoderConfiguration(videoConfig) // - set video encoding configuration (dimensions, frame-rate, bitrate, orientation
    agoraKit.enableVideo() // - enable video
    agoraKit.setVideoSource(self.arVideoSource) // - set the video source to the custom AR source
    agoraKit.enableExternalAudioSource(withSampleRate: 44100, channelsPerFrame: 1) // - enable external audio souce (since video and audio are coming from seperate sources)
    self.agoraKit = agoraKit // set a reference to the Agora engine
}

Within the AudienceVC we will init the engine and set the Channel Profile in the loadView, but we will wait to configure video settings within the viewDidLoad.

override func loadView() {
    super.loadView()
    createUI() // init and add the UI elements to the view
    //  TODO: setup touch gestures

    // Add Agora setup
    guard let appID = getValue(withKey: "AppID", within: "keys") else { return }  // get the AppID from keys.plist
    self.agoraKit = AgoraRtcEngineKit.sharedEngine(withAppId: appID, delegate: self) // - init engine
    self.agoraKit.setChannelProfile(.communication) // - set channel profile
}

Note: We’ll add in the touch gestures functionality later on in this tutorial.

Let’s also set up the video configuration within the AudienceVC. Within the viewDidLoad call the setupLocalVideo function.

override func viewDidLoad() {
    super.viewDidLoad()
  ...
    // Agora implementation
    setupLocalVideo() //  - set video configuration
    //  - join the channel
  ...
}

Add the code below to the setupLocalVideo function.

func setupLocalVideo() {
    guard let localVideoView = self.localVideoView else { return } // get a reference to the localVideo UI element

    // enable the local video stream
    self.agoraKit.enableVideo()

    // Set video encoding configuration (dimensions, frame-rate, bitrate, orientation)
    let videoConfig = AgoraVideoEncoderConfiguration(size: AgoraVideoDimension360x360, frameRate: .fps15, bitrate: AgoraVideoBitrateStandard, orientationMode: .fixedPortrait)
    self.agoraKit.setVideoEncoderConfiguration(videoConfig)
    // Set up local video view
    let videoCanvas = AgoraRtcVideoCanvas()
    videoCanvas.uid = 0
    videoCanvas.view = localVideoView
    videoCanvas.renderMode = .hidden
    // Set the local video view.
    self.agoraKit.setupLocalVideo(videoCanvas)

    // stylin - round the corners for the view
    guard let videoView = localVideoView.subviews.first else { return }
    videoView.layer.cornerRadius = 25
}

Next we’ll join the channels from the viewDidLoad. Both ViewControllers use the same function to join the channel. In each BroadcasterVC and AudienceVC call the joinChannel function within the viewDidLoad.

override func viewDidLoad() {
    super.viewDidLoad()
  ...
    joinChannel() // Agora - join the channel
}

Add the code below to the joinChannel function.

func joinChannel() {
    // Set audio route to speaker
    self.agoraKit.setDefaultAudioRouteToSpeakerphone(true)
    // get the token - returns nil if no value is set
    let token = getValue(withKey: "token", within: "keys")
    // Join the channel
    self.agoraKit.joinChannel(byToken: token, channelId: self.channelName, info: nil, uid: 0) { (channel, uid, elapsed) in
      if self.debug {
          print("Successfully joined: \(channel), with \(uid): \(elapsed) secongs ago")
      }
    }
    UIApplication.shared.isIdleTimerDisabled = true     // Disable idle timmer
}

The joinChannel function will set the device to use the speakerphone for audio playback, and join the channel set by the ViewController.swift.

Note: This function will attempt to get the token value stored in keys.plist. This line is there in case you would like to use a temporary token from the Agora Console. For simplicity I have chosen to not use token security, so we have not set the value. In this case the function will return nil, and the Agora engine will not use token based security for this channel.

Now that users can join a channel, we should add functionality to leave the channel. Similar to joinChannel, both ViewControllers use the same function to leave the channel. In each BroadcasterVC and AudienceVC add the code below to the leaveChannel function.

func leaveChannel() {
    self.agoraKit.leaveChannel(nil)                     // leave channel and end chat
    self.sessionIsActive = false                        // session is no longer active
    UIApplication.shared.isIdleTimerDisabled = false    // Enable idle timer
}

The leaveChannel function will get called in popView and viewWillDisapear because we want to make sure we leave the channel whenever the user clicks to exit the view or if they dismiss the app (backgrounded/exit).

The last "Video Chat" feature we need to implement is the toggleMic function, which gets called anytime the user taps the microphone button. Both BroadcasterVC and AudienceVC use the same function, so add the code below to the toggleMic function.

@IBAction func toggleMic() {
    guard let activeMicImg = UIImage(named: "mic") else { return }
    guard let disabledMicImg = UIImage(named: "mute") else { return }
    if self.micBtn.imageView?.image == activeMicImg {
        self.agoraKit.muteLocalAudioStream(true) // Disable Mic using Agora Engine
        self.micBtn.setImage(disabledMicImg, for: .normal)
        if debug {
            print("disable active mic")
        }
    } else {
        self.agoraKit.muteLocalAudioStream(false) // Enable Mic using Agora Engine
        self.micBtn.setImage(activeMicImg, for: .normal)
        if debug {
            print("enable mic")
        }
    }
}

Handling touch gestures

In our app, the AudienceVC will provide remote assistance by using their finger to draw on their screen. Within the AudienceVC we’ll need to capture and handle the user’s touches.

First we’ll want to capture the location whenever the user initially touches the screen. Set that point as the starting point. As the user drag’s their finger across the screen, we’ll want to keep track of all those points, so we’ll use the touchPoints array to add each point, so we need to ensure an empty array with every new touch. I prefer to reset the array in the touchesBegan to mitigate against instances where the user adds a second finger to the screen.

override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
    // get the initial touch event
    if self.sessionIsActive, let touch = touches.first {
        let position = touch.location(in: self.view)
        self.touchStart = position
        self.touchPoints = []
        if debug {
            print(position)
        }
    }
    // check if the color selection menu is visible
    if let colorSelectionBtn = self.colorSelectionBtn, colorSelectionBtn.alpha < 1 {
        toggleColorSelection() // make sure to hide the color menu
    }
}

Note: This example will only support drawing with a single finger. It is possible to support multi-touch drawing, it would require some more effort to track the uniqueness of the touch event.

To handle the finger movement, let’s use a Pan Gesture. Within this gesture we’ll listen for the gesture start, change, and end states. Let’s start by registering the Pan Gesture.

func setupGestures() {
    // pan gesture
    let panGesture = UIPanGestureRecognizer(target: self, action: #selector(handlePan(_:)))
    panGesture.delegate = self
    self.view.addGestureRecognizer(panGesture)
}

Once the Pan Gesture is recognized, we’ll calculate the position of the touch within the view. The GestureRecognizer gives us the touch positions as values relative to the Gesture’s initial touch. This means that the translation from GestureRecognizer at GestureRecognizer.began is (0,0). The self.touchStart will help us to calculate the x,y values relative to the view’s coordinate system.

@IBAction func handlePan(_ gestureRecognizer: UIPanGestureRecognizer) {

    // TODO: send touch started event

    // keep track of points captured during pan gesture
    if self.sessionIsActive && (gestureRecognizer.state == .began || gestureRecognizer.state == .changed) {
        let translation = gestureRecognizer.translation(in: self.view)
        // calculate touch movement relative to the superview
        guard let touchStart = self.touchStart else { return } // ignore accidental finger drags
        let pixelTranslation = CGPoint(x: touchStart.x + translation.x, y: touchStart.y + translation.y)

        // normalize the touch point to use view center as the reference point
        let translationFromCenter = CGPoint(x: pixelTranslation.x - (0.5 * self.view.frame.width), y: pixelTranslation.y - (0.5 * self.view.frame.height))

        self.touchPoints.append(pixelTranslation)

        // TODO: Send captured points

        DispatchQueue.main.async {
            // draw user touches to the DrawView
            guard let drawView = self.drawingView else { return }
            guard let lineColor: UIColor = self.lineColor else { return }
            let layer = CAShapeLayer()
            layer.path = UIBezierPath(roundedRect: CGRect(x:  pixelTranslation.x, y: pixelTranslation.y, width: 25, height: 25), cornerRadius: 50).cgPath
            layer.fillColor = lineColor.cgColor
            drawView.layer.addSublayer(layer)
        }

        if debug {
            print(translationFromCenter)
            print(pixelTranslation)
        }
    }

    if gestureRecognizer.state == .ended {
        // TODO: send message to remote user that touches have ended

        // clear list of points
        if let touchPointsList = self.touchPoints {
            self.touchStart = nil // clear starting point
            if debug {
                print(touchPointsList)
            }
        }
    }
}

Once we’ve calculated the pixelTranslation (x,y values relative to the view’s coordinate system), we can use these values to draw the points to the screen and then to “normalize” the points relative to the screen’s center point.

I’ll discuss normalizing the touches in a moment, but first lets go through drawing the touches to the screen. Since we are drawing to the screen we’ll want to use the Main thread. So within a Dispatch block we’ll use the the pixelTranslation to draw the points into the DrawingView. For now don’t worry about removing the points because we’ll handle that when we transmit the points.

Before we can transmit the user’s touches we need to "normalize" the point relative to the screen’s center. UIKit considers the upper left hand corner of the view to be (0,0), but within AR we don't have any screen bounds so in ARKit we’ll need to add the points relative to the ARCamera’s center point. To achieve this we’ll calculate the translationFromCenter using the pixelTranslation and subtracting half of the view’s height and widths.

Transmitting touches and colors

To add an interactive layer, we’ll use the DataStream provided as part of the Agora engine. Agora’s video SDK allows for the ability to create a data stream capable of sending up to 30 (1kb) packets per second. Since we will be sending small data messages this will work well for us.

Let’s start by enabling the DataStream within the firstRemoteVideoDecoded. We’ll do this in both BroadcasterVC and AudienceVC.

func rtcEngine(_ engine: AgoraRtcEngineKit, firstRemoteVideoDecodedOfUid uid:UInt, size:CGSize, elapsed:Int) {
  // ... 
  if self.remoteUser == uid {
    // ...
    // create the data stream
    self.streamIsEnabled = self.agoraKit.createDataStream(&self.dataStreamId, reliable: true, ordered: true)
    if self.debug {
        print("Data Stream initiated - STATUS: \(self.streamIsEnabled)")
    }
  }
}

If the data stream is enabled successfully, self.streamIsEnabled will have a value of 0. We’ll check this value before attempting to send any messages.

Now that the DataStream is enabled, we’ll start with AudienceVC. Let’s review what data we need to send: touch-start, touch-end, the points, and color. Starting with the touch events, we’ll update the PanGesture to send the appropriate messages.

Note: Agora’s Video SDK DataStream uses raw data so we need to convert all messages to Strings and then use the .data attribute to pass the raw data bytes.

@IBAction func handlePan(_ gestureRecognizer: UIPanGestureRecognizer) {
        if self.sessionIsActive && gestureRecognizer.state == .began && self.streamIsEnabled == 0 {
            // send message to remote user that touches have started
            self.agoraKit.sendStreamMessage(self.dataStreamId, data: "touch-start".data(using: String.Encoding.ascii)!)
        }

        if self.sessionIsActive && (gestureRecognizer.state == .began || gestureRecognizer.state == .changed) {
            let translation = gestureRecognizer.translation(in: self.view)
            // calculate touch movement relative to the superview
            guard let touchStart = self.touchStart else { return } // ignore accidental finger drags
            let pixelTranslation = CGPoint(x: touchStart.x + translation.x, y: touchStart.y + translation.y)

            // normalize the touch point to use view center as the reference point
            let translationFromCenter = CGPoint(x: pixelTranslation.x - (0.5 * self.view.frame.width), y: pixelTranslation.y - (0.5 * self.view.frame.height))

            self.touchPoints.append(pixelTranslation)

            if self.streamIsEnabled == 0 {
                // send data to remote user
                let pointToSend = CGPoint(x: translationFromCenter.x, y: translationFromCenter.y)
                self.dataPointsArray.append(pointToSend)
                if self.dataPointsArray.count == 10 {
                    sendTouchPoints() // send touch data to remote user
                    clearSubLayers() // remove touches drawn to the screen
                }

                if debug {
                    print("streaming data: \(pointToSend)\n - STRING: \(self.dataPointsArray)\n - DATA: \(self.dataPointsArray.description.data(using: String.Encoding.ascii)!)")
                }
            }

            DispatchQueue.main.async {
                // draw user touches to the DrawView
                guard let drawView = self.drawingView else { return }
                guard let lineColor: UIColor = self.lineColor else { return }
                let layer = CAShapeLayer()
                layer.path = UIBezierPath(roundedRect: CGRect(x:  pixelTranslation.x, y: pixelTranslation.y, width: 25, height: 25), cornerRadius: 50).cgPath
                layer.fillColor = lineColor.cgColor
                drawView.layer.addSublayer(layer)
            }

            if debug {
                print(translationFromCenter)
                print(pixelTranslation)
            }
        }

        if gestureRecognizer.state == .ended {
            // send message to remote user that touches have ended
            if self.streamIsEnabled == 0 {
                // transmit any left over points
                if self.dataPointsArray.count > 0 {
                    sendTouchPoints() // send touch data to remote user
                    clearSubLayers() // remove touches drawn to the screen
                }
                self.agoraKit.sendStreamMessage(self.dataStreamId, data: "touch-end".data(using: String.Encoding.ascii)!)
            }
            // clear list of points
            if let touchPointsList = self.touchPoints {
                self.touchStart = nil // clear starting point
                if debug {
                    print(touchPointsList)
                }
            }
        }
    }

ARKit runs at 60 fps so sending the points individually would cause us to hit the 30 packet limit resulting in point data not getting sent. So we’ll add the points to the dataPointsArray and transmit them every 10 points. Each touch-point is about 30–50 bytes, so by transmitting every tenth point we will stay well within the limits of the DataStream.

func sendTouchPoints() {
    let pointsAsString: String = self.dataPointsArray.description
    self.agoraKit.sendStreamMessage(self.dataStreamId, data: pointsAsString.data(using: String.Encoding.ascii)!)
    self.dataPointsArray = []
}

When sending the touch data, we can also clear the DrawingView. To keep it simple we can get the DrawingView sublayers, loop through and remove them from the SuperLayer.

func clearSubLayers() {
    DispatchQueue.main.async {
        // loop through layers drawn from touches and remove them from the view
        guard let sublayers = self.drawingView.layer.sublayers else { return }
        for layer in sublayers {
            layer.isHidden = true
            layer.removeFromSuperlayer()
        }
    }
}

Lastly, we need to add support for changing the color of the lines. We’ll send the cgColor.components to get the color value as a comma delimited string. We’ll prefix the message with color: so that we don’t confuse it with touch data.

@IBAction func setColor(_ sender: UIButton) {
    guard let colorSelectionBtn = self.colorSelectionBtn else { return }
    colorSelectionBtn.tintColor = sender.backgroundColor
    self.lineColor = colorSelectionBtn.tintColor
    toggleColorSelection()
    // send data message with color components
    if self.streamIsEnabled == 0 {
        guard let colorComponents = sender.backgroundColor?.cgColor.components else { return }
        self.agoraKit.sendStreamMessage(self.dataStreamId, data: "color: \(colorComponents)".data(using: String.Encoding.ascii)!)
        if debug {
            print("color: \(colorComponents)")
        }
    }
}

Now that we’re able to send data from the AudienceVC, let’s add the ability for BroadcasterVC to receive and decode the data. We’ll use the rtcEngine delegate’s receiveStreamMessage function, to handle all data that is received from the DataStream.

func rtcEngine(_ engine: AgoraRtcEngineKit, receiveStreamMessageFromUid uid: UInt, streamId: Int, data: Data) {
    // successfully received message from user
    guard let dataAsString = String(bytes: data, encoding: String.Encoding.ascii) else { return }

    if debug {
        print("STREAMID: \(streamId)\n - DATA: \(data)\n - STRING: \(dataAsString)\n")
    }

    // check data message
    switch dataAsString {
        case var dataString where dataString.contains("color:"):
            if debug {
                print("color msg recieved\n - \(dataString)")
            }
            // remove the [ ] characters from the string
            if let closeBracketIndex = dataString.firstIndex(of: "]") {
                dataString.remove(at: closeBracketIndex)
                dataString = dataString.replacingOccurrences(of: "color: [", with: "")
            }
             // convert the string into an array -- using , as delimeter
            let colorComponentsStringArray = dataString.components(separatedBy: ", ")
            // safely convert the string values into numbers
            guard let redColor = NumberFormatter().number(from: colorComponentsStringArray[0]) else { return }
            guard let greenColor = NumberFormatter().number(from: colorComponentsStringArray[1]) else { return }
            guard let blueColor = NumberFormatter().number(from: colorComponentsStringArray[2]) else { return }
            guard let colorAlpha = NumberFormatter().number(from: colorComponentsStringArray[3]) else { return }
            // set line color to UIColor from remote user
            self.lineColor = UIColor.init(red: CGFloat(truncating: redColor), green: CGFloat(truncating: greenColor), blue: CGFloat(truncating:blueColor), alpha: CGFloat(truncating:colorAlpha))
        case "undo":
            // TODO: add undo
        case "touch-start":
            // touch-starts
            print("touch-start msg recieved")
            // TODO: handle event
        case "touch-end":
            if debug {
                print("touch-end msg recieved")
            }
        default:
            if debug {
                print("touch points msg recieved")
            }
            // TODO: add points in ARSCN

    }
}

There are a few different cases that we need to account for, so we’ll use a Switch to check the message and handle it appropriately.

When we receive the message to change the color, we need to isolate the component values, so we need to remove any excess characters from the string. Then we can use the components to initialize the UIColor.

In the next section we’ll go through handling the touch-start and adding the touch points into the ARSCN.

Display gestures in augmented reality

Upon receiving the message that a touch has started, we’ll want to add a new node to the scene and then parent all the touches to this node. We do this to group all the touch points and force them to always rotate to face the ARCamera.

case "touch-start":
    print("touch-start msg recieved")
    // add root node for points received
    guard let pointOfView = self.sceneView.pointOfView else { return }
    let transform = pointOfView.transform // transformation matrix
    let orientation = SCNVector3(-transform.m31, -transform.m32, -transform.m33) // camera rotation
    let location = SCNVector3(transform.m41, transform.m42, transform.m43) // location of camera frustum
    let currentPostionOfCamera = orientation + location // center of frustum in world space
    DispatchQueue.main.async {
        let touchRootNode : SCNNode = SCNNode() // create an empty node to serve as our root for the incoming points
        touchRootNode.position = currentPostionOfCamera // place the root node ad the center of the camera's frustum
        touchRootNode.scale = SCNVector3(1.25, 1.25, 1.25)// touches projected in Z will appear smaller than expected - increase scale of root node to compensate
        guard let sceneView = self.sceneView else { return }
        sceneView.scene.rootNode.addChildNode(touchRootNode) // add the root node to the scene
        let constraint = SCNLookAtConstraint(target: self.sceneView.pointOfView) // force root node to always face the camera
        constraint.isGimbalLockEnabled = true // enable gimbal locking to avoid issues with rotations from LookAtConstraint
        touchRootNode.constraints = [constraint] // apply LookAtConstraint

        self.touchRoots.append(touchRootNode)
    }

Note: We need to impose the LookAt constraint to ensure the drawn points always face the user. Points will need to be drawn always facing the camera.

When we receive touch-points we’ll need to decode the String into an Array of CGPoints that we can then append to the self.remotePoints array.

default:
    if debug {
        print("touch points msg recieved")
    }
    // convert data string into an array -- using given pattern as delimeter
    let arrayOfPoints = dataAsString.components(separatedBy: "), (")

    if debug {
        print("arrayOfPoints: \(arrayOfPoints)")
    }

    for pointString in arrayOfPoints {
        let pointArray: [String] = pointString.components(separatedBy: ", ")
        // make sure we have 2 points and convert them from String to number
        if pointArray.count == 2, let x = NumberFormatter().number(from: pointArray[0]), let y = NumberFormatter().number(from: pointArray[1]) {
            let remotePoint: CGPoint = CGPoint(x: CGFloat(truncating: x), y: CGFloat(truncating: y))
            self.remotePoints.append(remotePoint)
            if debug {
                print("POINT - \(pointString)")
                print("CGPOINT: \(remotePoint)")
            }
        }
    }

Within the session delegate’s didUpdate we’ll check the self.remotePoints array. We’ll pop the first point from the list and render a single point per frame to create the effect that the line is being drawn. We’ll parent the nodes to a single root node that gets created in the upon receipt of the "touch-start" message.

func session(_ session: ARSession, didUpdate frame: ARFrame) {
    // if we have points - draw one point per frame
    if self.remotePoints.count > 0 {
        let remotePoint: CGPoint = self.remotePoints.removeFirst() // pop the first node every frame
        DispatchQueue.main.async {
            guard let touchRootNode = self.touchRoots.last else { return }
            let sphereNode : SCNNode = SCNNode(geometry: SCNSphere(radius: 0.015))
            sphereNode.position = SCNVector3(-1*Float(remotePoint.x/1000), -1*Float(remotePoint.y/1000), 0)
            sphereNode.geometry?.firstMaterial?.diffuse.contents = self.lineColor
            touchRootNode.addChildNode(sphereNode)  // add point to the active root
        }
    }
}

Add "Undo"

Now that we have the data transmission layer setup, we can quickly keep track of each touch gesture and undo it. We’ll start by sending the undo message from the AudienceVC to the BroadcasterVC. We’ll add the code below to the sendUndoMsg function within our AudienceVC.

@IBAction func sendUndoMsg() {
    // if data stream is enabled, send undo message
    if self.streamIsEnabled == 0 {
        self.agoraKit.sendStreamMessage(self.dataStreamId, data: "undo".data(using: String.Encoding.ascii)!)
    }
}

Within the BroadcasterVC we’ll check for the undo message within the rtcEngine delegate’s receiveStreamMessage function. Since each set of touch points are parented to their own root nodes, with every undo message we’ll remove the last rootNode (in the array) from the scene.

case "undo":
    if !self.touchRoots.isEmpty {
        let latestTouchRoot: SCNNode = self.touchRoots.removeLast()
        latestTouchRoot.isHidden = true
        latestTouchRoot.removeFromParentNode()
    }

Build and Run

Now we are ready to build and run our app. Plug in your two test devices, build and run the app on each device. On one device enter the channel name and Create the channel, and then on the other device enter the channel name and Join the channel.

Thanks for following and coding along with me, below is a link to the completed project. Feel free to fork and make pull requests with any feature enhancements.

AR-Remote Support

For more information about the Agora.io Video SDK, please refer to the Agora.io API Reference.