DEV Community: Rick Cheng

Quickstart: Add Video Chat to Vision Pro with Agora

Rick Cheng — Wed, 04 Sep 2024 18:07:14 +0000

The future is here, and you're holding it – the Apple VisionPro device. It's a marvel of technology, but sometimes, you just need to see a friendly face across the digital divide. That's where Agora's ubiquitous Real-Time Communication (RTC) SDK comes in. With Agora, video chat on your VisionPro device transcends physical boundaries. You can connect with other VisionPro users, and Agora's expansive reach allows you to chat seamlessly with people on entirely different platforms. Imagine video calling your friends and family on their smartphones, tablets, or laptops – all through the power of Agora on your VisionPro.

This quick start guide will equip you to unlock the potential of Agora's native RTC SDK and transform your VisionPro into a powerful video chat hub. We will show you the way on how to set up and run the Quickstart project using both the simulator and the actual device. And we will walk through the code with an architecture overview and provide remarks on important APIs.

If you are a Unity developer, you may refer to this blog for the corresponding quick start guide.

Prerequisites:

Apple VisionPro
Apple computer with Silicon CPU Chip (M1 or newer generation)
Xcode (15.4) with VisionOS Support
Agora Developer account

Setups

Clone the GitHub project from this GitHub repo.
Open the project with Xcode: import the Agora SDK plugin. The project includes the Agora Video SDK as a dependency package. As soon as Xcode loads the project, the package download should start automatically and finish in a few minutes.

If the package did not load for some reason, e.g. networking issues, or you are setting up a brand new project of your own, see the following information to cover the package manager, CDN, and cocoapods.

To manually add the package to your Xcode project, put the specific branch of the git location as the dependency:

Zip file CDN download
Cocoa Pods

   pod 'AgoraRtcEngine_Special_iOS', '4.2.6.133.VISION'

Running the App

Open VisionVideoCallView.swift. and fill in your Agora app ID in the appID field.

This guide is made for Agora projects without security enabled. You should use a Test Mode app ID for the testing of this project. It is important to understand that authentication security is crucial and must be implemented for your actual application. Please refer to this guide for token security implementation.

Build and run for either the Simulator or your physical VisionPro device. At the same time, prepare a second user to join the chat as a remote user.

Remote User Setup

Since we are making an app that enables user video chat, we will have the VisionPro user as the local user and we will need another user as the remote user. Both users should appear in the application.

You have two options for the remote users application.

Use another physical device, including but not limited to another VisionPro device. If you not the first time Agora developer, and have already built another Agora app with the sample app ID on any platform, you are good to go.
Use the Agora Web demo app. This is the quickest way to test any new applications.

In this tutorial, we will use the web demo to run as our remote user. The following screenshot shows a web user published video feed using the same App ID and channel name.

Execution Sequence

You can start either the VisionPro user or the web user first.
Upon launch, the VisionPro user should see a dialog waiting for input. Enter the channel name (e.g. "visionpro") and hit the Join button.

The flow should be similar on other platforms. In our case, the web user will need to enter the channel name, press the button in Step 1 to 3 as shown in above screenshot.
The remote user should show up on the VisionPro demo's display panel automatically.
For the web demo, the remote user id should populate itself when the VisionPro user joins successfully. Press "Subscribe and Play" button to show the video stream from VisionPro.
When the chat session is finished on VisionPro, press the "<" button to leave the channel.

More environment specific description follows.

Running on Simulator

Since there is no camera simulation in the VisionPro simulator, your local video stream won’t be presented to the other users. However, you can still see other users who have enabled their cameras or custom video feeds on their devices. The following screenshot captures what you will see in the simulator.

Running on VisionPro Device

Before building your project to the VisionPro device, Add the following keys to your Info.plist file to request the necessary permissions:

<key>NSCameraUsageDescription</key>
<string>camera for self capture</string>
<key>NSMicrophoneUsageDescription</key>
<string>mic for my voice</string>

The permission request dialog should show up when you press the Join button. You should confirm your permission to allow the app to use your VisionPro’s camera and microphone. Once you joins the channel, you should see yourself in your avatar form and the remote user should show up on your screen!

On the other device/web browsers, the user can also see you in your digital form and you can start chatting together now!

Basic Code Walkthrough

If you are wondering how the components work together in this project, please read on. The following section aims to provide a clear and concise explanation of the code structure, logic, and flow.

Code Structure

Class	Purpose	Notes
Agora_VisionApp	Entry point of application	Standard entry point
ContentView	UI View presenting to user at the beginning	Accepts the meeting’s channel name. Calls VisionVideoCallView
VisionVideoCallView	Showing user’s video feeds as a grid format	Input the appID here
AgoraVideoCanvasView	The video canvas view for rendering the video	This is a reusable class that works on iOS/MacOS as well.
AgoraManager	The controller of the app business logic that implements Agora RTC Engine’s delegates.	Respond to join channel and leave channel; handles callback events like didJoinedOfUid and didJoinChannel.

Logic Flow

This quickstart demo follows a typical SwiftUI logic hierarchy. The Agora_VisionApp class defines the entry point, which brings a the ContentView. The ContentView is a container that uses a NavigationStack to allow Join channel user interface and the VisionVideoCallView interface to swap places, according to the current session state.

When the user hits the Join button, the VisionVideoCallView is constructed and pushed into the view stack. And video call session starts after the AgoraManager's initialization.

AgoraManager is an observed object that will implicitly causes updates to the view. The localUserId property and the allUsers property from AgoraManager are being observed.

When the VisionPro user joins the channel, the didJoinChannel event callback populates the localUserId. When a remote user joins the channel, the didJoinedOfUid callback adds that user's uid to the allUsers collection. On the other hand, when a remote user leaves the channel, the didOfflineOfUid callback remove the user from the allUsers collection. Whenever the allUsers value changes, VisionVideoCallView updates its canvas view grid automatically.

Class AgoraVideoCanvasView encapsulates the logic for populating a canvas with video stream using an uid as the input. The code is well documented and aims to be reusable for the iOS/MacOS and visionOS projects.

When the user hits the back button, NavigationStack pops the view stack and thus the VisionVideoCallView gets destroyed. At such point, AgoraManager invokes LeaveChannel(), which will cleanup the canvas views and dispose the Agora engine.

View Update Sequence

Essential API Calls

Create an Agora RTC engine

        let config: AgoraRtcEngineConfig = AgoraRtcEngineConfig()
        config.appId = appId
        config.audioScenario = .gameStreaming
        config.channelProfile = .liveBroadcasting      
        let eng = AgoraRtcEngineKit.sharedEngine(with:config , delegate: self)

Join a channel

        self.agoraEngine.joinChannel(byToken: token, channelId: channel, info: info, uid: uid)

Leave a channel

        self.agoraEngine.leaveChannel(leaveChannelBlock)

Dispose of the engine

        AgoraRtcEngineKit.destroy()

Local Camera Stream

If you are an experienced Agora developer, you may see all the APIs are the same as you've done in implementing an iOS app. You may wonder - how do I create a stream to capture my Avatar from the VisionPro? The answer is you don't need to. The beauty of the visionOS is that - your avatar rendering frames are used for your local camera's view. Therefore, without any change to the API calls, you get the same behavior to stream your local camera view!

Conclusion

As you've seen, integrating Agora's RTC SDK into your VisionPro app for video chat is surprisingly straightforward. The provided quick start project on GitHub serves as a fantastic springboard, offering a basic structure you can build upon. With just a few steps and a little code modification, you've unlocked the power of real-time communication for your VisionPro device.

The true beauty lies in Agora's cross-platform capabilities. Forget limitations! You can now video chat with friends and family on any device, regardless of their operating system. This opens doors for unparalleled collaboration and connection, shattering the boundaries between platforms.

So, dive into the world of Agora's RTC SDK and unleash the full potential of video chat on your VisionPro. With its ease of use and expansive reach, Agora empowers you to connect with the world on a whole new level.

For further reading, it is worthwhile to check out the following resources:

Cloud Recording for Unity Video Chat

Rick Cheng — Fri, 30 Apr 2021 19:19:25 +0000

Cloud Recording for Unity Video Chat

In this advanced real-time engagement topic, we show you how the Agora Cloud Recording API can be called to handle the task for Unity applications. You should be familiar with the basic setup of a simple video chat app in Unity. If not, you can follow the tutorial in this blog, which will get you ready in 30 minutes. We further expand the logic to control the clouding recording with a custom server.

Prerequisites

Clone the quick start repo as the base project to work on.
Fulfill the dependencies on the quick start project, such as getting an Agora App Id*.
Set up a working Amazon S3 Bucket. Be sure you verify that you can upload files to it by scripting.
Have basic server-side coding knowledge and environment.

** Note: This guide does not implement token authentication which is recommended for all RTE apps running in production environments. For more information about token based authentication within the Agora platform please refer to this guide: https://bit.ly/3sNiFRs*

Architecture

We will create a simple client-server system to simulate common usage of the feature. Here are the parts as depicted in figure1:

Unity App client 1: the commanding client, which controls the start and stop of the video recording
Unity App client 2: the normal client, which chats with client 1
App Server: a trusted server that stores secret keys, takes command from client 1 and makes RESTFul API calls to the SD-RTN back end
SD-RTN back end: the Agora network process video that makes magic happen (Software Defined Real Time Network)
Amazon S3 Bucket: the cloud storage with APIs for the video upload

Figure 1: Project Architecture

Note: Theoretically, we can omit the app server from this architecture and put the secret keys and RESTFul API calls directly in the client. But that wouldn’t be the best practice.

Project Overview

This project has three major steps:

Environment: This involves enabling the feature on your Agora account and setting up client-server and an S3 account.
Server: The code is written in PHP.
Client : A Unity application built on the basic video chat project.

Environment Setup

First, familiarize yourself with the background description from the documentation page. Follow the steps in the QuickStart section on how to enable the service from your Agora developer account console. You’ll need a customer Id *and a *customer secret from the console. Adding the setup from the basic video chat project, you should have the following tokens so far:

App Id: **Created with a new project. **I chose not to use a certificate for this tutorial.
App token: Only if you enabled certificate for your AppId.
**Customer Id: **Press “Add a secret” to get one (figure 2).
**Customer secret: **After getting the customer Id, the console generates a secret token for download.

Figure 2: Agora RESTful Console

Amazon S3 Credentials

You will need the following information for the clouding recording configuration:

Bucket name: In my example from figure 3, the name is “agoracdn”.
Region: **Since **my bucket is in “us-west-1”, it is mapped to “2” according to this table.
*User access key: **Get this from AWS IAM under *Users (figure 4).
*User Secret key: **Get this from AWS IAM under *Users.

Figure 3: S3 Management Console

Figure 4 : IAM Management Console

Server Environment

You should use the server framework that you are most familiar with. I run a quick PHP server on the localhost from my MacBook. I use this command to start the server from my PHP source code directory:

$ php -S localhost:8000

For more information, such as how to do the same on Windows machines, check the PHP manual page.

Server Implementation

The main goal of our server is to separate the necessary credential keys away from the client and relay the RESTful API calls to the Agora SD-RTN back end. The Cloud Recording Quick Start documentation shows the following essential steps for its life cycle:

Acquire
Start
Query
Stop

These four steps are mapped to four PHP scripts running from our test server. They will share the configuration that we gather from the Environment Setup step above.

Configuration

Here is the code of config.php. You need to fill into your credential tokens in the empty space and set the region number for your Amazon S3 bucket. The script combines the Customer Id and the Customer Secret token into an Authorization Secret (AuthSecret), which will be used throughout the API calls. You should create an integer string for the RecUID (Recorder User Id). This Id is different from the clients’ user Ids.

Acquire

The Aquire step initializes a Cloud Recording request to the Agora SD-RTN. In return, the RTN provides a resourceId. The channel name is passed from the client in a POST body. As a result, the resourceId is sent to the client. The client should maintain that id in its memory for the next steps.

Start

After you obtain a resourceId, you can tell the SD-RTN when the recording starts. The SD-RTN sends chunks of video data in .ts format to your S3 bucket. The resourceId and the channel name are passed from the client in a POST body. If the call is successful, you should get back the same resourceId and a sid (Session Id).

Query

It is important to know if a Recording is in session or the Start failed due to an AWS authentication issue. The Query command checks the current status with the stored resourceId and sid. The server responds with the current video name for the file list (m3u8) and other information.

Stop

When you decide the recording is done, call Stop to complete the cloud recording session.

Note: If you set up your Agora project with the certificate enabled, you need to add a token field into the CURLOPT_POSTFIELDS in the Acquire, Start, Stop, and Query API calls. Since my project was set up without one, I removed the field from the list.

Checkpoint

So far, the server code and configuration are set up. You can test the recording manually before the client code is created. To do so, simply assign the necessary values by hand instead of getting the values from the POST body.

If you are interested in learning more about the RESTful calls, you can use the Agora-RESTful-Service project to test cloud recording on the Postman app. In fact, the scripts and JSON body that I used in the PHP code above are inspired by the Postman app. You can quickly create the corresponding code for your chosen framework and language. See figures 5 and 6.

Figure 5 : Get the Code

Figure 6 : Use any language

Unity Client

As discussed in the Architecture section, there are two clients. Client 1 is the commanding client that controls the start and stop of the recording. Client 2 is a normal chat client. The original VideoChat project applies to client 2. We update the project with new features to support the commanding client.

The commanding feature is a good example application for the MVC design pattern. We will write the Unity C# code for the Model, the View and the Controller.

Model

From observation of the returning JSON output from the SD-RTN, we find the following sample structure covers the response body for Acquire, Start, Stop, and Query:

{“resourceId”:”xxxx”, “sid”:”yyy”, “serverResponse”:{“status”:5,”fileList”:”zzzz.m3u8",”fileListMode”:”string”,”sliceStartTime”:1606357122528} }

Turning that into C# classes, we have the following implementation:

View

We will update the Video Chat demo scene to the following:

Add a Start/Stop button to the lower-left side of the view area. Name it “RecordButton”. Set the text to display “Start”. Later, we modify the text to display “Stop” in the controller logic.
Add a Query button below the Start/Stop button. Make the RecordButton its parent in the hierarchy.

Figure 7: The Cloud Record Commanding Client

Controller

A CloudRecordController class responds to the button events and encapsulates the networking logic to drive the cloud recording sequence. Here is a top-down outline of the class structure:

Figure 8: Class Outline

Fields

ServerURL
recordButton
queryButton

Properties

ChannelName: Set by main controller (VideoChat.cs),
ResourceId: Get from server message,
SID: Get from server message,
IsRecording: Indicates if recording is in progress.

UI Controls

HandleStartStop(): Responds to the RecordButton click and toggles between the two states. The RecordButton is visible only after the user joins a channel and it appears as a Start button. When the user taps the Start button, the client sends an Acquire command to the server first. If the Acquire is successful, then Start begins automatically in the response handler to Acquire. The Start button becomes a Stop button. The Stop button stops the recording upon tapping by the user.

HandleQuery(): Responds to the QueryButton click. The output prints to the console.

RestoreRecordState(): Helps to maintain states between two Unity client sessions. If a recording is in progress but the user quits, the Start state is saved in the device’s persistent memory (PlayerPrefs). This function gets called at the Awake step of the hosting object.

SetRecordUI(): Changes the UI appearance for the Start state and the Stop state.

Server Interaction

The client interacts with the server with the four API calls defined earlier. It is easy to see that there are four API callers and four matching server response handlers for Acquire, Start, Stop, and Query.

The four functions use the same logic flow for the caller and the handler. We use the UnityWebRequest *class to package and send the POST request and parse the result using the model we defined for *CloudRecordResponseModel. At the end, invoke the callback to the handler to finish the processing. Figure 9 shows an example from the _Start function.

Figure 9 : Start function

_Start() runs separately from the Main thread in Unity in a coroutine so that the network calls won’t block the execution of the UI display.

The complete code listing for CloudRecordController class:

Integration

Now we have the MVC components all defined. Let’s integrate everything. We need a game object to host the CloudRecordController. We simply add that as an extra component to the RecordButton.

First, go back to the Unity Editor and drag the CloudRecordController.cs script to RecordButton. Second, assign the RecordButton and QueryButton to their fields. Third, enter the following URL in the Server URL field:

*http://localhost:8000*

You Inspector should look like this:

Figure 10: Record Button as a Controller

We update the main controller logic to link to the CloudRecordController.

Update the AgoraTest.cs script with a new serialized field for CloudRecordingObject (see figure 11).
In the Start() method, hide the CloudRecordingObject (see figure 11).

Figure 11: New Code in AgoraTest Class

In the OnJoinChannelSuccessHandler() method, update the code to pass the channel information to CloudRecordController (see figure 12)

Figure 12: Passing Channel Name

In the Unity Editor, drag and drop the RecordButton into the CloudRecordingObject field of AgoraTest inside GameController (see figure 13).

Figure 13: Linking Recording Object

Test

First, start the server by going to the Terminal and entering the php command, as shown in figure 14.

Figure 14: Start Server

Run the project from the Unity Editor, and execute in the following order:

Join
Start
Query
Stop
Leave

When you are done, head over to your AWS S3 account to check the files. You should find a list of .ts files uploaded. Here is the quick test I just did:

Figure 15: Sample Run

Conclusion

This project is a bit complex, with various depending configurations to set up. Thank you for following my post up to the end. The completed project can be found in our community Github repo.

Other Resources

For more information about Agora.io applications, take a look at the Agora Video Call Quickstart Guide and Agora API Reference.

I also invite you to join the Agoira.io Developer Slack community.

How to Add Chat with Face Filters to Your Multiplayer Unity Game

Rick Cheng — Wed, 19 Aug 2020 17:16:01 +0000

How to Add Chat with Face Filters to Your Multiplayer Unity Game

Do you have any thoughts about real time engagement in your game? Undoubtedly, having video chat during a co-op or multiplayer online game adds jokes and laughs to the experience. How about extend the fun with a face filter? It will definitely increase a player’s session time during this COVID-19 lock down.

In this tutorial, we will make use of an existing project, written by Joel Thomas and apply a face filter feature to the player. To follow what has been done, please check out the tutorial here. Make sure you can download the full project from GitHub to start. For comparison to the original project, consider the following preview:

Prerequisites

Create an Agora.io developer account here to get your AppID.
Create a Photon developer account for their AppID.
Clone my GitHub project as the starter project, which contains Viking demo’s assets, Agora Video SDK and Photon SDK.
Banuba Face SDK for Unity.
Unity 2019.3 or up.

You need to have a Banuba client token to run the project. Contact info@banuba.com for a trial token and SDK package if you don’t want to pay up front at the Asset Store.

Getting Started

First, we will use the Banuba Face SDK to start the AR face masking. Import the package that you downloaded from the Asset store or the unzipped SDK to the Unity project directory. You should see the current project directory structure and find the BanubaClientToken file in the Resource folder. Open the file in text editor and paste your SDK token key there.

Next, enter your Photon project info in Window > Photon Unity Networking > PUN Wizard > Setup Project. Paste your AppID or email you used for your Photon account.

Note that the GitHub project already contains a copy of the Agora Video SDK. You may choose update to the latest from the Asset Store. It is recommended that you delete the folder Assets/AgoraEngine/Plugins for a clean replacement.

Create BNB_AR Prefab

Open the Sample scene from BanabuFaceAR/Scenes, and do the following:

Create an empty game object, name it “BNBARStuff”.
Drag and drop items Camera, Faces and Surface *into **BNBARStuff.*
Change layers of BNBARStuff to “default” recursively.
Update the Camera component to culling mask on “default” only.
Drag the BNBARStuff object into th Prefabs folder to make it a prefab.
Open the prefab, rename Camera to “ARCamera”, and rename Surface to “ARSurface”. Change ARCamera’s culling mask to Default only. Uncheck Default from MainCamera’s culling mask list.

If your Banuba SDK has a FPS game object under Surface, you can remove it from the prefab. Also update the code in CameraController to do a null check:

**if (fpsTextObj != null)** fpsTextObj.text = “FPS: “ + fps;

Do NOT save the Sample scene. Open VikingsScene from DemoVikings and set up the BNBARStuff object using the prefab:

Drag and drop the BNBARStuff prefab to the Hierarchy.
Enable the one of the filters under Faces (e.g. Afro).
Highlight the Camera under BNBARStuff and play the app in Editor.

√ You can observe yourself with the AR filter applied in the Camera preview.

Save the VikingsScene scene if everything looks good so far.

Create Render Texture

Create a render texture to host the ARCamera input:

Create a RenderTexture; name it “ARRenderTexture”.
Assign ARRenderTexture into ARCamera’s Target Texture field.

Create Local View Prefab

In the* Viking* scene, create a UI hierarchy as following:

Create a new Image, change its width to 140 and height to 140; name it “ViewContainerWithMask”. Add a Mask component to this object.
Add a new game object under ViewContainerWithMask, name it “AvatarRenderTarget”. Make sure its scale is set to 1.
Add a RawImage component to AvatarRenderTarget. Drag and drop the ARRenderTexture into the Texture field of the RawImage component.
Add a button that has no text and no image as a child object of AvatarRenderTarget, set Normal Color and Pressed Color’s alpha to 0 (transparent). Set a low alpha (e.g. 35) for the other states. This button will invoke the appearance of the FilterDropdown.
Attach this “AvatarViewController.cs” script to AvatarRenderTarget object.

Assign ViewContainerWithMask object into the “Mask RectTan” field of the AvatarViewController component on the AvatarRenderTarget object.

Verify you set up as describe in Step 1–4. Drag the game object ViewContainerWithMask to the Prefab folder. Delete it from the Vikings Scene. Save.

Set up dropdown list for the masks

In the VikingsScene, create a UI Dropdown and place it to the upper right corner. Make sure you set the Canvas scaler to “Scale with Screen”. Assign the dropdown’s layer to “UI” and name it “FilterDropdown”. Hide this from the scene for now, since it will be activated later only when user taps the self-video display.

Code Modification

Spawn with Banuba FaceFilter enabled Prefab

In AgoraVideoChat.cs, update the CreateUserVideoSurface method with the following code snippet:

Update Prefab Field Assignments

Open Assets/DemoVikings/Resources/Charprefab, updated the fields:

Assign the prefab ViewContainerWithMask into the “AvatarVideoPrefab” field (see line 2 in the above code snippet).
Update the AppID and Channel ID.

Send the Frames as Custom Source Video

Add the following code snippet to the end of the AgoraVideoChat.cs script. The code runs at every frame to capture what is displaying on the ARCamera and sends it as a video frame into the channel.

Last Check

Since we created a bigger rectangle for Avatar image view (140 pt), we should also update Asset/Prefabs/UserVideo’s RectTransform to 140x140. Also, in AvatarViewController, we provided an option for zoom level. You may want to try 1.5x zooming for better view of yourself in that window. Here is the look after everything connects up:

That’s it! With a face mask on, you look much cooler than before! Have fun playing with friends and talking to them at the same time with your cool new look!

In Summary

We connected to Agora’s network to display our video chat channel.
We enabled other users to join our party, see their faces, and talk with them in real time.
We took it one step further and let user to pick a face mask provided by the Banuba ARFace Filter SDK.

If you have any questions or hit a snag in the course of building your own networked group video chat, please feel free to reach out directly or via the Agora Slack Channel!

Check out the link to the full project (it is the CompletedProject branch of the same repo):UnityVikingGameChatDemo

Adding Video Communication to A Multiplayer Unity Game

Rick Cheng — Thu, 04 Jun 2020 02:15:22 +0000

Do you ever imagine when you play against your friends on a multiplayer game on your lovely mobile device, you want to see each other’s facial expression or tease each other with jokes and funny faces? You’ve found a solution here without leaving the game itself to another chat App. In this tutorial we are going to take Unity’s popular Tanks game to the next level and make it into a game with live video chats!

Before we get started, there are a few prerequisites for anyone reading this article.

Prerequisites

Unity (2018) and a Unity Developer account
Knowledge of how to build your Unity project for iOS and Android
A cross-platform mobile multiplayer Unity game (I chose to use Tanks!!!)
An understanding of C# and scripting within Unity
An Agora.io developer account
At least two mobile devices (one iOS & one Android is ideal)

Project Setup

If you plan to use your own existing Unity project, go ahead and open it now and skip down to “Integrating Group Video Chat”.

For those readers that don’t have an existing project, keep reading; the next few sections are for you. (Note, this step is exactly the same to the Project Setup you may find in or have done most of the set up by following Hermes’ “Adding Voice Chat to a Multiplayer Cross-Platform Unity game” tutorial.)

New Unity Project

Please bear with me as the basic setup has a few steps and I’ll do my best to cover it swiftly with lots of images. Let’s start by opening Unity, creating a blank project. I recommend starting this project with the latest Unity 2018 LTS version. Note that the networking module UNet has been deprecated in 2019. So 2018 LTS is essentially the best option here.

Create a new project from Unity Hub

Download and import the “Tanks!!! Reference Project” from the Unity Store:

Searched by “Tanks Reference” and download this asset

When Unity prompts for if you want to overwrite the existing project with the new asset, click Yes. Furthermore, accept the API update prompt that will come up next.

Theres a couple more steps to getting the Tanks!!! reference project ready for building on mobile. First we need to enable Unity Live Mode for the project through the Unity dashboard. (select project → Multiplayer → Unet Config).

Set max players to 6 even though Tanks!!! limits the game to 4 players and click save

Once Unity Live Mode is enabled

Building for iOS

Now that we have Unity’s multiplayer enable, we are ready to build the iOS version. Let’s start by opening our Build Settings and switch our platform to iOS and build the project for testing.

Update the Bundle id and Usage Descriptions

Please note: you need to have Xcode installed and setup before attempting to build the project for iOS.

When building for the first time, create a new folder “Builds” and save the build as iOS

After the project has successfully built for iOS, we will see the project in the **Builds* folder*

Let’s open Unity-iPhone.xcodeproj, sign, and build / run on our test device.

Enable automatic signing to simplify the signing process. Remove In-App-Purchase if it shows up.

Don’t start celebrating just yet. Now that we have a working iOS build we still need to get the Android build running.

Building for Android

Android is a bit simpler than iOS since Unity can build, sign, and deploy to Android without the need to open Android Studio. For this section I’m going to assume everyone reading this has already linked Unity with their Android SDK folder. Let’s start by opening our Build Settings and switching our platform to Android.

Before we try to “Build and Run” the project on Android we need to make a couple adjustments to the code. Don’t worry this part is really simple, we only need to comment out a few lines of code, add a simple return statement, and replace one file.

**Some background:* the Tanks!!! Android build contains the Everyplay plugin for screen recording and sharing your game session. Unfortunately Everyplay shutdown in October 2018 and the plugin contains some issues that if not addressed will cause the project to fail to compile and to quit unexpectedly once it compiles.*

The first change we need to make is to correct a mistake in the syntax within the Everplay plugin’s build.gradle file. Start by navigating to our project’s Plugins folder and click into the Android folder and then go into the everyplay folder and open the build.gradle file in your favorite code editor.

Now that we have the Gradle file open, select all and replace it with the code below. The team that built Tanks!!! updated the code on GitHub but for some reason it didn’t make its way into the Unity Store plugin.

The last change we need to make is to disable EveryPlay. Why would we want to disable EveryPlay, you may ask. That’s because when the plugin tries to initialize itself it causes the Android app to crash. The fastest way I found was to update a couple lines within the EveryPlaySettings.cs, (Assets → Plugins → EveryPlay → Scripts) so that whenever EveryPlay attempts to check if it’s supported or enabled, we return false.

Now we are finally ready to build the project for Android! Within Unity open the Build Settings (File > Build Settings), select Android from the Platform list and click Switch Platform. Once Unity finishes its setup process, open the Player Settings. We need to make sure our Android app also has a unique Package Name, I chose com.agora.tanks.videodemo.

You may also need to create a key store for the Android app. See this section of the PlayerSettings in Unity Editor:

Android KeyStore setting

Integrating Video Chat

For this project Agora.io Video SDK for Unity was chosen, because it makes implementation into our cross-platform mobile project, really simple.

Let’s open up the Unity Store and search for “Agora Video SDK”.

You only download the asset once, and then you can import it to different projects.

Once the plugin page has loaded, go ahead and click **Download. *Once the download is complete, click and *Import **the assets into your project.

Uncheck the last four items before import

You should then open the Lobby as your main scene. The following shows the also how the service page would look like for the multiplayer settings:

Discussion: in the following sections we will go through how the project to be updated with new code and prefabs changes. For those just want to quickly try out everything. Here is a plugin file to import all the changes. You will just need to enter the AppId to the GameSettings object as described after importing.

Modify the Tank Prefab

Let add a plane on to the top of the tank to render the video display. Find the CompleteTank prefab from the project. Add a 3D object Plane to the prefab. Make sure the follow values are updated for the best result:

Y =8 for position; Scale to 0.7. Rotate -45 degrees on X, 45 degrees on Y.
Do not cast shadow
Disable the Mesh Collider script

Plane Prefab values

Attach VideoSurface.cs script from the Agora SDK to the Plane game object.

Save the change, and test the prefab in the game by going to Training to see the outcome. You should see a tank similar to the following screen:

Tank with Plane attached

Create UI for Mic/Camera Controls

Next, open the GameManager prefab and create a container game object and add three toggles under it:

Mic On
Cam On
FrontCam

GameManager UI Canvas

That’s basically all the UI changes we need for this project. The controller script will be added to the prefab later in the following sections.

Controller Scripts

Next we will go over some scripts to make the video chat to work for this game. Before adding new scripts, we will modify a script to allow the input for the Agora AppId.

GameSettings

Two updates to the scripts to make the game to work with Agora SDK.

(1) Add a SerializedField here for the AppId.

Go to your Agora developer’s account and get the AppId (you may need to following the instruction to create the project first):

Agora AppId

In the Lobby scene of the Unity Editor, paste the value of the App ID there and save:

Set the App ID for Agora API

(2) Add support for Android devices by asking for Microphone and Camera permissions.

Agora Controller Scripts

Before jump right into the code, let’s understand what capabilities are needed. They are:

Interface to the Agora Video SDK to do join channel, show video, mute microphone, flip camera, etc.
Actual implementation for the Agora SDK event callbacks.
Mapping from the Unity Multiplayer’s id to Agora user’s id.
A manager to respond to the UI Toggle actions that we created earlier.

The following script capture shows the corresponding scripts hierarchy. A discussion about the four classes follows.

Controller Script Hierarchy

AgoraApiHandlerImpl: this class implements most of the Agora video SDK event callbacks. Many of them are placeholders. To support the minimum capability in this game, the following handlers are of the most interest:

JoinChannelSuccessHandler — when local user joins a channel, this will corrospond to create a game and start a server. The server name is the same as the channel number for Agora SDK.
UserJoinedHandler — when a remote user joins the game.
UserOfflineHandler — when a remote user leaves the game.

SDKWarningHandler is commented out to reduce the noise in the debugging log. But it is recommended to enable it for an actual project.

AgoraVideoController: this singleton class is the main entry point for the Tanks project to interact with the Agora SDK. It will create the AgoraApiHandlerImpl** **instance and handles interface call to join channel, mute functions, etc. The code also checks for Camera and Microphone access permission for Android devices.

AgoraPlayerController: while Unity Unet library maintains the Network player’s profile, the Agora user’s id is created asynchronously. We will maintain a list of network player and a list of Agora user ids. When the game scene actually starts, we will bind the two list together into a dictionary so the Agora id can be looked up by using a Networkplayer’s profile. (We don’t need this binding mechanism if user id is known. In an actual production project, it is recommended to let the game server to provide the user ids to set for JoinChannel() call.)

AgoraUIManager: position the container game object to top right location of the game screen. It provides three toggling functions:

Mic On : mute the audio input.
Cam On: mute the local camera streaming and turn off the display of the local player.
CamSwitch: switch between the front camera or the back camera on the mobile device.

Tanks Code Modifications

We will interact the above controllers code into the existing project by updating the Tanks code in the following classes:

TankManager

Add a field to bring in the VideoSurface instance that we added to the Plane and drag the Plane game object from the children to the field.

Add a constant to name the video-surface.

public const string **LocalTankVideoName **= "Video-Local";

Change code near the end of the initialize() method, where it looked like this before:

old code

The new code:

Discussion: here is the code to associate the plane display that we created earlier to render the video feed. The VideoSurface script handles this work. The only thing it needs is the Agora Id. If it is the local player, the Agora Id will be default to 0, and the SDK will automatically render the device’s camera video onto the hosting plane. If this is a remote player, then the non-zero Agora Id is required to get the stream to render.

Calls JoinChannel()

The JoinChannel() function calls in AgoraVideoController class establish the local player status and starts a channel server. There are three places to initiate the call.

*CreateGame.cs: **add a line to the *StartMatchmakingGame() function inside the callback. It will look like this:

    private void StartMatchmakingGame()
    {
       GameSettings settings = GameSettings.s_Instance;
       settings.SetMapIndex(m_MapSelect.currentIndex);
       settings.SetModeIndex(m_ModeSelect.currentIndex);

       m_MenuUi.ShowConnectingModal(false);

       Debug.Log(GetGameName());
       m_NetManager.StartMatchmakingGame(GetGameName(), (success, matchInfo) =>
          {
             if (!success)
             {
                m_MenuUi.ShowInfoPopup("Failed to create game.", null);
             }
             else
             {
                m_MenuUi.HideInfoPopup();
                m_MenuUi.ShowLobbyPanel();
                AgoraVideoController.instance.JoinChannel(m_MatchNameInput.text);
             }
          });
    }

LevelSelect.cs: add the call in OnStartClick(). And the function will look like this:

    public void OnStartClick()
    {
       SinglePlayerMapDetails details = m_MapList[m_CurrentIndex];
       if (details.medalCountRequired > m_TotalMedalCount)
       {
          return;
       }

       GameSettings settings = GameSettings.s_Instance;
       settings.SetupSinglePlayer(m_CurrentIndex, new ModeDetails(details.name, details.description, details.rulesProcessor));

       m_NetManager.ProgressToGameScene();
      AgoraVideoController.instance.JoinChannel(details.name);
    }

LobbyServerEntry.cs: add the call in JoinMatch(). Modify the function signature to add string channelName. And the function will look like this:

    private void JoinMatch(NetworkID networkId, string channelName)
    {
       MainMenuUI menuUi = MainMenuUI.s_Instance;

       menuUi.ShowConnectingModal(true);

       m_NetManager.JoinMatchmakingGame(networkId, (success, matchInfo) =>
          {
             //Failure flow
             if (!success)
             {
                menuUi.ShowInfoPopup("Failed to join game.", null);
             }
             //Success flow
             else
             {
                menuUi.HideInfoPopup();
                menuUi.ShowInfoPopup("Entering lobby...");
                m_NetManager.gameModeUpdated += menuUi.ShowLobbyPanelForConnection;

                AgoraVideoController.instance.JoinChannel(channelName);
             }
          });
    }

   public void Populate(MatchInfoSnapshot match, Color c)
    {
       string[] split = match.name.Split(new char[1] { '|' }, StringSplitOptions.RemoveEmptyEntries);
       string channel_name = split[1].Replace(" ", string.Empty);
       m_ServerInfoText.text = channel_name**;

       m_ModeText.text = split[0];

       m_SlotInfo.text = string.Format("{0}/{1}", match.currentSize, match.maxSize);

       NetworkID networkId = match.networkId;

       m_JoinButton.onClick.RemoveAllListeners();
       m_JoinButton.onClick.AddListener(() => JoinMatch(networkId, **channel_name**));

       m_JoinButton.interactable = match.currentSize < match.maxSize;
   }

NetworkManager.cs

Insert code for player leaving the channel in Disconnect():

    public void Disconnect()
    {
       switch (gameType)
       {
          case NetworkGameType.Direct:
             StopDirectMultiplayerGame();
             break;
          case NetworkGameType.Matchmaking:
             StopMatchmakingGame();
             break;
          case NetworkGameType.Singleplayer:
             StopSingleplayerGame();
             break;
       }
       AgoraVideoController.instance.LeaveChannel();
    }

That’s basically all code changes we need to get the video streaming on local and remote player working! But wait, there is a catch we missed. The plane changes rotation with the tank when moving! See one of the tilted position:

Tank Moved with Plane

We will need another script to fix the rotation:

Build the project deploy the game to iOS or Android devices and start playing to a friend! You may see other person’s face (and yours) on top of the tanks and you can yell to each other now!

So, we are done building a fun project!

Bear vs Duck in a Tank Battle!

The current, complete code is hosted on Github.

Other Resources

The complete API documentation is available in the Document Center.
For technical support, submit a ticket using the Agora Dashboard or reach out directly to our Developer Relations team devrel@agora.io
Come join the Slack community: https://agoraiodev.slack.com/messages/unity-help-me

Video Chat with Unity3D and AR Foundation — Chapter 3: Remote Assistant App

Rick Cheng — Tue, 07 Apr 2020 05:35:00 +0000

Video Chat with Unity3D and AR Foundation — Chapter 3: Remote Assistant App

Previously, we had shown two relatively simple sample applications on how to do video chat with AR Foundation on Unity. Here are the links to the blogs:

Questions continue to come up:

“How do I share AR objects from the sender’s AR camera?”

“Also, I don’t want to share my HUD UIs, how do I do that with screen sharing?”

To address these technical challenges, we will make a more sophisticated application on the basis of knowing how to utilize AgoraIO’s Video SDK. Some of you may have already seen from this blog about a Remote Assistant App on iOS. We will make a similar App on Unity3D with AR-Foundation.

Prerequisites

Unity Editor (Version 2019.2 or above)
2 devices to test on (one to broadcast, one to view)
Broadcast device will be a mobile device to run AR scene: Apple device of iOS 11 or above; Android device with API level Android 7 or above.
Viewing device does not need AR capability — pretty much any device with Windows, Mac, Android, or iOS operating systems will work
A developer account with Agora.io

Getting Started

To start, we will need to integrate the Agora Video SDK for Unity3D into our project by searching for it in the Unity Asset Store or click this link to begin the download.

Figure 0. Video SDK on Asset Store

This project should be compatible with latest SDK. In case this tutorial could not keep to date, you may fall back to the current SDK archive here.

After you finish downloading and importing the SDK into your project, you should be able to see the README files for the different platforms the SDK supports. You should be familiar with the set up for the demo by following the previous tutorials.

Unity AR Packages

On UnityEditor (2019), open Package Manager from the Window tab. Install the following packages:

AR Foundation 3.0.1
ARCore XR Plugin 3.0.1
ARKit XR Plugin 3.0.1

If you use other versions of Unity Editor please check with the README file on GitHub for the verified setups. Or you may want to try different versions, since AR Foundation is changing a lot between versions.

Project Set Up

Please follow this link to download the completed project “RemoteAssistantAR” on Github.

Open the project in Unity Editor. Go to the Unity Asset Store page to import Agora Video SDK. When importing the Agora Video SDK, unselect “demo” from the list since those file names are modified in the Github repo. Switch to iOS or Android platform and follow the README instruction on how to set up the build environment for AR Foundation.

Open up the Main scene, and your project should look like this:

Figure 1. Main Scene

You should fill your App ID into the field of the GameController object on this screen. You may then build the project for iOS or Android to run on device. Just like the previous chapter, you will need a mobile device to run as the broadcaster and another mobile or desktop device (include Unity Editor) to run as the audience.

Project Architecture

The RemoteAssistantAR project consists of three scenes. Here are their descriptions:

Main — the entry screen with buttons to open the two screens for different purposes.
AudPlay —audience client, in which the user sees the the remote client’s video stream on the main screen; the user can choose color and draw things on the screen.
CastAR — the AR camera client, in which the user sees the real world on the back camera, mixed with AR objects; the user casts what he/she sees to the audience client.

This picture shows relationship among the scenes:

Figure 2. Scenes

Shared Interfaces

The AudPlay client and the CastAR client share the programming interface IVideoChatClient. As you can see from Figure 2, the two clients also share a common user interface in the prefab ChatCanvasUI, as depicted below:

Figure 3. ChatCanvasUI Prefab

Color Controller

A color controller is the additional UI controller that the AudPlay client contains. User may tap the ink droplet and get a list of color to pick. The follow picture shows the construct of this prefab:

Figure 4.1 Color Controller

BroadCast View Controller

The BroadCast View Controller (or “BroadcastVC”), in most sense, is the equivalent client to the Chapter 2’s “ARClient”. You may go back to read about how to use external video source to share the video frames to the receiver. However, in this controller, we capture the AR camera image in a different way. We are actually getting the raw image data from a RenderTexture, in this method:

RenderTexture is the key to answer those questions at the beginning of this tutorial.

An Approach to Share AR Screen

So far, we’ve discovered two ways to share screens. The first way, is to share the entire screen as if you are continuously making screen capture of your devices and send it. This is shown in the blog “How to Broadcast Your Screen with Unity3D and Agora.io”. The second way was discussed in my previous tutorial. The view from the AR Camera was shared without AR objects. What we want for this application is in between — we want the AR camera and AR objects (called it “AR Screen”) but not the HUD/UI elements. There could be different ways to achieve this. One way is the use of RenderTexture. In summary, it will follow these steps:

Create multiple cameras that uses different culling masks.
Assign everything under the Canvas UI to the “UI” layer.
Label/assign in code the AR Objects to another layer, e.g., “ARLayer” (anyhow, I picked the built-in “Ignored Raycast” for convenience, see Figure 4.2 and the DrawDot function in Figure 7.3).
Assign “Target Texture” field in the AR Camera to a Rendered Texture.
Assign the same Rendered Texture to another Camera (“Render Camera”) that will render the AR Objects.
Create a Material to use the Render Texture.
Create a Quad to use the Material from the last step.
Create a camera (“View Camera”) to render the quad texture.

Figure 4.2 Layer

Cameras

So far, there are three cameras being used in the CastAR scene. They are laid out in the following wireframe:

Figure 5. Wireframe of the Cameras

AR Camera: Takes the actual physical back camera from a device as input, sets each frame as output to a RenderTexture, e.g., CasterRenderTexture. Figure 6.1 illustrates the important parameters for this camera:

Figure 6.1 — AR Camera

Render Camera: **Renders *3D AR objects. One may ask, why can’t AR Camera be used for 3D object rendering? The reason being, is that the AR Camera is a physical camera. As we can see from the last tutorial, the 3D objects are not captured in this camera. Our Render Camera is placed on the same world position as the AR Camera. And its output is also being sent to the same RenderTexture — *CasterRenderTexture. See figure 6.2:

Figure 6.2 — Render Camera

View Camera: Looks at the quad, which uses a material that “reads” from the shared RenderTexture CasterRenderTexture. This camera provides the actual “view” for the user using this device. Note that since both AR Camera and Render Camera are rendering to the RenderTexture, the image is not rendered to the screen itself. That’s why this View Camera came into place. Moreover, the positions of View Camera and Quad are separated from AR Camera and Render Camera, so that the Quad object is not taken into the Render Camera’s view.

From the Hierarchy view in Figure 6.2, we may also see that View Camera *parents *Quad, so Quad turns and changes position with the camera’s movement. Same parent-child relationship applies to AR Camera and both View Camera *and *Render Camera. Their view angles and positions are kept in-sync this way.

Figure 6.3 — View Camera

The follow diagram illustrates the relationships between the cameras and the client views:

Figure 6.4 — Relationship Diagram

Next we will discuss the details about the remote assistant Part.

Remote Assistant

The idea about the remote assistant app, is that a field operator may be out to an area to work on something, but he needs some guidance on where the things to look at from a help elsewhere. Our CastAR client will be used the field operator in this scenario; and a help operator uses the AudPlay client to draw an outline of the important part on the screen. Technically, we will need to do this in the following steps:

Capture the finger motion on AudPlay client’s screen and record the touch points.
Send the touch points to CastAR client.
CastAR client receives the data points, and renders the outline in its 3D space.

Data model

We will use a small sphere to represent a dot, and the user’s dragging motion will create a series of small colored spheres. We will establish the model for our remote drawing data:

Unlike the iOS version of this App, we don’t need to worry about the offset of screen points for different devices. We will convert the points from the Screen Space to ViewPort Space for the transport. This page will help you to refresh your memory about the different point systems in Unity. A point from screen touch is “normalized” in this line:

Vector3 vp = Camera.main.ScreenToViewportPoint(screenPos)

We use a buffer list to accumulate the touch points to limit the number of the data stream call, which can be a bit of overhead adding to the performance measure.

Data Stream

On the AudPlay client side, one of the initialization steps in AudienceVC involves the creation of the data stream, in the following line:

dataStreamId = rtcEngine.CreateDataStream(reliable: true, ordered: true);

The dataStreamId is later used in the send stream message method as following:

On the CastAR client side, we dedicated a game object “DrawListener” to listen to the data stream and process the request.

Figure 7.1 — DrawListener

The DrawListener’s controller script RemoteDrawer registers a callback to handle the data stream event:

rtcEngine.OnStreamMessage += HandleStreamMessage;

The data passing between the two ends is in character string, which is a JSON formatted string. And we invoke this method to draw the dots in the CastAR client’s 3D world:

Here the referenceObject is the sphere. It shares the same parent of the container of the dots to be drawn. The DeNormalizedPosition() function on line 19 does the opposite of what we did previously in Normalize():

camera.ScreenToWorldPoint(pos);

The important thing here, is to use the correct camera for the viewport space. Since the Render Camera is responsible to render the AR 3D objects, it is used for the conversion here.

Conclusion

Linking all the information above should provide you a better understanding of the design and the technical implementation of the RemoteAssistantAR project in Unity.

Some known issues here:

We skipped the “Undo” feature that exists in the iOS sample project, instead, we have the “Clear” button to clear all the drawing once.
The drawn outline may not “stay” on the object if the camera keeps moving around. This is due to the hardcoded z-space for the drawing object. A possible fix may be using RayCast to determine where to place the Z position at.
There is a lag after both devices joining a channel to start the video streaming.
Video recording of the app (through iOS controls or desktop Quicktime Player) causes longer lags.

There are definitely plenty room of improvement here. And the multi camera solution for AR screen sharing is not the only valid approach. Please let me know if there is any suggestion or even better — make a Pull Request of your modification on Github will be very welcomed!

Other Resources

The complete API documentation is available in the Document Center.
For technical support, submit a ticket using the Agora Dashboard or reach out directly to our Developer Relations team devrel@agora.io
Come join the Slack community: https://agoraiodev.slack.com/messages/unity-help-me

Video Chat with Unity3D and AR Foundation — Chapter 2: ScreenSharing

Rick Cheng — Fri, 03 Apr 2020 23:03:18 +0000

Video Chat with Unity3D and AR Foundation — Chapter 2: ScreenSharing

Many of you may have read my previous blog about the simple modification to the demo App to incorporate AR Foundation. If not, you may go back to read the story, and use it for the set up for this new project. The most frequently asked question from the previous tutorial may have been

“How do I share my AR camera screen?”

In this tutorial, we will emphasize how to use external video source to stream the AR camera view to the remote user. To get the best from AR Foundation, we will use the latest stable version of the Unity Editor.

Prerequisites

Unity Editor (Version 2019.2 or above)
2 devices to test on (one to broadcast, one to view)
Broadcast device will be a mobile device to run AR scene: Apple device of iOS 11 or above; Android device with API level Android 7 or above.
Viewing device does not need AR capability: pretty much any device with Windows, Mac, Android, or iOS operating systems will work
A developer account with Agora.io

Getting Started

To start, we will need to integrate the Agora Video SDK for Unity3D into our project by searching for it in the Unity Asset Store or click this link to begin the download. Note the current SDK version is archived here, in case that the future SDK release has a different layout.

Video SDK on Asset Store

Unity AR Packages

On UnityEditor (2019), open Package Manager from the Window tab. Install the following packages:

AR Foundation 3.0.1
ARCore XR Plugin 3.0.1
ARKit XR Plugin 3.0.1

Project Set Up

Please follow this link to download the completed project “ARUnityClient” on Github.

Open the project in Unity Editor. Go to the asset store page to import Agora Video SDK. When importing the Agora Video SDK, unselect “demo” from the list since those file names are modified in the Github repo. Switch to iOS or Android platform and follow the README instruction on how to set up the build environment for AR Foundation.

Open up the SceneHome scene, your project should look like this:

Home Scene

Again, the project is based on the original demo that comes with the SDK. Scenes and scripts are modified to run the screen sharing. You should fill in your App ID in the field of GameController before running the demo.

Project Architecture

The ARUnityClient consists of three scenes and here are their descriptions:

Home — the entry screen with buttons to open the two screens for different purposes.
HelloVideo — the 1-to-1 chat client, in which the remote user is shown on the big screen while the local user is in the small image box.
ARClient — the big screen will show the real-world environment with AR objects: a sphere for anchoring, and a cube hosts remote user’s video.

This picture shows their relationship:

The application will require two devices to run. There are two scenarios:

Both devices use “1-to-1 Chat”.
One device uses “1-to-1 Chat”, while the other uses “AR Camera”.

=> You may not want to run “AR Camera” on both device.

We will focus on Scenario 2.

Shared Interface

The AR Camera client and the 1-to-1 client both implement the IVideoChatClient interface for the following methods:

TestUnityARClient.cs and TestHelloVideo.cs implement their version of the client on this common interface. User chooses the client on the Home screen to create the instance of the client and load the respective scene.

AR Camera Client

AR Camera client’s join(channel) method is implemented this way:

Notice the following lines are different from what we wrote in previous tutorial:

         //mRtcEngine.EnableLocalVideo(false);
         CameraCapturerConfiguration config = new CameraCapturerConfiguration();
         config.preference =      CAPTURER_OUTPUT_PREFERENCE.CAPTURER_OUTPUT_PREFERENCE_AUTO;
         config.cameraDirection = CAMERA_DIRECTION.CAMERA_REAR;
         mRtcEngine.SetCameraCapturerConfiguration(config);
         mRtcEngine.SetExternalVideoSource(true, false);

The code snippet above does the following:

Set the automatic video quality preference.
Use the rear camera for capturing video
The video sent out from an external video source. What this means is that the stream won’t be taken directly off the camera. Instead, we will encapsulate a source of video raw data manually.

So for the video stream on AR Camera client, there are two steps of work done in continuous fashion:

A. Capture the image from AR camera

B. Send the image raw data to the external video source API

The source code is listed in TestUnityARClient.cs. We will discuss the important parts of the code that addresses these steps.

Step A — AR Foundation Camera Image capture

The Unity documentation provides very insightful help on how to retrieve the data from AR Camera on CPU.

The first thing you will need is the access to the ARCameraManager. Make sure the component is added to the AR Camera instance in the scene, in our example - the ARClient scene. We called the instance of the ARCameraManager “cameraManager” in code.

ARCameraManager

Next, you will need to register a delegate to wake a function to collect the raw data from the image on each camera frame.

cameraManager.frameReceived += OnCameraFrameReceived;

On each frame received from the AR camera, we use cameraManager to get the latest image, and extract the raw binary data into a byte array. Lastly send to the external video source. The complete function snippet is following:

Note that line 39, which converts the raw data into binary bytes, is actually accessing a memory pointer for the copy operation. Therefore this is an “unsafe” operation in such context.

image.Convert(conversionParams, new System.IntPtr(buffer.GetUnsafePtr()), buffer.Length);

For this reason, the CaputreARBuffer() function requires an “unsafe” modifier. And it also requires “allow unsafe code” enabled in the project setting. See this picture:

Allow unsafe code

The last line in this function is a call to run PushFrame() as a Coroutine. The PushFrame Coroutine takes the raw image bytes to send it out as the external video source. Note that on the callback delegate (line 46) we dispose the image and buffer to clean up the memory.

Step B — External Video Source

Using external video source is actually a concept you may find in another Agora demo app on screen sharing — How to Broadcast Your Screen with Unity3D and Agora.io.

The code is similar for the most part. Here is the snippet from TestUnityARClient.cs:

In summary, this function serves the purpose: create and populate an instance of the ExternalVideoFrame data structure; then send the data using the API pushVideoFrame().

Testing

Please refer to the previous tutorial or the SDK Readme file for the other project settings for AR applications. You should be able to build the project and run it on your devices without any code changes.

The following picture illustrates the two screens for the test.

All Done!

Thank you for following along. It is very straight-forward, isn’t it? If you have any questions please feel free to leave a comment, DM me, or join our Slack channel agoraiodev.slack.com/messages/unity-help-me. There will be another chapter coming. We will make an even more complex application on AR Foundation. See you next time!

Other Resources

Next AR Foundation Chapter

The complete API documentation is available in the Document Center.

For technical support, submit a ticket using the Agora Dashboard or reach out directly to our Developer Relations team devrel@agora.io

Video Chat with Unity3D, the ARFoundation Version

Rick Cheng — Thu, 02 Apr 2020 00:53:47 +0000

Video Chat with Unity3D, the ARFoundation Version

Many of you took steps in creating your very own video chat app using our How To: Create a Video Chat App in Unity. Now, that you’ve got that covered, let’s take your app a couple notches by adding an immersive experience to it. In this Augmented Reality (AR) tutorial, you’ll learn how to communicate and chat with friend in AR. This quick start guide is very similar to the previous tutorial. We just need to make few changes to make it work in less than an hour.

Prerequisites

Unity Editor (Version 2018 or above)
2 devices to test on (one to broadcast, one to view)
Broadcast device will be a mobile device to run AR scene: Apple device of iOS 11 or above; Android device with Android 7 or above.
Viewing device will run the standard Video Chat demo app - pretty much any device with Windows, Mac, Android, or iOS operating systems will work
A developer account with Agora.io

Getting Started

Video SDK on Asset Store

After you finish downloading and importing the SDK into your project, you should be able to see the README.md files for the different platforms the SDK supports. For your convenience, you can also access the quick start tutorials for each platform below.

Android- Here
iOS- Here

Unity AR Packages

On UnityEditor, open Package Manager from the Window tab. Install the following packages:

For Unity 2018:

AR Foundation 1.0.0 — preview.22 (the latest for 1.0.0)
ARCore XR Plugin 1.0.0 — preview.24 (the latest for 1.0.0)
ARKit XR Plugin 1.0.0-preview.27 (the latest for 1.0.0)

For Unity 2019:

AR Foundation 3.0.1
ARCore XR Plugin 3.0.1
ARKit XR Plugin 3.0.1

Modify the Existing Project

Modify the play scene

Open up the TestSceneHelloVideo scene. Take out the Cube and Cylinder. Delete the Main Camera since we will use an AR Camera later.

Test Scene — before

On the Hierarchy panel, create a “AR Session” and “AR Session Origin”. Click on the AR Camera and change the tag to “Main Camera”, then create a Sphere 3D object. Modify its transform position to (0,0,5.67) so it can be visible in your Editor Game view and Save.

Test Scene — after

Unlike the Cube or Cylinder, the purpose of this sphere is just for positional reference. You would find this sphere in your AR view when running on a mobile device. We will need to add video view objects relative to this sphere’s position by code.

Modify Test Scene Script

Open TestHelloUnityVideo.cs, change the onUserJoin() method to generate a cube instead of a plane. We will add a function to provide a new position for each new remote user joining the chat.

In the Join() method, add the following line in the “enable video” section:

mRtcEngine.EnableLocalVideo(false);

This call disables the front camera, so it won’t get into conflict with the back camera which is used by the AR Camera on the device.

Last but not the least, fill in your APP ID for the variable declared in the beginning section of the TestHelloUnityVideo class.

If you haven’t already go to Agora.io, log in and get an APP ID (it’s free). This will give you connectivity to the global Agora Real-time Communication network and allow you to broadcast across your living room or around the world, also your first **10,000 minutes are free every month.

Build Project

The configuration for building an ARFoundation enabled project is slightly different from the standard demo project.

Here is a quick checklist of things to set:

IOS:

Rendering Color Space = Linear
Graphics API = Metal
Architecture = ARM64
Target Minimum iOS Version = 11.0
A unique bundle id

Android:

Graphics API = GLES3
Multithreaded Rendering = off
Minimum API Level = Android 7.0 (API level 24)
Create a new key store in the Publishing Settings

Now build the Application to either iOS or Android. Run the standard demo application for the remote users, from any of the four platforms that we discussed at the beginning of this tutorial. To test your demo, stand up and use the device to look around you and you should find the sphere. A joining remote user’s video will now be placed on the cubes next to the sphere.

Great job! You’ve built a simple AR world of video chatters!

The ARFoundation Demo

All Done!

Thank you for following along. If you have any questions please feel free to leave a comment, DM me, or join our Slack channel agoraiodev.slack/unity-help-me.

Note there will be another two chapters of the AR Foundation Video Chat tutorial with deeper contents. Stay in touch!

Other Resources

The complete API documentation is available in the Document Center.
For technical support, submit a ticket using the Agora Dashboard or reach out directly to our Developer Relations team devrel@agora.io