DEV Community: ysc1995

Build a Streaming Application with Face Filter on Android

ysc1995 — Wed, 29 Jul 2020 00:24:24 +0000

Over the last decade, live streaming has gradually become one of the most popular ways of entertainment. More and more people are enjoying sharing their lives to the public through live streaming.

Nowadays, there are plenty of streaming application choices such as Twitch, Facebook Live, and Youtube Live; these apps provide many fun features such as face filter and voice changer. But have you ever thought about building a streaming application on your own? Are you hesitating because some of these features seem too fancy or complicated?

Luckily, using the Agora Video SDK and Banuba Face AR SDK makes developing a streaming application with a face filter quick and easy. Today, I’ll show you how to do this on Android step by step.

Prerequisites

A basic-to-intermediate understanding of Java and Android SDK
Agora.io developer account
Banuba Face AR demo app
Android Studio and 2 Android devices

Please Note: While no Java/Android knowledge is needed to follow along, certain basic concepts in Java/Android won’t be explained along the way.

Overview

This guide will go over the steps for building a live streaming application on Android using the Agora SDK. This is a list of the core features that will be included in our app:

Users (streams and audiences) can create and login into their account. User account information will be saved in Google Firebase Realtime Database.
Users can set up virtual rooms to host live streams and become a streamer.
Users can find all live streams and join the virtual room as an audience.
Streamers can use the face filter feature to stream with a virtual mask or animation.
Streamers can change their voice through a voice changer.
Audiences in a virtual room can send text messages which can be seen by everyone in that room.
Users can search for other users by their name and send private text messages to them.

You can find this demo app as a reference for this article.

Download Banuba FaceAR Demo App

It is easier to build our application on top of the Banuba FaceAR demo apps. The package contains a default Banuba SDK Demo app demonstrating major Banuba SDK features and a beautification example app with neural networks coloring feature. We are going to focus on the Banuba SDK Demo app for our project. Feel free to explore the beautification example app on your own.

Now, download the Banuba FaceAR demo app and open it in Android Studio. Follow the online instruction to set up the project.

You need to have a Banuba client token to run the project. Contact info@banuba.com for more information.

After you’re done, run the “app” project and you should be able to see something like this with a lot of cool face filter options:

This will be our streamer’s view when he starts a live stream. However, the Banuba demo only adds the face filters on the local camera view. If we want to send this to all of the audience, that is where Agora Video SDK comes very handy.

Integrate Agora Video SDK

To integrate Agora SDK in your project, you need to add the following lines in the /app/build.gradle file of the project.

//Agora RTC SDK for video call
implementation 'io.agora.rtc:full-sdk:3.0.1.1'

//Agora RTM SDK for chat messaging
implementation 'io.agora.rtm:rtm-sdk:1.2.2'
//Agora UIKit
implementation 'io.agora.uikit:agorauikit:2.0.1'

The first implementation is for the Agora Video SDK. While the second one is for the Agora Real-time Messaging SDK. And the last one is for Agora Android UIKit which is a Agora Android UI wrapper. Using the UIKit, you just need a few lines of code to start a video call. And I will demonstrate that in the later section of this blog when we are dealing with the audience view.

Add Project Permissions

The next thing we need to do is to add project permissions in the /app/src/main/AndroidManifest.xml file for device access according to your needs:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    ...>
<uses-permission android:name="android.permission.READ_PHONE_STATE" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" /> 
<!-- The Agora SDK requires Bluetooth permissions in case users are using Bluetooth devices. -->
<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
...
</manifest>

Setup Google Firebase Database

Since our application allows users to search and chat with other users in the app, we need to use Firebase Realtime Database to save users’ account information. Here are the steps you need to connect your application to Firebase Realtime Database:

In your Android Studio, Click “Tools”, then select “Firebase”.
On the right hand side, you should see a Firebase assistant tab showing up. Find “Realtime Database” and click “Save and retrieve Data”.
Then you should see the detailed page for Realtime Database. Click the “Connect to Firebase” button and “Add the Realtime Database to your app” button. Follow the on-screen instruction, if any.
Now your application is connected to Firebase Realtime Database. The last thing you need to do is to go to your Firebase console and change the database rules. Remember to select the Realtime Database instead of the Cloud Firestore. Change the rules for read and write to “true” so that you can have access to the database.

User Login

First, let’s build our application from the first page. Same as most of the streaming applications, our application needs a user login page. And that should look like this:

When the user clicks the “Next” button, we need to get the user’s name and pass that to the next activity, HomeActivity. Here is the code for that.

public void onLoginNextClick(View view) {
    EditText userNameEditText = findViewById(R.id.et_login_user_name);
    String userName = userNameEditText.getText().toString();

    if(userName == null || userName.equals("")) {
        Toast.makeText(this, "user name cannot be empty", Toast.LENGTH_SHORT).show();
    }else {
        Intent intent = new Intent(this, HomeActivity.class);
        intent.putExtra("userName", userName);
        startActivity(intent);
    }
}

Building HomeActivity

In the HomeActivity, we will create a three-page bottom navigation bar with a fragment connected to each page. If the user taps on the first bottom navigation menu, he will be directed to the Live fragment. Similarly, the second menu will direct the user to the Chat fragment and the last one is the Other fragment.

In the onCreate method of the HomeActivity, we will get the user name passed from the previous activity and save that user information on Firebase Database.

@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_home);
    userName = getIntent().getStringExtra("userName");

    saveUserOnFireDatabase(userName);
}
private void saveUserOnFireDatabase(String userName) {
    mRef = FirebaseDatabase.getInstance().getReference("Users");
    mRef.push();
    mRef.child(userName).setValue(new DBUser(userName, false));
}

Live Fragment

The Live fragment is to show all the live streamers’ names and allow other users to join the streamer’s virtual room by clicking a button.

In the center of the fragment is a recycler view displaying all the live streamers. You can retrieve the streamer data from the Firebase database by registering a ChildEventListener on the Firebase database reference.

childEventListener = new ChildEventListener() {
    @Override
    public void onChildAdded(@NonNull DataSnapshot dataSnapshot, @Nullable String s) {
        DBUser result = dataSnapshot.getValue(DBUser.class);
        if (!result.getName().equals(userName) && result.getStreaming() == true) {
            streamerList.add(result);
        }
    }

    ...
};

mRef.orderByChild("name").addChildEventListener(childEventListener);

On the side of the streamer’s name, there is a button for users to join that virtual room. And that will direct users to the Audience activity using an Intent.

private void joinStream(String streamerName) {
    Intent intent = new Intent(getActivity(), AudienceActivity.class);
    intent.putExtra("streamerName", streamerName);
    intent.putExtra("userName", userName);
    startActivity(intent);
}

When the user clicks the “GO LIVE” button on the bottom, he will be directed to the MainActivity using an Intent. The MainActivity is the activity to enable the Banuba face filter features which we have discussed above.

@Override
public void onClick(View v) {
    Intent intent = new Intent(getActivity(), MainActivity.class);
    intent.putExtra("userName", userName);
    startActivity(intent);
}

Chat Fragment

The Chat fragment is the place for users to search for other users and send private text messages to them. So you will build your UI like this:

Send Local Camera View to Remote Users

Let’s open the MainActivity class. From the previous steps, we know this activity is for adding the face filter features on the local camera views. Our job here is to send these local camera views with face filters out to the remote users.

Initialize RtcEngine and Join Channel

Inside the onCreate callback, get the username passed from the home activity and initialize the Agora RtcEngine.

userName = getIntent().getStringExtra("userName");

try {
    mRtcEngine = RtcEngine.create(getBaseContext(), getString(appID), mRtcEventHandler);
} catch (Exception e) {
    Log.e(TAG, Log.getStackTraceString(e));
    throw new RuntimeException("NEED TO check rtc sdk init fatal error\n" + Log.getStackTraceString(e));
}

mRtcEngine.enableVideo();

Here, you need to put the appID as a parameter when initialize the RtcEngine. In order to get the appID in the parameter, follow these steps:

Create an Agora project in the Agora Console.
Click the Project Management tab on the left navigation panel.
Click “Create” and follow the on-screen instructions to set the project name, choose an authentication mechanism, and click “Submit”.
On the Project Management page, find the App ID of your project.

The mRtcEventHandler is a handler to manage different events occurring with the RtcEngine.
Then we need to call setExternalVideoSource to switch Agora video input to an external video source which is the camera frame from Banuba SDK.

mRtcEngine.setExternalVideoSource(true, false, true);

After that we are ready to join the channel by calling joinChannel() on the RtcEngine instance. The channelName is the user name we get from the previous activity. In this case, each streamer will start a video call channel with his entered user name when he login.

mRtcEngine.joinChannel(token, channelName, "Extra Info", 0);

Please Note: The token in the parameter can be set to null. You can get more information about token here.

Push External Video Frame

We have already let the Agora RtcEngine know that we will use the external video source as the video input. However, we haven’t provided the video frame yet.

First, call startForwardingFrames() on BanubaSDKManager instance and that will trigger the onFrameRendered() callback.

mSdkManager.startForwardingFrames();

Now, in the onFrameRendered() callback, we can get the camera frame data which contains the local camera view and the AR Face Filter view. Then create an AgoraVideoFrame instance and pass this information in. Call pushExternalVideoFrame() on the Agora RtcEngine instance and pass the AgoraVideoFrame instance in the parameter.

@Override
public void onFrameRendered(@NonNull Data data, int width, int height) {
    byte[] arr = new byte[data.data.remaining()];
    data.data.get(arr);

    AgoraVideoFrame agoraVideoFrame = new AgoraVideoFrame();
    agoraVideoFrame.buf = arr;
    agoraVideoFrame.stride = width;
    agoraVideoFrame.height = height;
    agoraVideoFrame.format = AgoraVideoFrame.FORMAT_RGBA;
    agoraVideoFrame.timeStamp = System.currentTimeMillis();
    mRtcEngine.pushExternalVideoFrame(agoraVideoFrame);
    data.close();
}

Now these cool face filters and animations can be sent to the remote users who are in the same Rtc channel.

You can test it by using Agora Web Showcase. Enter your appID, token and channel name and you should be able to see the video.

Add the Voice Changer

To make the streaming experience more interesting, we can add a voice changer in the application. Create a button in the MainActivity and set the onClick method for that. When the user clicks that, we will call setLocalVoiceChanger to set the voice changer.

public void onVoiceChangerClick(View view) {
    mRtcEngine.setLocalVoiceChanger(VOICE_CHANGER_BABYGIRL);
    Toast.makeText(this, "Voice Changer ON", Toast.LENGTH_SHORT).show();
}

There are many different voice changers you can choose. Here I used the baby girl’s voice. For more voice changer options, check the setLocalVoiceChanger api.

Set Up Audience View

Setting up the audience view is easier. Let’s go to the Audience activity and use the Agora UIKit.

To use the UIKit, simply call this in the onCreate() method.

@Override
protected void onCreate(Bundle savedInstanceState) {
    ...
    AgoraRTC.instance().bootstrap(this, appID, channelName);
    setContentView(R.layout.group_template);
}

Here, the appId is the same appID we entered in the MainActivity and the channelName is the streamer’s name we passed from the LiveFragment.

For more information, visit UIKit Github.

Just 2 lines of code, you are good to start a video call!

However, in our case, we need to add some customizations to make it to the audience view. First, we don’t want the audience to send the video and audio streams to the channel.

AgoraRTC.instance().muteLocalVideoStream(true);
AgoraRTC.instance().muteLocalAudioStream(true);

Also, we want the remote view (streamer view) always on the main screen. To do this, we need to register the IRtcEnginEventHandler instance.

IRtcEngineEventHandler eventHandler = new IRtcEngineEventHandler() {

    @Override
    public void onRemoteVideoStateChanged(final int uid, int state, int reason, int elapsed) {
        super.onRemoteVideoStateChanged(uid, state, reason, elapsed);

        if (state == REMOTE_VIDEO_STATE_STARTING) {
            runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    agoraView.setUID(uid);
                }
            });
        }
    }
};
@Override
protected void onCreate(Bundle savedInstanceState) {
    ...
    agoraView = findViewById(R.id.max_view);
    AgoraRTC.instance().registerListener(eventHandler);
    ...
}

The last customization we need to do is to add a button click listener on the end call button to redirect the user to the previous activity when the user clicks the end call button.

endCallButton.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View v) {
        finish();
    }
});

Till this point, our streaming application is able to allow users to stream with face filters features on or watch the stream as an audience. Let’s make it more robust by adding the text messaging system using Agora RTM SDK.

Text Messaging

Text messaging in a streaming application can be divided into two parts: channel messaging and private messaging. The first one sends messages to the channel and everyone in the channel can receive that message while the second one sends private messages from a peer to another peer.

Channel Messaging

We want to allow all the audiences to send messages in the streamer’s room. To achieve that, we need to create a RtmClient using Agora Messaging SDK.

mRtmclient = RtmClient.createInstance(mContext, appID, new RtmClientListener() {
        @Override
        public void onConnectionStateChanged(int state, int reason){
            for (RtmClientListener listener : mListenerList) {
                listener.onConnectionStateChanged(state, reason);
            }
        }@Override
        public void onMessageReceived(RtmMessage rtmMessage, String peerId) { ... }
});

Here, we pass the same appID as we used when we initialize the Agora Video engine.

In the onCreateView method in the LiveFragment, we need to login RtmClient.

mRtmClient.login(null, userName, new io.agora.rtm.ResultCallback<Void>() {
    @Override
    public void onSuccess(Void aVoid) {
        ...
    }
    @Override
    public void onFailure(ErrorInfo errorInfo) {
        ...
    }
});

We use the same userName as the one we used to join the video call channel. In this case, users will join the RTM channel and RTC channel with the same channel name.

In the AudienceActivity onCreate() method, we are going to call createChannel() on the mRtmClient we initialized earlier to create a RtmChannel instance.

mRtmChannel = mRtmClient.createChannel(mChannelName, new MyChannelListener());

We need to define the MyChannelListener which implements RtmChannelListener with callback onMessageReceived() to handle the logic when we receive messages sent from any member in the chat room.

class MyChannelListener implements RtmChannelListener {
    @Override
    public void onMessageReceived(final RtmMessage message, final RtmChannelMember fromMember) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                String account = fromMember.getUserId();
                String msg = message.getText();
                MessageBean messageBean = new MessageBean(account, msg, false);
                messageBean.setBackground(getMessageColor(account));
                mMessageBeanList.add(messageBean);
                mMessageAdapter.notifyItemRangeChanged(mMessageBeanList.size(), 1);
                mRecyclerView.scrollToPosition(mMessageBeanList.size() - 1);
            }
        });
    }
    ...
}

Then join the channel by calling join() on the RtmChannel instance.

mRtmChannel.join(new ResultCallback<Void>() {
    @Override
    public void onSuccess(Void responseInfo) { ... }@Override
    public void onFailure(ErrorInfo errorInfo) { ... }
});

In the onSuccess() callback, we can call getMembers() on the RtmChannel instance to get the number of members in the chat room.

We also need to add some UI components to let users enter their text messages and see the chat history. The audience layout should be similar to this:

When a user clicks the “Send” button, we will send his message out to the channel by calling sendMessage() method on the RtmChannel instance.

private void sendChannelMessage(String content) {
    // step 1: create a message
    RtmMessage message = mRtmClient.createMessage();
    message.setText(content);
    // step 2: send message to channel
    mRtmChannel.sendMessage(message, new ResultCallback<Void>() {
        @Override
        public void onSuccess(Void aVoid) {
            ...
        }

        @Override
        public void onFailure(ErrorInfo errorInfo) {
            ...
        }
    });
}

Private Messaging

In the ChatFragment, users can find their friend by searching the friend’s name and send private text messages to them.

To search friends by names, you need to register the ChildEventListener and get data from Firebase Database.

childEventListener = new ChildEventListener() {
  @Override
  public void onChildAdded(@NonNull DataSnapshot dataSnapshot, @Nullable String s) {
    DBUser result = dataSnapshot.getValue(DBUser.class);
    showFriendText.setText(result.getName());
    mRef.orderByChild("name").startAt(friendName).endAt(friendName + "\uf8ff").removeEventListener(childEventListener);
  }
  ...
};

mRef.orderByChild("name").startAt(friendName).endAt(friendName + "\uf8ff").addChildEventListener(childEventListener);

Sending the private chat message is very similar as sending the group message. There are a few differences you need to pay attention to:

You don’t need to create a RTM channel to send message.
You need to register a RtmClientListener and receive messages from the onMessageReceived callback.

class MyRtmClientListener implements RtmClientListener {

    @Override
    public void onMessageReceived(final RtmMessage message, final String peerId) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                String content = message.getText();
                if (peerId.equals(mPeerId)) {
                    MessageBean messageBean = new MessageBean(peerId, content,false);
                    messageBean.setBackground(getMessageColor(peerId));
                    mMessageBeanList.add(messageBean);
                    mMessageAdapter.notifyItemRangeChanged(mMessageBeanList.size(), 1);
                    mRecyclerView.scrollToPosition(mMessageBeanList.size() - 1);
                } else {
                    MessageUtil.addMessageBean(peerId, content);
                }
            }
        });
    }

    ...
}

3.When you send the message out, call sendMessageToPeer() instead of sendMessage() which we used to send channel messages.

mRtmClient.sendMessageToPeer(mPeerId, message, mChatManager.getSendMessageOptions(), new ResultCallback<Void>() {
    @Override
    public void onSuccess(Void aVoid) {
       ...
    }

    @Override
    public void onFailure(ErrorInfo errorInfo) {
       ...
    }
});

Build And Test on Device

Now let’s run our application!

In Android Studio, make sure your Android device is plugged in, and click Run.

Done!

Congratulations! You just built yourself a live streaming application with face filter features and chat messaging!

Thank you for following along. If you want to see more features such as virtual gifting and in app gaming, please leave a comment below! Here is the email address for any of the questions you might have: devrel@agora.io.

How To Build A Drop-in Video Chat Application Using Android

ysc1995 — Thu, 14 May 2020 01:35:39 +0000

Many cities and states have been under lockdown since the outbreak of the coronavirus epidemic. During this difficult time, we’re all looking for new ways to stay connected and support each other. This is when social networking applications such as Houseparty become especially relevant and helpful.

These applications let users meet up and have fun with their friends without having to leave their homes. Users can enter their friend’s virtual room by just clicking a button. Houseparty, in particular, also provides some built-in games that users can play together.

If you’ve ever wondered how these cool applications are made, read on! This blog post will help get you started on the basics of building a similar social networking application on Android.

Prerequisites

A basic-to-intermediate understanding of Java and Android SDK
Agora.io developer account
Android Studio and 2 Android devices

Please Note: While no Java/Android knowledge is needed to follow along, certain basic concepts in Java/Android won’t be explained along the way.

Overview

This guide will go over the steps for building a social networking application similar to Houseparty. This is a list of the core features that will be included in our app:

Users can create and login into their account. User account information will be saved in Google Firebase Realtime Database.
Users can set up virtual rooms to host video calls.
Users can configure the accessibility of their virtual rooms. “Public” rooms are open for all friends to join and “private” rooms are invitation-only.
During a video call, users can send private messages to another user in the same room by double-clicking on that user’s remote video.
Users can chat with friends who are not in the room by clicking a button next to their names in the friend list.

You can find my Github demo app as a reference for this article.

Set Up New Project

To start, let’s open Android Studio and create a new, blank project.

Open Android Studio and click Start a new Android Studio project.
On the Choose your project panel, choose Phone and Tablet __> __Empty Activity, and click Next.
Click Finish. Follow the on-screen instructions if you need to install any plug-ins.

Add Project Permissions

Add project permissions in the /app/src/main/AndroidManifest.xml file for device access according to your needs:

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="com.example.housepartyagora">
<uses-permission android:name="android.permission.READ_PHONE_STATE" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" /> 
<!-- The Agora SDK requires Bluetooth permissions in case users are using Bluetooth devices. -->
<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
...
</manifest>

Integrate the SDK

Add the following line in the /app/build.gradle file of your project:

dependencies {
    ...
    implementation 'com.google.firebase:firebase-database:16.0.4'
    //Agora RTC SDK for video call
    implementation 'io.agora.rtc:full-sdk:3.0.0.2'
    //Agora RTM SDK for chat messaging
    implementation 'io.agora.rtm:rtm-sdk:1.2.2'
}

Setup Google Firebase Database

Since our application allows users to search and add their friends to their friend lists in the app, we need to use Firebase Realtime Database to save users’ account information. Here are the steps you need to connect your application to Firebase Realtime Database:

In your Android Studio, Click “Tools”, then select “Firebase”.
On the right hand side, you should see a Firebase assistant tab showing up. Find “Realtime Database” and click “Save and retrieve Data”.
Then you should see the detailed page for Realtime Database. Click the “Connect to Firebase” button and “Add the Realtime Database to your app” button. Follow the on-screen instruction, if any.
Now your application is connected to Firebase Realtime Database. The last thing you need to do is to go to your Firebase console and change the database rules. Remember to select the Realtime Database instead of the Cloud Firestore. Change the rules for read and write to “true” so that you can have access to the database.

User Login

First, let’s create a landing page for our application.

Please Note: You can find the .xml file here.

If the user clicks “SIGN UP” button, we will jump to sign up page. If the user clicks “I Already Have An Account” text, we will jump to the login page. Let’s implement the activities transaction logic using Intent.

public void onLoginButtonClick(View view) {
    Intent intent = new Intent(SplashActivity.this, LoginActivity.class);
    startActivity(intent);
}

public void onSignUpButtonClick(View view) {
    Intent intent = new Intent(SplashActivity.this, SigninActivity.class);
    startActivity(intent);
}

Now, let’s create the UIs for the login page and sign up page.

Please Note: You can find the .xml file for Login page here and Sign Up page here.

When the user clicks the “Next” button, we need to get the user’s name and pass that to the next activity, VideoCallActivity, to start the video call. Here is the code for that in the Login activity, and the Signup activity will be similar.

public void onLoginNextClick(View view) {
    EditText userNameEditText = findViewById(R.id.et_login_user_name);
    String userName = userNameEditText.getText().toString();

    if(userName == null || userName == "") {
        Toast.makeText(this, "user name cannot be empty", Toast.LENGTH_SHORT).show();
    }else {
        Intent intent = new Intent(this, VideoCallActivity.class);
        intent.putExtra("userName", userName);
        startActivity(intent);
    }
}

Save User Account In Firebase Database

In the onCreate method in the VideoCallActivity, we need to save user’s information in the Firebase Database. First, get the database reference by calling:

mRef = FirebaseDatabase.getInstance().getReference("Users");

Then we call setValue to save user information as a child in the database. User information is kept in the DBUser format. In a DBUser object, it maintains a user name, a uid, a user room state and a list of friends.

mRef.push();
mRef.child(userName).setValue(new DBUser(userName, user.getAgoraUid(), localState, DBFriend));

At this point, if you run this application, you should be able to see new user information saved in your database after you login into the app.

Start Video Call

After the user logged in into the app, he is in his virtual room by default. This user should be in a video call channel so that his friend can join his channel for a call. So in the VideoCallActivity, we need to implement the video call logic. But how to start a video call? It seems very complicated to do that. Luckily, I found out that Agora Video SDK provides the easiest way to start a video call.

First, let’s create a UI for this activity:

Please Note: You can find the .xml file here.

Then, in the onCreate() method in the VideoCallActivity, let’s do following several things:

Initialize Agora RtcEngine
Setup local video Canvas
Join channel

1. Initialize Agora Rtc Engine

In order to initialize the Agora video engine, just simply call RtcEngine.create(context, appID, RtcEventHandler) to create a RtcEngine instance.

mRtcEngine = RtcEngine.create(getBaseContext(), appID, mRtcEventHandler);

In order to get the App ID in the parameter, follow these steps:

Create an Agora project in the Agora Console.
Click the Project Management tab on the left navigation panel.
Click “Create” and follow the on-screen instructions to set the project name, choose an authentication mechanism, and click “Submit”.
On the Project Management page, find the App ID of your project.

The mRtcEventHandler is a handler to manage different events occurring with the RtcEngine. Let’s implement it with some basic event handlers needed for this application.

private final IRtcEngineEventHandler mRtcEventHandler = new IRtcEngineEventHandler() {
    @Override
    // Listen for the onJoinChannelSuccess callback.
    // This callback occurs when the local user successfully joins the channel.
    public void onJoinChannelSuccess(String channel, final int uid, int elapsed) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                user.setAgoraUid(uid);
                mRef.child(getUserName()).setValue(new DBUser(getUserName(), user.getAgoraUid(), localState, DBFriend));
            }
        });
    }

    @Override
    // Listen for the onFirstRemoteVideoDecoded callback.
    // This callback occurs when the first video frame of a remote user is received and decoded after the remote user successfully joins the channel.
    // You can call the setupRemoteVideo method in this callback to set up the remote video view.
    public void onFirstRemoteVideoDecoded(final int uid, int width, int height, int elapsed) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                SurfaceView mRemoteView = RtcEngine.CreateRendererView(getApplicationContext());

                mRemoteView.setZOrderOnTop(true);
                mRemoteView.setZOrderMediaOverlay(true);
                mRtcEngine.setupRemoteVideo(new VideoCanvas(mRemoteView, VideoCanvas.RENDER_MODE_HIDDEN, uid));
            }
        });
    }

    @Override
    // Listen for the onUserOffline callback.
    // This callback occurs when the remote user leaves the channel or drops offline.
    public void onUserOffline(final int uid, int reason) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                onRemoteUserLeft(uid);
            }
        });
    }
};

Please check the comments on the top of each event handler methods to have a better understanding of them. For more RtcEngine event handlers that you can use, check the Agora Rtc API document.

Please Note: Some of the logic to show video views on the screen is hidden. You can check the Github demo app for a better understanding of how to display and remove video views on the screen dynamically.

2. Setup local video Canvas

To start a local video (see yourself on the screen), you need to call two functions: enableVideo() and setupLocalVideo() on a RtcEngine instance. In function setupLocalVideo(), a surfaceView created by calling RtcEngine.CreateRenderView(context) is passed as a parameter.

mRtcEngine.enableVideo();
mRtcEngine.enableInEarMonitoring(true);
mRtcEngine.setInEarMonitoringVolume(80);

SurfaceView surfaceView = RtcEngine.CreateRendererView(getBaseContext());
mRtcEngine.setupLocalVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_HIDDEN, 0));
surfaceView.setZOrderOnTop(false);
surfaceView.setZOrderMediaOverlay(false);

3. Join channel

Now we are ready to join the channel by calling joinChannel() on the RtcEngine instance. The channelName is the user name we get from the previous activity. In this case, each user will join into a video call channel with his entered user name when he login.

mRtcEngine.joinChannel(token, channelName, "Extra Optional Data", 0);

Please Note: The token in the parameter can be set to null. You can get more information about token here.

By calling this function and successfully joining the channel, the RtcEngineEventHandler will trigger the onJoinChannelSuccess() method that we implemented in the previous step. It will return a unique id from Agora server. We need to update the user information with the uid in the database.

@Override
public void onJoinChannelSuccess(String channel, final int uid, int elapsed) {
    runOnUiThread(new Runnable() {
        @Override
        public void run() {
            user.setAgoraUid(uid);
            mRef.child(getUserName()).setValue(new DBUser(getUserName(), user.getAgoraUid(), localState, DBFriend));
        }
    });
}

4. Search and add friends

Like a real social application, a user should be able to search his friends by names and add them in his friend list. So when user clicks on the top right button, the app should show a search friend panel like this:

The user can enter his friend’s name in the edit text and click the search button. We need to search his friend’s name in the database and display the result in the recycler view. Here is how to search people by name in the database:

childEventListener = new ChildEventListener() {
    @Override
    public void onChildAdded(@NonNull DataSnapshot dataSnapshot, @Nullable String s) {
        DBUser result = dataSnapshot.getValue(DBUser.class);

        searchFriendList.add(result);
        mRef.orderByChild("name").startAt(searchFriendName).endAt(searchFriendName + "\uf8ff").removeEventListener(childEventListener);

        mFriendListRecyclerViewAdapter = new FriendListRecyclerViewAdapter(searchFriendList);
        mFriendListRecyclerViewAdapter.setOnItemClickListener(new FriendListRecyclerViewAdapter.ClickListener() {
            @Override
            public void onItemClick(int position, View v) {
                addFriend(searchFriendList.get(position).getName());
                mSearchFriendEditText.setText("");
                searchFriendList.clear();
                mFriendListRecyclerView.setAdapter(mFriendListRecyclerViewAdapter);
            }
        });
        RecyclerView.LayoutManager manager = new GridLayoutManager(getBaseContext(), 1);
        mFriendListRecyclerView.setLayoutManager(manager);

        mFriendListRecyclerView.setAdapter(mFriendListRecyclerViewAdapter);
    }

    ...
};

mRef.orderByChild("name").startAt(searchFriendName).endAt(searchFriendName + "\uf8ff").addChildEventListener(childEventListener);

In the recycler view, we will display the friend name searched in the database with a “Add” button on the side. If the user clicks the “Add” button, that friend is added in the user’s friend list. We also need to update the user’s friend list in the database.

public void addFriend(String userName) {
    DBFriend.add(userName);
    mRef.child(this.userName).setValue(new DBUser(this.userName, user.getAgoraUid(), localState, DBFriend));
}

5. Join a friend

Since we have an updated friend list, we can display the user’s friends panel when the user clicks on the top left button. Now the user can see his friend list and choose which friend to join the call. On the side of his friend’s name, create a “Join” button for the user to join this friend’s virtual room and have a video call with him.

public void joinFriend(String friendName){
    channelName = friendName;
    finishCalling();
    startCalling();
}

Let’s change the channelName to the user’s friend’s name so that when the startCalling method is triggered, the user will join a different channel with the same name as the name of his friend.

private void startCalling() {
    //set up local video canvas
    mRtcEngine.enableVideo();
    mRtcEngine.enableInEarMonitoring(true);
    mRtcEngine.setInEarMonitoringVolume(80);

    SurfaceView surfaceView = RtcEngine.CreateRendererView(getBaseContext());
    mRtcEngine.setupLocalVideo(new VideoCanvas(surfaceView, VideoCanvas.RENDER_MODE_HIDDEN, 0));
    surfaceView.setZOrderOnTop(false);
    surfaceView.setZOrderMediaOverlay(false);

    //join the channel
    mRtcEngine.joinChannel(null, channelName, "Extra Optional Data", 0);}

If the user wants to leave the call from his friend’s room and go back to his own virtual room, he can click on the “X” button at the center bottom to achieve that. Let’s create a boolean called isLocalCall to track if the user is in his own virtual room or not. We need to set the channelName as the user’s userName so that he will join the channel with the same channel name as his username after calling startCalling method. In this way, he is back to his virtual room.

if (isLocalCall) {
    //when the user is in his own room
    ...
    }
}else {
    //when user is joining other people's room
    //leave that room and come back to user's own room
    isLocalCall = true;
    finishCalling();
    channelName = userName;
    startCalling();
}

6. Lock the room

Sometimes, a user wants to set his own virtual room to become a “private” room so that no one else can join his room except those who are already in the room. So how should we achieve that?

Create a String called localState to track the state of the room. There are two states: “Open” or “Lock”. By default, a room state is “Open”. When the user clicks the “X” button at the center bottom while he is in his own virtual room (which mean isLocalCall == true), we need to set the room state to “Lock” to set the room state to “private”. If he clicks that again, we need to set the room state back to “Open” again. Also, we have to update the room state information in the database so that other users can know the current state for this room.

if (isLocalCall) {
    //when the user is in his own room
    if (localState.equals(Constant.USER_STATE_LOCK)) {
        //set the room to public
        localState = Constant.USER_STATE_OPEN;
        mRef.child(this.userName).setValue(new DBUser(this.userName, user.getAgoraUid(), localState, DBFriend));
        showToast("Room set to public");
    }else {
        //set the room to private so that no one can join the room
        localState = Constant.USER_STATE_LOCK;
        mRef.child(this.userName).setValue(new DBUser(this.userName, user.getAgoraUid(), localState, DBFriend));
        showToast("Room set to private");
    }
}else {
    //when user is joining other people's room
    //leave that room and come back to user's own room
    isLocalCall = true;
    finishCalling();
    channelName = userName;
    startCalling();
}

After that, every time before a user joins a friend’s channel, we need to check if that friend’s room is not private. Otherwise, we will display a toast on the screen.

joinFriendChildEventListener = new ChildEventListener() {
    @Override
    public void onChildAdded(@NonNull DataSnapshot dataSnapshot, @Nullable String s) {
        DBUser result = dataSnapshot.getValue(DBUser.class);
        if (result.getState().equals(Constant.USER_STATE_OPEN)) {
            joinFriend(DBFriend.get(position));
            mShowFriendLinearLayout.setVisibility(View.GONE);
        }else {
            showToast(DBFriend.get(position) + "'s room is locked. You can message him to say hi!");
        }

        mRef.orderByChild("name").startAt(DBFriend.get(position)).endAt(DBFriend.get(position) + "\uf8ff").removeEventListener(joinFriendChildEventListener);
    }

    ...
};
mRef.orderByChild("name").startAt(DBFriend.get(position)).endAt(DBFriend.get(position) + "\uf8ff").addChildEventListener(joinFriendChildEventListener);

At this point, users in our application are able to search and add friends, join a friend’s virtual room and lock his virtual room. Now let’s implement the chat messaging functionality for our application.

Chat Messaging

Implementing chat messaging functionality is also very simple using Agora Messaging SDK.

Start RTMClient

In order to enable chatting function in our application, we need to create a RtmClient using Agora Messaging SDK.

mRtmclient = RtmClient.createInstance(mContext, appID, new RtmClientListener() {
        @Override
        public void onConnectionStateChanged(int state, int reason){
            for (RtmClientListener listener : mListenerList) {
                listener.onConnectionStateChanged(state, reason);
            }
        }@Override
        public void onMessageReceived(RtmMessage rtmMessage, String peerId) { ... }
});

Here, we pass the same appID as we used when we initialize the Agora Video engine.

In the onConnectionStateChanged() callback, we are calling RtmClientListener. We will implement this later in the Message activity.

In the onCreate method in the VideoCallActivity, we need to login RtmClient.

mRtmClient.login(null, userName, new io.agora.rtm.ResultCallback<Void>() {
    @Override
    public void onSuccess(Void aVoid) {
        ...
    }

    @Override
    public void onFailure(ErrorInfo errorInfo) {
        ...
    }
});

We use the same userName as the one we used for video call.

Now, let’s create a “Chat” button next to the “Join” button in user’s friend list so that when user clicks it, it will display a chat messaging panel like this:

User can edit his message in the edit text and send the message to his friend by clicking the “Send” button. The messages between the user and his friend will be displayed in the recycler view.

First implement the onClick logic for the “Send” button.

public void onClickSend(View v) {
    String content = mMsgEditText.getText().toString();
    RtmMessage message = mRtmClient.createMessage();
    message.setText(content);

    // step 2: send message to peer
    mRtmClient.sendMessageToPeer(mPeerId, message, mChatManager.getSendMessageOptions(), new ResultCallback<Void>() {
    @Override
    public void onSuccess(Void aVoid) {
        ...
    }

    @Override
    public void onFailure(ErrorInfo errorInfo) {
        ...
    }
    });
}

Then we need to add logic to receive the chat messages. To receive private chat messages, we need to register RtmClientListener. Remember that when we created the RtmClient instance, we were calling RtmClientListener in the callbacks. Now, let’s implement this. In the callback onMessageReceived(), we put the logic to show the message on the RecyclerView.

class MyRtmClientListener implements RtmClientListener {

    @Override
    public void onConnectionStateChanged(final int state, final int reason) {
        ...
    }

    @Override
    public void onMessageReceived(final RtmMessage message, final String peerId) {
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                String content = message.getText();
                if (peerId.equals(mPeerId)) {
                    MessageBean messageBean = new MessageBean(peerId, content,false);
                    messageBean.setBackground(getMessageColor(peerId));
                    mMessageBeanList.add(messageBean);
                    mMessageAdapter.notifyItemRangeChanged(mMessageBeanList.size(), 1);
                    mRecyclerView.scrollToPosition(mMessageBeanList.size() - 1);
                } else {
                    MessageUtil.addMessageBean(peerId, content);
                }
            }
        });
    }
    ...
}

Now we are able to let users chat within the app. But they only can start chatting by clicking a button in their friend list. Let’s make it a little fancier! We want to allow the user to start chatting with his friend by double clicking on this friend’s remote video when they are in a video call. How can we achieve that.

First, we need to add the double click listener for the VideoViewContainer. Then, when the double click is triggered, we can get the remote user’s Agora uid and find that user’s name in the database.

chatSearchChildEventListener = new ChildEventListener() {
    @Override
    public void onChildAdded(@NonNull DataSnapshot dataSnapshot, @Nullable String s) {
        DBUser result = dataSnapshot.getValue(DBUser.class);
        startMessaging(result.getName());

        mRef.orderByChild("uid").startAt(user.mUid).endAt(user.mUid + "\uf8ff").removeEventListener(chatSearchChildEventListener);

    }

    ...
};

mRef.orderByChild("uid").startAt(user.mUid).endAt(user.mUid + "\uf8ff").addChildEventListener(chatSearchChildEventListener);

After that, we can follow the previous steps to start the messaging functionality by passing the user name.

Switch Camera And Audio Mute

At this point, all the core functionalities in our application are achieved. Let’s implement the switch camera and audio mute buttons onClick logic.

Switch camera

This button is to switch from using your mobile’s front camera to rear camera or vise versa. To do that, simply call switchCamera() on the RtcEngine instance.

mRtcEngine.switchCamera();

Audio mute/unmute

Sometimes users want to mute their input voice during a video call. That’s also easy to implement by just calling muteLocalAudioStream() on the RtcEngine instance and passing whether the user is already muted in the parameter.

mRtcEngine.muteLocalAudioStream(isMuted);

Build And Test on Device

Now let’s run our application!

Go to Android Studio, make sure your Android device is plugged in, and click Run to build the application on your device. Remember to build the application on two devices to start the video call.

Done!

Congratulations! You just built yourself a social application similar to Houseparty!

Thank you for following along. If you want to see more features such as push notification, please leave a comment below! Here is the email address for any of the questions you might have: devrel@agora.io.

How To: Build an Augmented Reality Remote Assistance App in Android

ysc1995 — Fri, 10 Apr 2020 20:15:01 +0000

This might have happened to many of us at some point: you plugged in that hair dryer or that heater and all of a sudden, all your lights went dark and your TV turned black.

Now, you have to call the customer support of your service provider in desperation. You explained what happened and they said you might just need to switch on your air circuit-breaker. However, when you open the electric panel, you find yourself wondering: which one of these is the air circuit-breaker? After a 20-minute conversation with the customer support representative trying to locate this switch, you find that you don’t have one at your house in the first place.

Most remote assistance today is done through voice calls or text messages. This is hardly convenient for users trying to describe their issues or understand the new concepts and terminology needed for the troubleshooting.

Luckily, technology has reached a point where we can solve this issue by using Video Chat and Augmented Reality. In this guide, I will provide a step-by-step guide to show you how to build an Android app that uses ARCore and video chat to create an interactive experience.

Prerequisites

A basic to intermediate understanding of Java and Android SDK
Basic understanding of ARCore and Augmented Reality concepts
Agora.io Developer Account
Hardware: 2 Android devices running Android API level 24 or higher and originally shipped with Google Play Store

*You can check detailed device requirements here.

Please Note: While no Java/Android knowledge is needed to follow along, certain basic concepts in Java/ARCore won’t be explained along the way.

Overview

In this guide, we are going to build a AR-enabled customer support app. One user will create a channel by inputting a channel name, joining the channel as a streamer (people who need support). While the other user can join the same channel by entering the same channel name as an audience member (customer support). When both users are in the channels, the streamer will broadcast their rear camera to the audience. The audience can draw on their own device and have the touch input rendered in augmented reality in the streamer’s world!

Here are all the steps that we’ll be going through in this article:

Set Up New Project
Create UI
Enable ARCore
Enable Streamer’s Video Call
Enable Audience’s Video Call
Remote Assistance Feature
Build And Test on Device

You can find my Github demo app as a reference for this article.

Set Up New Project

To start, let’s open Android studio and create a new, blank project.

Open Android Studio and click Start a new Android Studio project.
On the Choose your project panel, choose Phone and Tablet > Empty Activity, and click Next.
Click Finish. Follow the on-screen instructions if you need to install any plug-ins.

Integrate the SDK

Add the following line in the /app/build.gradle file of your project:

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    implementation 'androidx.appcompat:appcompat:1.1.0'
    implementation 'androidx.constraintlayout:constraintlayout:1.1.3'
    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'androidx.test.ext:junit:1.1.1'
    androidTestImplementation 'androidx.test.espresso:espresso-core:3.2.0'

    //ARCore 
    implementation 'com.google.ar:core:1.0.0'
    implementation 'de.javagl:obj:0.2.1'
    implementation 'com.google.android.material:material:1.1.0'
    implementation 'com.android.support:appcompat-v7:27.0.2'
    implementation 'com.android.support:design:27.0.2'

    //Video 
    implementation 'io.agora.rtc:full-sdk:2.9.4'
}

Sync the project after the changes. Add the following project permissions in /app/src/main/AndroidManifest.xml file:

    <uses-permission android:name="android.permission.CAMERA" />
    <uses-permission android:name="android.permission.INTERNET" />
    <uses-permission android:name="android.permission.RECORD_AUDIO" />
    <uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />
    <uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
    <uses-permission android:name="android.permission.BLUETOOTH" />

In order to run ARCore, we also need to add the following in the AndroidManifest.xml file. This indicates that this application requires ARCore.

<uses-feature
    android:name="android.hardware.camera.ar"
    android:required="true" />
<application
...
<meta-data
    android:name="com.google.ar.core"
    android:value="required" />
</application>

Create UI

Let’s create the UI component for both users. For the user who is sharing their world with AR, we are going to refer to them as the “streamer”.For the other user who joins the channel for support, we will refer to them as the “audience”. Below is an example UI that I will be using:

Please Note: You can find the .xml files for the UI here.

The main difference between the streamer UI and audience UI is the streamer UI is using android.opengl.GLSurfaceView to render the AR camera view whereas the audience’s UI is using a RelativeLayout to render the video coming from the streamer.

The container on the top right corner of the streamer’s UI page is for rendering the remote video coming from the audience local camera. The container on the top right corner of the audience’s UI page is for rendering the audience’s local camera view.

We will also create a screen for the user to enter the channel name and join the channel as a streamer or audience.

Please Note: You can find the .xml files for the Join Channel UI here.

The logic under the Channel UI is very simple. If the user clicks the CREATE button, it will jump to the streamer’s activity page which handles streaming logic. If the user clicks the JOIN button, it will jump to the audience's activity page which handles audience interaction logic. Now, we need to define the logic for both activities.

Set Up ARCore

Setting up the ARCore in the project is not as difficult as most people think. In the OnResume() method of the streamer activity, we need to create a Session instance. The Session instance is to manage the AR system state and handle the lifecycle. We can use it to receive frames that allow access to camera images. However, before that, we have to check if ARCore is installed.

@Override
protected void onResume() {
    super.onResume();

    if (mSession == null) {
        String message = null;
        try {
            switch (ArCoreApk.getInstance().requestInstall(this, !installRequested)) {
                case INSTALL_REQUESTED:
                    installRequested = true;
                    return;
                case INSTALLED:
                    break;
            }

            // ARCore requires camera permissions to operate. If we did not yet obtain runtime permission on Android M and above, now is a good time to ask the user for it.
            if (!CameraPermissionHelper.hasCameraPermission(this)) {
               CameraPermissionHelper.requestCameraPermission(this);
                return;
            }

            mSession = new Session(this);
        } catch (Exception e) {
            ...
        } 
        // Create default config and check if supported.
        Config config = new Config(mSession);
        if (!mSession.isSupported(config)) {
            showSnackbarMessage("This device does not support AR", true);
        }
        ...
}

Implements GLSurfaceView.Renderer

We will use the GLSurfaceView to render the AR camera. In order to do that, the streamer activity should implement GLSurfaceView.Renderer. There are three functions that need to be overridden: onSurfaceCreated, onSurfaceChanged and onDrawFrame.

Override onSurfaceCreated

In the method onSurfaceCreated, which is usually called at the beginning of the rendering, we will need to do some initializations for the AR scene.

Create a BackgroundRenderer instance and pass the id of that to the session camera.
Initialize the 3D object. This 3D object will be rendered in streamer’s AR world later.
Initialize the plane detection render.
Initialize the point cloud.

@Override
public void onSurfaceCreated(GL10 gl, EGLConfig config) {
    GLES20.glClearColor(0.1f,0.1f,0.1f,1.0f);

    // Create the texture and pass it to ARCore session to be filled during update().
    mBackgroundRenderer.createOnGlThread(/*context=*/ this);
    if (mSession != null) {
  mSession.setCameraTextureName(mBackgroundRenderer.getTextureId());
    }
    // Prepare the other rendering objects.
    try {
        mVirtualObject.createOnGlThread(/*context=*/this, "andy.obj", "andy.png");
        mVirtualObject.setMaterialProperties(0.0f, 3.5f, 1.0f, 6.0f);
        mVirtualObjectShadow.createOnGlThread(/*context=*/this,
                "andy_shadow.obj", "andy_shadow.png");
        mVirtualObjectShadow.setBlendMode(ObjectRenderer.BlendMode.Shadow);
        mVirtualObjectShadow.setMaterialProperties(1.0f, 0.0f, 0.0f, 1.0f);
    } catch (IOException e) {
        ...
    }
    try {
        mPlaneRenderer.createOnGlThread(/*context=*/this, "trigrid.png");
    } catch (IOException e) {
        ...
    }
    mPointCloud.createOnGlThread(/*context=*/this);
}

Override onSurfaceChanged

In the onSurfaceChanged method, which is called after the surface is created and when the surface size changes, we will set our viewport.

@Override
public void onSurfaceChanged(GL10 gl, int width, int height) {
    ...
    GLES20.glViewport(0, 0, width, height);
}

Override onDrawFrame

In the method onDrawFrame, which is called to draw the current frame, we need to implement the render logic for the GLSurfaceView.

Clear screen.
Get the latest frame from the ARSession.
Capture user’s taps and check if any planes in the scene were found. If so, create an anchor at that point.
Draw the background.
Draw the point cloud.
Iterate all the anchors and draw the 3D object on each of the anchors.

@Override
public void onDrawFrame(GL10 gl) {
    // Clear screen to notify driver it should not load any pixels from previous frame.
    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_DEPTH_BUFFER_BIT);
    ...

    try {
        // Obtain the current frame from ARSession. When the configuration is set to
        // UpdateMode.BLOCKING (it is by default), this will throttle the rendering to the camera framerate.
        Frame frame = mSession.update();
        Camera camera = frame.getCamera();

        // Handle taps. Handling only one tap per frame, as taps are usually low frequency
        // compared to frame rate.
        MotionEvent tap = queuedSingleTaps.poll();
        if (tap != null && camera.getTrackingState() == TrackingState.TRACKING) {
            for (HitResult hit : frame.hitTest(tap)) {
                // Check if any plane was hit, and if it was hit inside the plane polygon
                Trackable trackable = hit.getTrackable();
                // Creates an anchor if a plane or an oriented point was hit.
                if ((trackable instanceof Plane && ((Plane) trackable).isPoseInPolygon(hit.getHitPose()))
                        || (trackable instanceof Point
                        && ((Point) trackable).getOrientationMode()
                        == Point.OrientationMode.ESTIMATED_SURFACE_NORMAL)) {
                    // Hits are sorted by depth. Consider only closest hit on a plane or oriented point.
                    // Cap the number of objects created. This avoids overloading both the
                    // rendering system and ARCore.
                    if (anchors.size() >= 250) {
                        anchors.get(0).detach();
                        anchors.remove(0);
                    }
                    // Adding an Anchor tells ARCore that it should track this position in
                    // space. This anchor is created on the Plane to place the 3D model
                    // in the correct position relative both to the world and to the plane.
                    anchors.add(hit.createAnchor());
                    break;
                }
            }
        }

        // Draw background.
        mBackgroundRenderer.draw(frame);

        ...
        if (isShowPointCloud()) {
            // Visualize tracked points.
            PointCloud pointCloud = frame.acquirePointCloud();
            mPointCloud.update(pointCloud);
            mPointCloud.draw(viewmtx, projmtx);

            // Application is responsible for releasing the point cloud resources after
            // using it.
            pointCloud.release();
        }

        ...

        if (isShowPlane()) {
            // Visualize planes.
            mPlaneRenderer.drawPlanes(
                    mSession.getAllTrackables(Plane.class), camera.getDisplayOrientedPose(), projmtx);
        }

        // Visualize anchors created by touch.
        float scaleFactor = 1.0f;

        for (Anchor anchor : anchors) {
            if (anchor.getTrackingState() != TrackingState.TRACKING) {
                continue;
            }
            // Get the current pose of an Anchor in world space. The Anchor pose is updated
            // during calls to session.update() as ARCore refines its estimate of the world.
            anchor.getPose().toMatrix(mAnchorMatrix, 0);


            // Update and draw the model and its shadow.
            mVirtualObject.updateModelMatrix(mAnchorMatrix, mScaleFactor);
            mVirtualObjectShadow.updateModelMatrix(mAnchorMatrix, scaleFactor);
            mVirtualObject.draw(viewmtx, projmtx, lightIntensity);
            mVirtualObjectShadow.draw(viewmtx, projmtx, lightIntensity);
        }
    } catch (Throwable t) {
        ...
    }
}

Please note: Some of the concepts are not explained here. Check the Github code for more understanding.

Enable Streamer’s Video Call

Set Up Video Call from Streamer

In the onCreate() method in the AgoraARStreamerActivity, let’s do following things:

Initialize the GLSurfaceView onTouchListener
Initialize the Agora RtcEngine
Set up custom video source
Join channel

1. Initialize the GLSurfaceView onTouchListener

Setting the onTouchListener for the GLSurfaceView allows us to capture the touch position and set an AR anchor at that position.

mGestureDetector = new GestureDetector(this,
        new GestureDetector.SimpleOnGestureListener() {
            @Override
            public boolean onSingleTapUp(MotionEvent e) {
                onSingleTap(e);
                return true;
            }

            @Override
            public boolean onDown(MotionEvent e) {
                return true;
            }
        });

mSurfaceView.setOnTouchListener(new View.OnTouchListener() {
    @Override
    public boolean onTouch(View v, MotionEvent event) {
        return mGestureDetector.onTouchEvent(event);
    }
});

When the GestureDetector detects a single tap, it will trigger the onSingleTap method. In that method, we add this tap to a single tap queue.

private void onSingleTap(MotionEvent e) {
    queuedSingleTaps.offer(e);
}

Remember when we created anchors in onDrawFrame method? We polled the user’s taps from the queuedSingleTaps. This is where the taps are added to the queue.

2. Initialize the Agora RtcEngine

In order to initialize the Agora video engine, just simply call RtcEngine.create(context, appid, RtcEventHandler) to create a RtcEngine instance.

mRtcEngine = RtcEngine.create(this, getString(R.string.private_broadcasting_app_id), mRtcEventHandler);

In order to get the App ID in the parameter, follow these steps:

Create an Agora project in the Agora Console.
Click the Project Management tab on the left navigation panel.
Click “Create” and follow the on-screen instructions to set the project name, choose an authentication mechanism, and click “Submit”.
On the Project Management page, find the App ID of your project.

The mRtcEventHandler is a handler to manage different events occurring with the RtcEngine. Let’s implement it with some basic event handlers needed for this application.

private IRtcEngineEventHandler mRtcEventHandler = new IRtcEngineEventHandler() {
    @Override
    public void onJoinChannelSuccess(final String channel, int uid, int elapsed) {
        //when local user joined the channel
        ...
    }

    @Override
    public void onRemoteVideoStateChanged(final int uid, int state, int reason, int elapsed) {
        super.onRemoteVideoStateChanged(uid, state, reason, elapsed);
        //when remote user join the channel
        if (state == Constants.REMOTE_VIDEO_STATE_STARTING) {
            runOnUiThread(new Runnable() {
                @Override
                public void run() {
                    addRemoteRender(uid);
                }
            });
        }
    }

    @Override
    public void onUserOffline(int uid, int reason) {
        //when remote user leave the channel
        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                removeRemoteRender();
            }
        });
    }

    @Override
    public void onStreamMessage(int uid, int streamId, byte[] data) {
        //when received the remote user's stream message data
        ...
    }
};

Check the comments on the top of each event handler methods to have a better understanding of them. For more RtcEngine event handlers that you can use, check the Agora Rtc API document.

3. Set Up custom video source

In our application, we want to let the streamer send their AR world to the audience. So the video they send out is from a custom video source instead of a general camera video stream. Luckily, Agora Video SDK provides an API method to send custom video sources.

Create a class called AgoraVideoSource which implements IVideoSource interface. The IVideoSource interface defines a set of protocols to implement the custom video source and pass it to the underlying media engine to replace the default video source.

public class AgoraVideoSource implements IVideoSource {
    private IVideoFrameConsumer mConsumer;

    @Override
    public boolean onInitialize(IVideoFrameConsumer iVideoFrameConsumer) {
        mConsumer = iVideoFrameConsumer;
        return true;
    }

    @Override
    public boolean onStart() {
        return true;
    }

    @Override
    public void onStop() {
    }

    @Override
    public void onDispose() {
    }

    @Override
    public int getBufferType() {
        return MediaIO.BufferType.BYTE_ARRAY.intValue();
    }

    public IVideoFrameConsumer getConsumer() {
        return mConsumer;
    }
}

In the onCreate method within the AgoraARStreamerActivity, call the constructor method to create the AgoraVideoSource instance.

mSource = new AgoraVideoSource();

Create a class called AgoraVideoRender which implements IVideoSink interface. The IVideoSink interface defines a set of protocols to create a customized video sink and pass it to the media engine to replace the default video renderer.

public class AgoraVideoRender implements IVideoSink {
    private Peer mPeer;
    private boolean mIsLocal;

    public AgoraVideoRender(int uid, boolean local) {
        mPeer = new Peer();
        mPeer.uid = uid;
        mIsLocal = local;
    }

    public Peer getPeer() {
        return mPeer;
    }

    @Override
    public boolean onInitialize() {
        return true;
    }

    @Override
    public boolean onStart() {
        return true;
    }

    @Override
    public void onStop() {

    }

    @Override
    public void onDispose() {

    }

    @Override
    public long getEGLContextHandle() {
        return 0;
    }

    @Override
    public int getBufferType() {
        return MediaIO.BufferType.BYTE_BUFFER.intValue();
    }

    @Override
    public int getPixelFormat() {
        return MediaIO.PixelFormat.RGBA.intValue();
    }

    @Override
    public void consumeByteBufferFrame(ByteBuffer buffer, int format, int width, int height, int rotation, long ts) {
        if (!mIsLocal) {
            mPeer.data = buffer;
            mPeer.width = width;
            mPeer.height = height;
            mPeer.rotation = rotation;
            mPeer.ts = ts;
        }
    }

    @Override
    public void consumeByteArrayFrame(byte[] data, int format, int width, int height, int rotation, long ts) {

    }

    @Override
    public void consumeTextureFrame(int texId, int format, int width, int height, int rotation, long ts, float[] matrix) {

    }

}

Similar to the AgoraVideoSource instance, we create the AgoraVideoRender instance by calling its constructor. Here we pass uid as 0 to represent a local video render.

mRender = new AgoraVideoRender(0, true);

After creating two instances, we call

mRtcEngine.setVideoSource(mSource);
mRtcEngine.setLocalVideoRenderer(mRender);

to set the custom AR video source and local video renderer.

However, the video source we set does not have data in it. We need to pass the AR camera view to our video source. To do that, we are going to add logic at the end of the onDrawFrame method that we override before.

@Override
public void onDrawFrame(GL10 gl) {
    ...
    final Bitmap outBitmap = Bitmap.createBitmap(mSurfaceView.getWidth(), mSurfaceView.getHeight(), Bitmap.Config.ARGB_8888);
    PixelCopy.request(mSurfaceView, outBitmap, new PixelCopy.OnPixelCopyFinishedListener() {
        @Override
        public void onPixelCopyFinished(int copyResult) {
            if (copyResult == PixelCopy.SUCCESS) {
                sendARView(outBitmap);
            } else {
                Toast.makeText(AgoraARCoreActivity.this, "Pixel Copy Failed", Toast.LENGTH_SHORT);
            }
        }
    }, mSenderHandler);
}
private void sendARView(Bitmap bitmap) {
    if (bitmap == null) return;

    if (mSource.getConsumer() == null) return;

    //Bitmap bitmap = source.copy(Bitmap.Config.ARGB_8888,true);
    int width = bitmap.getWidth();
    int height = bitmap.getHeight();

    int size = bitmap.getRowBytes() * bitmap.getHeight();
    ByteBuffer byteBuffer = ByteBuffer.allocate(size);
    bitmap.copyPixelsToBuffer(byteBuffer);
    byte[] data = byteBuffer.array();

    mSource.getConsumer().consumeByteArrayFrame(data, MediaIO.PixelFormat.RGBA.intValue(), width, height, 0, System.currentTimeMillis());
}

The logic here is to copy the GLSurfaceView to a bitmap and send the bitmap buffer to our custom video source.

4. Join channel

Now, we are ready to join the channel by calling joinChannel() on the RtcEngine instance by passing the channelName from the previous activity.

mRtcEngine.joinChannel(null, channelName, "", 0);

Please Note: The token in the parameter can be set to null.

By calling this function and successfully joining the channel, the RtcEngineEventHandler will trigger the onJoinChannelSuccess() method that we implemented in the previous step. It will return a unique Agora video id which is generated by the Agora server.

Till this point, the streamer can join the channel and broadcast their AR world to the audience.

Enable Audience’s Video Call

Enabling the audience’s video call is very similar to the one we have written for the streamer. The only difference is to set up local camera video after initializing RtcEngine.

mRtcEngine.enableVideo();
mLocalView = RtcEngine.CreateRendererView(getBaseContext());
mLocalContainer.addView(mLocalView);
mLocalView.setZOrderMediaOverlay(true);

VideoCanvas localVideoCanvas = new VideoCanvas(mLocalView, VideoCanvas.RENDER_MODE_HIDDEN, 0);
mRtcEngine.setupLocalVideo(localVideoCanvas);

This will put the audience’s local camera view on the top right corner of the screen.

Please Note: Find Github code for more understanding.

We are now able to start a video call between streamer and audience. However, this is still not a feature-complete remote assistance app because the audience cannot interact with streamer’s AR world. We’ll start implementing the audience markup feature through ARCore next.

Remote Assistance Feature

Ideally, the remote technician (audience) should be able to provide assistance when they want to direct the customer (streamer) by drawing on the screen. This markup should be rendered on the customer’s side instantly and should remain in the same position as drawn.

In order to achieve that, we are going to collect the audience member’s touch positions and send them to the streamer. As long as the streamer receives those touch points, we can simulate the touches on the streamer's screen to create AR objects.

Let’s first collect the audience’s touch positions. In AgoraARAudienceActivity’s onCreate method, set up a touch listener on the remote view container. Collect all the touch point positions in relation to the center of the screen. Send them out to the streamer as a data stream message using the Agora API method, sendStreamMessage. This will trigger the streamer’s onStreamMessage callback. Since a user can only send 6kB of data per second, we send out the touch points whenever we collect 10 of them.

mRemoteContainer.setOnTouchListener(new View.OnTouchListener() {
    @Override
    public boolean onTouch(View v, MotionEvent event) {
        switch (event.getAction()) {
            case MotionEvent.ACTION_MOVE:
                //get the touch position related to the center of the screen
                touchCount++;
                float x = event.getRawX() - ((float)mWidth / 2);
                float y = event.getRawY() - ((float)mHeight / 2);
                floatList.add(x);
                floatList.add(y);
                if (touchCount == 10) {
                    //send the touch positions when collected 10 touch points
                    sendMessage(touchCount, floatList);
                    touchCount = 0;
                    floatList.clear();
                }
                break;
            case MotionEvent.ACTION_UP:
                //send touch positions after the touch motion
                sendMessage(touchCount, floatList);
                touchCount = 0;
                floatList.clear();
                break;
        }
        return true;
    }
});

Here is the sendMessage logic:

/**
 * send the touch points as a byte array to Agora sdk
 * @param touchCount
 * @param floatList
 */
private void sendMessage(int touchCount, List<Float> floatList) {
    byte[] motionByteArray = new byte[touchCount * 4 * 2];
    for (int i = 0; i < floatList.size(); i++) {
        byte[] curr = ByteBuffer.allocate(4).putFloat(floatList.get(i)).array();
        for (int j = 0; j < 4; j++) {
            motionByteArray[i * 4 + j] = curr[j];
        }
    }
    mRtcEngine.sendStreamMessage(dataChannel, motionByteArray);
}

Now, in AgoraARStreamerActivity, we need to override the onStreamMessage callback to receive touch points sent by the audience and simulate the touches on the streamer’s screen.

@Override
public void onStreamMessage(int uid, int streamId, byte[] data) {
    //when received the remote user's stream message data
    super.onStreamMessage(uid, streamId, data);
    int touchCount = data.length / 8;       //number of touch points from data array
    for (int k = 0; k < touchCount; k++) {
        //get the touch point's x,y position related to the center of the screen and calculated the raw position
        byte[] xByte = new byte[4];
        byte[] yByte = new byte[4];
        for (int i = 0; i < 4; i++) {
            xByte[i] = data[i + 8 * k];
            yByte[i] = data[i + 8 * k + 4];
        }
        float convertedX = ByteBuffer.wrap(xByte).getFloat();
        float convertedY = ByteBuffer.wrap(yByte).getFloat();
        float center_X = convertedX + ((float) mWidth / 2);
        float center_Y = convertedY + ((float) mHeight / 2);

        //simulate the clicks based on the touch position got from the data array
        instrumentation.sendPointerSync(MotionEvent.obtain(SystemClock.uptimeMillis(), SystemClock.uptimeMillis(), MotionEvent.ACTION_DOWN, center_X, center_Y, 0));
        instrumentation.sendPointerSync(MotionEvent.obtain(SystemClock.uptimeMillis(), SystemClock.uptimeMillis(), MotionEvent.ACTION_UP, center_X, center_Y, 0));
    }
}

Build And Test on Device

That’s all you need to build a remote assistance app. Now let’s run our application!

Go to Android Studio, make sure your Android device is plugged in, and click Run to build the application on your device. Remember to build the application on two devices to start the video call. Both devices must run Android API level 24 or higher and originally ship with Google Play Store.

You can check detailed device requirements here.

Done!

Congratulations! You just build yourself a remote assistance application with augmented reality features!

Thank you for following along. Please leave a comment below! Here is the email address for any of the questions you might have: devrel@agora.io.