Live Streaming to Multiple Platforms with Multiple Users

There are some pretty good solutions to have collaborative livestreams with other creators. But if you are like me, you probably think they are a little too expensive, and you probably wish there was more customizability. Thankfully, Agora exists and you can build your own services like this!

Prerequisites

• An Agora Developer Account - Sign Up Here
• Flutter SDK
• Agora RTC package
• Agora RTM package

Since this is such a big project the plan is to break it down and cover the most important concepts. If you want the full code, you can find it here:
https://github.com/tadaspetra/flutter-projects/tree/main/streamer

App Overview

Before getting into those 3 main points, I believe a general overview of what we are trying to accomplish and how we are going to do it would be beneficial. The goal of this application is simple: allow multiple users to join and stream their video to a streaming platform. (Using a host application that manage the users and what gets actually pushed to the streaming platform)

The flow of the application is made to be as simple as possible. You start off in the main screen where you need to enter which channel you are trying to join and your name. Then you have two options. You can join as a Director of the call, or a Participant within the call.

Director - Manages all the users and where and how the stream gets sent out.

Participant - Joins the call and has a simple call interface.

In this article we won't be covering the basics of Agora and how it works. If you aren't familiar with Agora here are a couple good places to start, then you can come back to this one.

Video Call: https://youtu.be/zVqs1EIpVxs

Livestreaming: https://youtu.be/kE0ehPMGgVc

Participant View

This article won't spend much time on the participant view. The logic here should be very similar to any other type of video call, except for 4 big differences.

1. When the user first joins they are in a lobby, so their camera and audio are off.
2. When the host brings them into the stage, their video and audio should turn on.
3. Video and Audio can be controlled by the host.
4. Instead of seeing everybody in the call, they will only see the people on stage.

To the person using the device, they shouldn't see much of a difference from any regular video call. The only extra thing is the "lobby screen" that they will have, before actually joining the call. If there are people on the stage, the current participant will be able to see them, even if they are not on the stage yet. They will be able to watch the output call that gets sent to the streaming platform, but without the custom transcoding. Now all these complex operations are done using Agora's RTM Messages

RTM Messaging

Since this is no simple video call application, it will require some way for the director to control the participants. For this we will use the agora_rtm package. This package allows you to send real time data between the everybody in the channel. The agora_rtc_engine package is actually built on top of agora_rtm. The difference is that RTM allows you to send any data, while RTC makes it easy to send video and audio data. For this application, only the director will be able to send out RTM messages, and the participants are only going to be receiving them. There are three types of functions we need to allow the director to have:

1. Mute or unmute audio.
2. Enable or disable video.
3. Send out list of active users.

To mute the user the director sends out a channel wide RTM message with the following format "mute uid", except instead of the word "uid" they send the specific uid of the user to be muted. Upon receiving this message the participant checks if this uid is their uid. If it is, then the user mutes themselves. This works the same way with unmuting and disabling and enabling video except using the keywords "unmute uid", "enable uid", and "disable uid".

The slightly trickier part is the active users. Normally if you're using agora, you would display all the broadcasters within that call. But in this case, some of them are in the lobby, and they should not be displayed to the viewers. To handle this we use RTM messages again to send all the users that should be displayed. The format is "activeUsers uid,uid,uid" except replace the word "uid" with the specific uid of the active users.

---

So far we have covered almost everything from the participant point of view. Now let's transition to the Director, which is where most of the magic happens.

---

Director Controller

The director in this app has a lot of functions and things to keep track of. To keep things organized we will use riverpod, a popular state management solution for Flutter.

If you have never used it, here is a good place to start: https://www.youtube.com/watch?v=8qzip8tVmqU

The DirectorController that we have defined here will be a StateNotifierProvider. This helps separate our business logic from the UI section of our application.

final directorController = StateNotifierProvider.autoDispose<DirectorController, DirectorModel>((ref) {
  return DirectorController(ref.read);
});

This controller will have functions that the rest of our code will be able to access, including joinCall(), leaveCall(), toggleUserAudio(), addUserToLobby(), promoteToActiveUser(), startStream(), and lots of other ones. This controller will also store all the data that we need to keep track of within our app.

As the participants are only receiving the RTM messages, the director will only be sending out RTM messages.

In order for the director to be able to send out RTM messages, you need to set up a client and a channel using RTM. This is very similar to what happens with RTC Engine behind the scenes. You need to create a client, log into the client, then create a channel and join the channel. Once this is done, you are ready to send out the RTM messages. The participants will need to do the same thing in order to receive messages on the onMessageReceived callback.

To actually send out the message you need to use the sendMessage function that is provided on that channel. To format the message correctly use this:

AgoraRtmMessage.fromText("unmute ${state.activeUsers.elementAt(index).uid}")

Use the same approach for all the other messages like "mute uid", "enable uid", "disable uid", and "activeUsers uid,uid,uid".

So those are the infrastructure details, of what enables us to actually be able to be manage users and streams. Let's get into the details of how the director part of this app actually works. The main three features that we are going to cover are:

• Muting and Disabling Video of other users
• Moving users between main stage and lobby
• Transcoding Each Video and pushing it to Streaming Platforms

Muting and Disabling Video

Now given that we have the infrastructure with RTM messaging all set up, this section might sound trivial, but there are actually a lot of pieces for this that need to be accounted for and synced up.

Director App

• Muting/Unmuting user's
• Disabling/Enabling user's video
• Current states of audio and video for each user
• Update if user changes their own state

Participant App

• Mute/Unmute yourself
• Disable/Enable own video
• Mute/Unmute from director
• Disable/Enable video from director
• Current State of audio and video

To do all this and have it synced up there are lots of different parts that control the audio. The best way to go about this is to look at the different scenarios.

• Participant mutes/unmutes themselves When the participant decides to mute themselves they need to call muteLocalAudioStream(), then update their own button state to show that they are muted, and then on the director side remoteAudioStateChanged event should get triggered, which in should update the current state for that specific user.
• Participant disables/enables video Same process as above, except call the function muteLocalVideoStream() and the event on the director side should be remoteVideoStateChanged.
• Director mutes/unmutes user Director needs to send an RTM message with either "mute uid" or "unmute uid". Then the user with the matching uid will follow the same execution as if they were muting themselves, and then again the director should see the remoteAudioStateChanged event trigger, and they can update the local state.
• Director disables/enables video Same process as muting, but instead the stream message will be "enable uid" or "disable uid"

Now let's add another layer of complexity to this.

Stage and Lobby

The idea here isn't too complex, but it comes with a couple caveats that need to be taken care of. The only person that will be able to see both lobby and stage is the director. The DirectorController will hold a separate list of active users and lobby users. A normal flow for a participant user would be to join the channel, and they will be directly added to the lobby. Then the director is in complete control and can move them to and from the stage as they please.

The flow for moving a person to and from the stage is very similar. First remove them from the previous list (lobby or active) and add them to the other list. Then update send out the new list of activeUsers to everybody using a RTM message

Not too bad, but here is where the complexity comes in. You don't want the lobby users to be able to talk over the users in the stage so they should be muted whilst in the lobby. And since they are in the lobby there is no need to take up extra bandwidth for their video either. Because of this we need to add a couple more scenarios for audio and video control.

• Participant first joins channel Since they are added directly into the lobby, they need to be muted and video disabled immediately. Whenever a participant joins a channel, they automatically mute themselves.
• Participant moved to Stage When they are moved to the main stage their video and audio need to be enabled, so that the audience can see them. This should follow the same logic as director unmuting or enabling video.
• Participant moved to Lobby When they are moved to the lobby their video and audio need to be enabled, so that the audience can see them. This should follow the same logic as director muting or disabling video.

Transcoding

Once you have the activeUser list synced up with both the director section of the app, and the participant section of the app, the last step left is to broadcast it out to the streaming platforms. For this we will first transcode all the incoming videos to the desired layout, and then publish and unpublish our streams to the desired platforms using Real-Time Messaging Protocol (RTMP).

First we need to define what layout we want to have for our output video. In this case we will only support up to 8 people in a call. But you can extend the same concept to as many callers as you want. We also take into account that our stream will be a 1080p streams so have 1920x1080 pixels to work with. Give that information the layout will look like this:

To actually send out this information we need to create a list of TranscodingUser and set up each of the users layouts accordingly. Once they are in the list we create a LiveTranscoding object with the list, and tell the RTCEngine that this is what we want it to look like.

List<TranscodingUser> transcodingUsers = [];
if (state.activeUsers.isEmpty) {
} else if (state.activeUsers.length == 1) {
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(0).uid, 0, 0, width: 1920, height: 1080, zOrder: 1, alpha: 1));
} else if (state.activeUsers.length == 2) {
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(0).uid, 0, 0, width: 960, height: 1080));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(1).uid, 960, 0, width: 960, height: 1080));
} else if (state.activeUsers.length == 3) {
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(0).uid, 0, 0, width: 640, height: 1080));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(1).uid, 640, 0, width: 640, height: 1080));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(2).uid, 1280, 0, width: 640, height: 1080));
} else if (state.activeUsers.length == 4) {
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(0).uid, 0, 0, width: 960, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(1).uid, 960, 0, width: 960, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(2).uid, 0, 540, width: 960, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(3).uid, 960, 540, width: 960, height: 540));
} else if (state.activeUsers.length == 5) {
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(0).uid, 0, 0, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(1).uid, 640, 0, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(2).uid, 1280, 0, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(3).uid, 0, 540, width: 960, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(4).uid, 960, 540, width: 960, height: 540));
} else if (state.activeUsers.length == 6) {
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(0).uid, 0, 0, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(1).uid, 640, 0, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(2).uid, 1280, 0, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(3).uid, 0, 540, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(4).uid, 640, 540, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(5).uid, 1280, 540, width: 640, height: 540));
} else if (state.activeUsers.length == 7) {
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(0).uid, 0, 0, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(1).uid, 480, 0, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(2).uid, 960, 0, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(3).uid, 1440, 0, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(4).uid, 0, 540, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(5).uid, 640, 540, width: 640, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(6).uid, 1280, 540, width: 640, height: 540));
} else if (state.activeUsers.length == 8) {
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(0).uid, 0, 0, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(1).uid, 480, 0, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(2).uid, 960, 0, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(3).uid, 1440, 0, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(4).uid, 0, 540, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(5).uid, 480, 540, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(6).uid, 960, 540, width: 480, height: 540));
  transcodingUsers.add(TranscodingUser(state.activeUsers.elementAt(7).uid, 1440, 540, width: 480, height: 540));
} else {
  throw ("too many members");
}

LiveTranscoding transcoding = LiveTranscoding(
  transcodingUsers,
  width: 1920,
  height: 1080,
);
state.engine!.setLiveTranscoding(transcoding);

Now we have the layout for our stream configured, we need to actually send it out. With this app we have the capabilities to send it out to multiple locations. For this you will need a URL where your stream should be pushed out to. For YouTube it is pretty straight forward. You will need the Stream Url + a backslash ("/") + the Stream Key which are all give in your livestreaming dashboard. For Twitch it is a similar concept, and you can read about it here: https://help.twitch.tv/s/article/guide-to-broadcast-health-and-using-twitch-inspector?language=en_US

Now that you have all your links, you can loop through all of them and call the addPublishUrl() on the RTCEngine with the transcodingEnabled parameter set to true. And it's done! Your stream should have appeared on the platforms.

Lastly, you will want to update the transcoding when someone is added or removed from the main stage, and end the stream. To update, you need to update the transcoding layout accordingly, and then setLiveTranscoding() again. And to remove a stream call the removePublishUrl().

Conclusion

If this app seems a bit complex, that's because it is. But there are full-blown companies that take months to build an MVP (Minimum Viable Product) for something like this. And even if they can build it, they don't have anywhere close to the infrastructure and reliability that the SD-RTN brings. This is a very complex application, but with Agora it becomes achievable.

You can find the code for this app here

Other Resources

To learn more about the Agora Flutter SDK and other use cases, see the developer guide here.

You can also have a look at the complete documentation for the functions discussed above and many more here.

And I invite you to join the Agora.io Developer Slack Community.