DEV Community: 100ms Inc.

Engage Your Audience with Interactive Polls and Quizzes - A Step-by-Step Guide

John Selvinraj — Mon, 16 Oct 2023 05:41:28 +0000

One of the most requested features from our customers was the ability to create polls and quizzes. We are excited to announce that this feature is now available!

Polls are a great way to get feedback from your audience, learn more about their interests, and engage them in a two-way conversation. Want to get feedback on the session? Create a poll. Let your audience rate the session as good or bad.

The questions can be made single-choice or multiple-choice. The feedback example discussed above would be a single-choice question. Whereas if you want to conduct market research on say what social media platform everyone in the room uses. That would require a multiple-choice question.

Needless to say, polls can enhance audience engagement and add interactivity to virtual meetings, conferences, or live streams. They also enable you to collect well-organized data by posing direct questions.

Create polls in 100ms rooms in the following way:

Once a poll is created and launched by a peer, other peers with permission to view the polls can interact with it.

Moreover, the Polls can be extended to create quizzes as well.

To delve a bit deeper, quizzes operate on a similar principle to polls, albeit with a twist.

In the world of quizzes, questions come with designated correct and incorrect answers, adding an element of assessment and knowledge evaluation to the interactive experience. It can be used to craft educational assessments, test comprehension, and promote active learning through engagement.

Let’s get into the mix of building things using 100ms Polls and Quizzes.

How to enable polls and quizzes for your rooms on the 100ms dashboard?

Access the 100ms Dashboard: Begin by visiting the 100ms dashboard.
Choose or Create a Template: Select the template where you want to enable polls and quizzes. Alternatively, create a new template by clicking the 'Create Template' button.
Access Template Configuration: Click on the 'Configure' button within your chosen template.
Assign Poll Creation Privileges: Select which role(s) should have the ability to create polls.
Set Permissions: In the 'Permissions' section, activate the following toggles: ◦ 'Create polls and quizzes' ◦ 'Read polls and quizzes'
Exclusive Interaction: If certain role(s) should only participate in polls without creating them, enable only the 'Read polls and quizzes' toggle for those roles.

How to add Polls to React sample app?

Let’s add polls to a React sample app to see how easy it is.

Start by creating a new React project (I’ll use Vite to create one) by running the following command:

npm create vite@latest my-polls-app -- --template react

This would create a new project folder my-polls-app in your desired directory with React configured in it.

Now, to quickly add the 100ms React SDK and enable live audio-video in our app, we will follow the React Quickstart Guide here.

Once done, we should have a React project with audio-video calling using 100ms set up.

The complete React Quickstart Guide code can be found here.

Create a new folder in the src directory called Components and add two subfolders Poll and UI to it. These folders would have the code for the necessary UI and Poll components.

We start by creating a Modal.jsx file inside the UI folder. This file would have the following code:

import { Fragment } from "react";
import ReactDOM from "react-dom";
import classes from "./Modal.module.css";

const Backdrop = (props) => {
  return <div className={classes.backdrop} onClick={props.onClose}></div>;
};

const ModalOverlay = (props) => {
  return (
    <div className={classes.modal}>
      <div className={classes.content}>{props.children}</div>
    </div>
  );
};

const portalElement = document.getElementById("overlays");

const Modal = (props) => {
  return (
    <Fragment>
      {ReactDOM.createPortal(
        <Backdrop onClose={props.onClose} />,
        portalElement
      )}
      {ReactDOM.createPortal(
        <ModalOverlay>{props.children}</ModalOverlay>,
        portalElement
      )}
    </Fragment>
  );
};
export default Modal;

The Modal component is made up of the Backdrop and ModalOverlay . The ModalOverlay would provide a white background to lay the children on passed to it using props.

You’d notice we have imported a css file but haven’t defined it yet. So, inside the same UI folder create a new file called Modal.module.css and add the following to it:

.backdrop {
  position: fixed;
  top: 0;
  left: 0;
  width: 100%;
  height: 100vh;
  z-index: 20;
  background-color: rgba(0, 0, 0, 0.75);
}

.modal {
  position: fixed;
  top: 20vh;
  left: 5%;
  width: 90%;
  background-color: #546e7a;
  padding: 1rem;
  border-radius: 14px;
  box-shadow: 0 2px 8px rgba(0, 0, 0, 0.25);
  z-index: 30;
  animation: slide-down 300ms ease-out forwards;
}

@media (min-width: 768px) {
  .modal {
    width: 40rem;
    left: calc(50% - 20rem);
  }
}

@keyframes slide-down {
  from {
    opacity: 0;
    transform: translateY(-3rem);
  }
  to {
    opacity: 1;
    transform: translateY(0);
  }
}

Lastly, to complete our Modal, we also need to modify the index.html in the root of the project directory to add another div with id="overlays" before the root div as below:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" type="image/svg+xml" href="/vite.svg" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>100ms Polls Demo</title>
  </head>
  <body>
    <div id="overlays"></div>
    <div id="root"></div>
    <script type="module" src="/src/main.jsx"></script>
  </body>
</html>

Now, we will work on a form to create a poll. Start by creating a form as follows:

import Modal from ".././UI/Modal";
import { Button } from "@100mslive/roomkit-react";
import { useState } from "react";
import { useHMSActions } from "@100mslive/react-sdk";
import "../../styles.css";

const PollForm = (props) => {
    const hmsActions = useHMSActions();
  const [inputs, setInputs] = useState({});

  return (
    <Modal onClose={props.onClose}>
      <form onSubmit={handleSubmit}>
        <label>
          Enter a Name for the Poll:
          <div className="input-container">
            <input
              type="text"
              name="name"
              value={inputs.name || ""}
              onChange={handleChange}
            />
          </div>
        </label>
        <label>
          Enter a question for the Poll:
          <div className="input-container">
            <input
              type="text"
              name="text"
              value={inputs.text || ""}
              onChange={handleChange}
            />
          </div>
        </label>
        <label>
          Enter the First Value:
          <div className="input-container">
            <input
              type="text"
              name="first"
              value={inputs.first || ""}
              onChange={handleChange}
            />
          </div>
        </label>
        <label>
          Enter the Second Value:
          <div className="input-container">
            <input
              type="text"
              name="second"
              value={inputs.second || ""}
              onChange={handleChange}
            />
          </div>
        </label>
        <input type="submit" />
      </form>
    </Modal>
  );
};
export default PollForm;

You’ll notice we are having the user input a name for the poll, a question and two options to choose from. To manage changes in values and successfully submit the form, we add the following functions.

const handleChange = (event) => {
    const name = event.target.name;
    const value = event.target.value;
    setInputs((values) => ({ ...values, [name]: value }));
  };

  const handleSubmit = async (event) => {
    event.preventDefault();
    const id = Date.now().toString();
    await hmsActions.interactivityCenter
      .createPoll({
        id,
        title: inputs.name,
        type: "poll",
        rolesThatCanViewResponses: ["host"],
      })
      .then(() => handleCreate(id))
      .catch((err) => console.log(err.message));
  };

const handleCreate = async (id) => {
    console.log("POLL CREATED with ${id}");
    await hmsActions.interactivityCenter.addQuestionsToPoll(id, [
      {
        text: inputs.text,
        type: "single-choice",
        options: [
          {
            text: inputs.first,
            isCorrectAnswer: false,
          },
          {
            text: inputs.second,
            isCorrectAnswer: false,
          },
        ],
        skippable: true,
      },
    ]);
    await hmsActions.interactivityCenter.startPoll(id);
  };

The handleChange takes any changes in the text input fields and sets the inputs variable accordingly. On form submission, the handleSubmit is called which uses the useHMSActions hook to create a new poll.

Once a poll is created, handleCreate is run to addQuestionsToPoll and also startPoll using the same hmsActions.

The complete PollForm.jsx file would then look as this.

Next, we return to our App.jsx file to display the PollForm when the user clicks on a Create Poll button.

Start by importing the following:

import { useState } from "react";
import { Button } from "@100mslive/roomkit-react";

We will use useState to manage the state of the PollForm if we need to show or hide it.

const [pollFormIsShown, setPollFormIsShownn] = useState(false);

const showPollFormHandler = () => {
    setPollFormIsShownn(true);
  };

const hidePollFormHandler = () => {
    setPollFormIsShownn(false);
    };

Add the PollForm component and a button with showPollFormHandler being passed to it as below:

{pollFormIsShown && <PollForm onClose={hidePollFormHandler} />}
<Button onClick={showPollFormHandler}>Create Poll</Button>

Once this is done, we should be able to toggle and view the PollForm and dismiss it by clicking the backdrop area. We should also be able to create and start a poll with the name, question and choices entered by the user.

Let us now work to show the Poll to any other users using our app.

We can use the useHMSNotifications hook to be notified when a new poll is started. We will use this to show a toast using react-toastify when a new poll has begun.

Add the package by running: npm install --save react-toastify

Add the following imports to the App.jsx file.

import {
  selectIsConnectedToRoom,
  selectLocalPeerID,
  useHMSActions,
  HMSNotificationTypes,
  useHMSStore,
  useHMSNotifications,
} from "@100mslive/react-sdk";
import { ToastContainer, toast } from "react-toastify";
import "react-toastify/dist/ReactToastify.css";
import { useEffect, useState } from "react";

Initialise notifications and localPeerID to use them as follows:

const notification = useHMSNotifications();
const localPeerID = useHMSStore(selectLocalPeerID);

Using useEffect we check for any new notifications. Add the following code to App.jsx

const [pollNotificationData, setPollNotificationData] = useState();

useEffect(() => {
    if (!notification) {
      return;
    }
    switch (notification.type) {
      case HMSNotificationTypes.POLL_STARTED:
        if (notification.data.startedBy !== localPeerID) {
          console.log("NOTIFICATION RECEIVED");
          console.log(notification.data);
          setPollNotificationData(notification.data);
          toast(`A new Poll is available: ${notification.data.title}!`);
        }
        break;
      default:
        break;
    }
  }, [notification]);

Lastly, add <ToastContainer /> to the return of App.jsx to view a toast when some other remote user creates a new poll. Notice how we have added a check to not show toast to the local peer itself.

Now, we want to be able to cast our vote on the created poll. To do that we again start by showing a modal with the poll data for the users to vote. To the App.jsx add the following:

const [pollModalIsShown, setPollModalIsShown] = useState(false);

const showPollModalHandler = () => {
    setPollModalIsShown(true);
  };

In the return, add a ViewPoll component as follows:

{pollModalIsShown && (
            <ViewPoll pollNotificationData={pollNotificationData} />
          )}

Let us now work on this component. Create a new file called ViewPoll.jsx inside the Poll folder in the Components directory. First, create a basic form layout with radio buttons using the same Modal component we had created earlier.

import Modal from "../UI/Modal";
import { useState } from "react";
import { Button } from "@100mslive/roomkit-react";
import { useHMSActions } from "@100mslive/react-sdk";
import { toast } from "react-toastify";
import "react-toastify/dist/ReactToastify.css";

const ViewPoll = (props) => {

  return (
    <Modal onClose={props.onClose}>
      <h1>Poll: {props.pollNotificationData.title}</h1>

      <h3>{props.pollNotificationData.questions[0].text}</h3>

      <form onSubmit={handleSubmit}>
        <div className="radio">
          <label>
            <input
              type="radio"
              value={props.pollNotificationData.questions[0].options[0].index}
              checked={Number(selectedOptionIndex) === 1}
              onChange={handleChange}
            />
            {props.pollNotificationData.questions[0].options[0].text}
          </label>
        </div>

        <div className="radio">
          <label>
            <input
              type="radio"
              value={props.pollNotificationData.questions[0].options[1].index}
              checked={Number(selectedOptionIndex) === 2}
              onChange={handleChange}
            />
            {props.pollNotificationData.questions[0].options[1].text}
          </label>
        </div>

        <br />
        <Button type="submit">Submit</Button>
      </form>
    </Modal>
  );
};
export default ViewPoll;

And, lastly, add functions to handle changes made in the form and submission as follows:

  const actions = useHMSActions();
  const [selectedOptionIndex, setSelectedOptionIndex] = useState();

  function handleChange(event) {
    setSelectedOptionIndex(event.target.value);
  }

  const handleSubmit = async (event) => {
    event.preventDefault();

    await actions.interactivityCenter.addResponsesToPoll(
      props.pollNotificationData.id,
      [
        {
          questionIndex: props.pollNotificationData.questions[0].index,
          option: Number(selectedOptionIndex),
        },
      ]
    );

    toast(`Vote done!`);
  };

Notice how we are using the useHMSActions hook to addResponsesToPoll by passing the poll id obtained from the data passed in as props.

With this done, our application should now be ready to test!

The code for this project is available on GitHub here.

The React app that we’ve built here allows you to create the poll, add single-choice questions as well, and start it for remote users. Using the 100ms SDK, we can also create multiple-choice polls and quizzes as mentioned in the article before.

With the ability to curate customized polls, incorporate thought-provoking questions, and initiate real-time participation from remote users through the React app, a world of engaging possibilities opens up. The dynamic fusion of technology and user interaction empowers developers to not only enhance the virtual experience but also extract valuable data and insights.

Ready to dive in? To grasp the full extent of what's achievable and delve into practical steps, look at our comprehensive documentation.

Get started on the web here.
Add Polls to your Android or iOS app.
Join our discord server in case of any doubts and to help us build it further.

Buying Guide: Agora, Twilio, Jitsi (JaaS), Zoom, 100ms

Vrushti — Thu, 15 Sep 2022 12:48:26 +0000

Video conferencing has been around forever. But what used to be the sporadic coast-to-coast team catch-up, or the predominantly inside sales call became a household necessity with the 2020 Covid-19 Pandemic. Everyday work. Family reunions. Even doctor consults, workout trainers, and online astrologers. Oh, and never forget the kids jumping in the background, drowning in the virtual class they were supposed to attend.

That said, smart businesses have been quick to offer video-enabled communication services since, and the trend just seems to be getting started, whether it’s telehealth, test-prep, dating, or shopping. While businesses can obviously look at building this audio-video infra in-house, most find it time and resource-intensive.

Of course, building a scalable video infrastructure from scratch is no mean feat, unless that’s the primary focus you want for your engineering team. Luckily there are at least a handful of Video SDK providers that offer best-in-class video infrastructure.

But then, how do you decide which one works best for you?

To answer that, we decided to put together this Buying Guide.

First, we’ve handpicked the best of the best video SDKs based on customer reviews, product usage, and capabilities offered. And then, we battled them out on features & functionality, compliance & security, support, and pricing.

From features, time to respond, implementation help, and total cost of ownership, you should find all the details you need to make an informed buying decision right here.

The Buying Guide comprises four separate articles. Each article focuses on comparing vendors across a single, relevant parameter. All information for said comparison has been obtained from each vendor’s publicly available documentation.

Here is a quick summary of each article. Click on the article link for a deep dive.

Functionality and Features

Each audio-video SDK offers specific features and functions that enable its customers to meet their goals. We’ve listed out the basic offerings across Agora, Twilio, Jitsi (Jitsi as a Service), Zoom, and 100ms below.

However, you can go deeper and explore how vendors provide specific features such as streaming out with RTMP and HLS, active speaker detection, chat, polls, whiteboard, hand raise, and more in our detailed article.

Full article: Features & Functionality for for Agora, Twilio, Jitsi (JaaS), Zoom & 100ms

Agora

Users can build the video-calling feature using the SDK, while interactive live streaming can be built using Agora’s Live Streaming SDK. Call recording can be enabled using the dashboard and API.

Additionally, a noise reduction feature can be built using an additional integration. Agora also has a virtual background extension that allows for background modification. Background blur can be implemented using the SDK.

Twilio

With Twilio, users can build the video-calling feature using the SDK, while interactive live streaming can be built with Twilio Live. As for noise reduction, it can be built using the SDK. Call recording can be enabled using the dashboard and API. Background modifications can be built using the Twilio video processor SDK.

Jitsi (Jitsi as a Service)

Jitsi users can build video-calling, noise reduction, call recording, and background modification using the SDK. With regard to interactive live streaming, there is no explicit mention of it in the JaaS documentation.

Zoom

With Zoom, we have examined the feature offerings of both the Zoom Video SDK as well as the Meeting SDK.

Zoom Video SDK Users can build video-calling, noise reduction, and background modification features using the SDK. Call recording can be enabled using the dashboard and API. There is no explicit mention of interactive live streams in the Zoom documentation.

Zoom Meeting SDK Users can build video-calling, noise reduction, and background modification features using the SDK. Call recording can be enabled using the dashboard and API. For interactive live streaming, Zoom allows for the streaming of sessions with up to 10k participants.

100ms

100ms enables users to build video-calling, interactive live-streaming (HLS + WebRTC in a single SDK), background modification and much more using the SDK. Noise reduction is currently available in Beta. 100ms also allows for instant streaming of video conferencing sessions with up to 10k participants.

Compliance and Security

With data thefts and security breaches becoming more common with each passing day, it’s important for decision-makers to understand the security features each audio-video SDK offers. To help with that, we’ve listed the compliance certifications that each audio-video infra provider holds, below.

However, for more details about security features such as access control, enterprise authentication, end-to-end encryption, privacy and encryption of recordings, and audit trails, please take a look at our detailed article.

Full article: Compliance & Security for Agora, Twilio, Jitsi (JaaS), Zoom & 100ms

Agora

GDPR
HIPAA
CCPA
COPPA
ISO/IEC 27001
SOC 2

Twilio

GDPR
ISO 27001
AICPA SOC 2
HIPAA

Jitsi (Jitsi as a Service)

HIPAA
GDPR compliant for data processors

Zoom

SOC 2 Type II
GDPR
HIPAA
ISO/IEC 27001:2013

100ms

SOC2 Type 1 & SOC 2 Type 2

Support

Vendor support is as important a factor as the features they provide. Integration and post-integration support extended by these providers will help you when something breaks on the backend, and you need expert support.

While you can find a short summary explaining the support extended by Agora, Twilio, Jitsi, Zoom, and 100ms below, we’ve also created an elaborate comparison to give you a full overview in terms of Cost of Support, Integration/Account Management Support, Post-Integration Support, and Community Support.

Full article: Support for Agora, Twilio, Jitsi (JaaS), Zoom & 100ms

Agora
Agora has different paid support plans - Standard, Premium, and Enterprise. Apart from these, the platform offers one free support plan.

Twilio
Twilio has four different support plans — Developer (free), Production (paid), Business (paid), and Personalized (paid). As for the plans, support is based on the volume of usage.

Jitsi (Jitsi as a Service)
Jitsi’s open-source version is supported by a large community - the Jitsi community forum. Apart from this, Jitsi’s paid version, 8x8 Jitsi as a Service, includes dedicated support for strategic customers, as explained in their Global Premium Plus Support plan.

Zoom
Zoom offers dedicated support for developers via Premier Developer Support which offers prioritized developer-specific resources. This guide does not explore Zoom support plans aimed at non-dev users and admins.

100ms
All support functions are available to paying customers at no additional cost. 100ms also offers testing support at no extra cost. This includes user testing, load testing, and network/device stress-testing.

Pricing

Pricing is one of the most crucial deciding factors when it comes to purchasing an SDK. As with the other parameters, here’s a quick overview of what each vendor charges.

However, details on each vendor’s pricing policies are beyond the scope of this piece. We’ve put together a detailed guide that explains the pricing models, pricing for recording, live streaming, add-ons, and more for each of these providers.

Full Article: Pricing for Agora, Twilio, Jitsi (JaaS), Zoom & 100ms

Agora
Agora’s pricing is based on usage, which includes the number of minutes used, the number of users, and the unit price. However, Agora follows pricing on the basis of aggregate resolution in calls - this has been explained in detail in the main article.

The unit price per 1,000 minutes is $0.99 (for audio) and $3.99 (for video HD). Pricing for Video Full HD, Video 2K, and Video 2K+ is explained in detail within the main article.

Twilio
Pricing for Twilio scales with participant minutes. We have examined the pricing of two Twilio video products - Twilio P2P & Twilio Video Groups.

Twilio P2P Allows up to 3 participants and up to 10 audio-only participants. Priced at $0.0015 per participant per minute.

Twilio Video Groups Allows users to create video apps for up to 50 participants. Priced at $0.004 per participant minute - defined only by the minutes a user spends connected in a room.

Jitsi (Jitsi as a Service)
Jitsi’s pricing is based on a per-user model and charges for monthly active users (MAU). According to Jitsi, an MAU is a user who attends at least one meeting with at least one other user within a particular month. An MAU is also tracked on the basis of the device they log in from.

JaaS offers various plans, and the pricing for each plan varies depending on the MAU. The JaaS Dev plan allows up to 25 MAU free, and only add-ons are charged extra. Under the JaaS Basic plan, pricing is $99 per month for 300 MAUs. The pricing for JaaS Standard, JaaS Business, and plans with more than 3000 MAU are mentioned in the main article.

Zoom
Zoom offers two SDKs: a Video SDK(charged on the basis of usage) and a Meeting SDK (charged on per user basis).

Zoom Meeting SDK The Meeting SDK offers four paid tiers: Basic, Pro, Business, and Enterprise. To use the Meeting SDK, the host is the only person who must purchase and hold a license. This license carries a specific limit on the number of participants who can be supported in each tier.

Zoom Video SDK Under Zoom Video SDK, there are two pricing levels. With pay-as-you-go, you get 10,000 minutes per month, after which the plan is priced at $0.0035 per minute. At the second level, you pay $1000 per year with 30,000 minutes included per month. After the limit is crossed, you pay $0.003 per minute.

100ms
100ms offers a single SDK offering both video conferencing and live streaming capabilities with straightforward pricing for both use cases.

100ms offers 10,000 free minutes for conferencing and another 10,000 free minutes for streaming for each business every month in addition to 1000 free encoding minutes.

Beyond this, video conferencing is charged at $0.004 per participant per minute, while audio-only calls are charged at $0.001 per participant per minute. Live Streaming costs $0.004 per broadcaster and $0.0012 per viewer per minute, while additional encoding minutes are charged at $0.04.

While there are various parameters and aspects that PMs, CTOs, and decision-makers need to keep in mind, these are a few that we considered indispensable. We recommend that you have a look at the separate pieces focusing on each of these parameters. They offer research-based data from each vendor, as well as helpful links you can use to conduct in-depth research yourself.

To know more about how 100ms can help fill in your video conferencing requirements, book a call with us.

Introducing 100ms Starter Kits

Vrushti — Tue, 06 Sep 2022 10:18:36 +0000

Building live audio or video is hard.

At 100ms, we’re working on simplifying that process because we believe that nearly all apps will have live audio/video in the future — a “live-first” digital world if you will.

100ms is industry-agnostic, thus enabling product managers (PMs), developers, and engineers to shape and build real-time, life-like interactions the way they see fit for a multitude of functions. In fact, with the 100ms SDK, our customers have already built diverse use cases across industries such as dating, gaming, education, and the like — delivering millions of live audio/video minutes to their users.

But, as modern users increasingly demand real-time interactive experiences online, the delta for delivering value has changed astronomically. To meet these demands closely, PMs and developers want to experience a feature before they build it out themselves. This is where the 100ms Starter Kits come in.

Bridging Imagination and Experience
After launching our Virtual Event Starter Kit in partnership with Vercel, we realized that our customers loved the fact that we were able to instantly deliver a working demo of a Virtual Event Use Case. Since the starter kit is open source, anyone could take the code to extend that experience.

By enabling this, we realized that users could instantly connect their imagination to what the app would actually look like.

In other words, we were able to bridge the gap between imagination and experience.

This is what inspired us to build the 100ms Starter Kits.

What are these Starter Kits?
These starter kits are proof-of-concept versions of use cases constructed on real-life interactions built with the 100ms SDK. We developed them with the hope that you’ll use them to actualize a feature/app you’ve been thinking about for a while, and go “This is what I wanted to build!”

For example, let’s say you wanted to simulate the action of tapping a colleague’s shoulder to discuss something — while in an online meeting. These starter kits enable you to experience that exact feature by providing a one-click demo. You can also choose to deploy them and start experimenting.

These starter kits are a quick jumping-off point to demonstrate a working, proof-of-concept version of whatever you are imagining. Moreover, these kits also serve as the building blocks for implementing your own ideas.

What can you do with these Starter Kits?

Instant Demos: Immediately experience our starter kits with the View Demo option in our Examples section. This allows you to measure the audio and video quality of our SDKs with the added context of User Interfaces.

Open Source Github Repos: You can download the source code of these starter kits and break it down for reference code implementations. You can also experiment and build on these starter kits with your ideas.

Deploy to Vercel: Deploy these starter kits to your Vercel account and experiment with them in your own environment.

Breaking down a Starter Kit
Our starter kits are open-sourced apps wrapped as frontend layers around our template policy (business logic around roles & permissions) and the 100ms SDK.

To break this down further, let’s use an example.

Let’s say you want to build a one-click engagement solution between coworkers — something as simple as tapping a colleague’s shoulder to talk to them, but in a virtual environment (like a Slack Huddle).

Before this launch, you could have created an audio room template to implement this action. But while you would’ve had an audio-first conversation with it, a holistic experience isn’t being realized. Now, with the starter kit, you can build on this experience, add an interactive one-click button to the app, and get much closer to a product that replicates real-world, in-person communication.

In the above example, the audio room is the template (use case), and the frontend UI is wrapped around it, allowing for the solution to show up as a one-click engagement. Combined with the 100ms SDK, the entire package comprises a single Starter Kit app.

Five Starter Kits You Can Try Out

As of now, we have rolled out five starter kits with various use-cases with 100ms Examples. Some of them have actually been developed with the help of our amazing community.

We look forward to adding more of these kits in the near future, so as to enable more relatable and delightful quick-start experiences. But for now, these are our initial rollouts:

1. Video Conference Starter Kit
Offer your customers engaging live conference experiences with excellent audio/video quality via 100ms’ Video Conferencing Starter Kit. This is a full-fledged feature-rich starter kit for building any audio/video conferencing product.

2. Virtual Event Starter Kit

Own your live event experience with this — a virtual events starter kit with real-time audio-video interactions that you can configure on the go. With this starter kit, you can host a live event or a live workshop with 10,000 viewers.

3. Slack Huddle Clone

Help users get on impromptu, lightweight audio calls for a quick, real-time conversation with the 100ms Slack Huddle Clone kit. With this starter kit, you can build the “quick tap conversation” use case discussed above.

4. Discord Clone

Offer your users an excellent audio conferencing/streaming experience with in-built advanced interactivity. Use our Discord Starter Kit to host or build a community discussion experience.

5. Audio Room

Build audio-first apps with this starter kit and engage your users with experiences like live audio calling, podcast streaming, clubhouse-like audio rooms, and more.

Closing Notes
With the launch of Starter Kits, we aim to unveil a new dimension of experimentation for our users. So head to the Examples section from the navbar and start building your own live experience. Don’t forget to share your apps with us, because we can’t wait to see what you build!

If you are looking to contribute to these Starter Kits or partner with us, reach out to us in our Discord community!

Server-side Considerations for your WebRTC Infrastructure

Vrushti — Fri, 19 Aug 2022 06:04:00 +0000

Internet users today expect flawless online experiences, be it browsing Instagram, streaming BTS, or debating anime fandoms. This expectation extends to online video communication.

If they are to meet the expectations of contemporary users, video conferences must offer sub-second latency and high-quality audio/video transmission. Usually, developers choose WebRTC to build video experiences of this caliber.

You might’ve read or heard about how WebRTC is a client-oriented protocol that usually doesn’t require any server to function. However, that’s not the whole story.

The Server in WebRTC

It is true that some WebRTC calls are possible without any need for an external server. But, even in those cases, a signaling server is required to establish the connection.

Most calls would simply fail on a direct connection and even if they do connect, issues start arising as more peers join in — something we will discuss later in the article. To solve these issues, it is recommended that you use different servers that provide workarounds and call performance optimization in WebRTC ecosystems.

This article will discuss a few elements on the server side you must consider when building a WebRTC solution. We will talk about the servers, multi-peer WebRTC architecture, and how it all works — so that you make the right architecture choices for your WebRTC application.

The following is the list of servers explored in this piece. Some of these servers are mandatory while the presence of others will depend on the architecture you choose to work with:

Signaling Server (Mandatory)
STUN/TURN Servers (Mesh Architecture)
WebRTC Media Servers a. MCU Server (Mixing Architecture) b. SFU Server (Routing Architecture) c. SFU Relay (Routing Architecture)

Signaling Server

Signaling refers to the exchange of information between peers in a network. It is required to set up, control, and terminate a WebRTC call. WebRTC doesn’t specify a rigid way for signaling peers, which makes it possible for developers to manage signaling as they see fit. To implement this out-of-band signaling, a dedicated signaling server is often used.

The signaling server is mainly used to initiate a call. Once that is done, WebRTC will take over. However, this does not mean that we won’t require the signaling server once the call has started.

Even though most state changes like voice mute/unmute can be notified to the other peer(s) through WebRTC data channels, the signaling server has to be present throughout the call. It must be used to handle unusual scenarios like network disconnection where the peer requires signaling to reconnect to the call again.

Now, we will take a look at the multi-peer architecture in WebRTC, before moving on to the servers we might need once signaling is complete.

WebRTC multi-peer architecture

In WebRTC, there are multiple architectures that define how the peers are connected in a call. Generally, the server-side requirements depend on the architecture that you choose. Picking the right architecture for your use case helps identify the servers you will need.

We will now take a look at the most popular WebRTC architectures:

Mesh architecture
Mixing architecture
Routing architecture

Below, we discuss these architectures along with the servers they need to function properly.

Mesh Architecture (STUN/TURN Servers)

In this architecture, every peer is directly connected to every other peer in the call. For example, in a call with 4 peers, every peer has to send their video to 3 other peers and receive video from the same 3.

This is generally suitable for WebRTC calls with a limited number of peers.

NAT restrictions and Firewall:

A NAT(Network Address Translation) router is used to map the private IP address to a public IP address under its network. When a peer is behind a NAT router, it only has knowledge of its private IP address (which is invalid outside its local network). Thus, it cannot exchange its actual public IP address during signaling phase.
Additionally, when the peers are behind Symmetric NAT(a type of NAT) router, it becomes impossible for the peer to connect directly due to its unique mapping technique that returns different port addresses for different connections in the network.
In some cases, the Firewall on a peer’s device might block a direct connection with another peer over the internet for security reasons.

When a direct connection is not possible in this architecture, NAT traversal servers like the STUN and TURN server can be used.

Session Traversal Utility for NAT (STUN)
A STUN server is used to retrieve the public IP address of a device behind NAT. This allows the device to communicate after learning its address on the internet. This is enough for roughly 80% of connections to be successful, but it cannot be used for cases where the peers are behind a Symmetric NAT.

Traversal Using Relay NAT (TURN)
A TURN server is used to relay media between peers when a direct connection is not possible — often due to a Symmetric NAT in the network or a firewall blocking connections.

The TURN server is also known as the “relay” server and costs more than STUN to maintain because it relays media throughout a WebRTC connection. Since the TURN server is an extension of STUN, its implementations include a STUN server built into it by default.

For more details on STUN/TURN servers and how to use them in a simple WebRTC video app, have a look at Build your first WebRTC app with Python and React.

Advantages and Disadvantages of Mesh Architecture

Advantages

No need for a central media server as the connection is peer-to-peer. This reduces server costs.
Relatively simple to implement in WebRTC.

Disadvantages

Each participant has to send media to every other peer, which requires N-1 uplinks & N-1 downlinks.
Not much control over the media quality.
Exploring the Mixing and Routing architectures requires some familiarity with the idea of WebRTC Media Servers. So let’s start with that.

WebRTC Media Servers

In the Mesh architecture, bandwidth expenditure becomes quite high for peers when the number of people in the call exceeds 4. The resource consumption tends to skyrocket, overheating the peer devices to the point that they can malfunction or even crash.

Therefore, for use cases with more than 4 people in a call, it is recommended that you choose an architecture based around a media server.

WebRTC Media Servers are central servers that peers send their media to, and receive processed media from. They act as “multimedia middleware” and can be used to offer several benefits. But, trying to implement one from scratch isn’t exactly a walk in the park.

Even if we use a media server, we must ensure that they’re available to the peers via TCP with the help of a TURN server sometimes.

A well-implemented media server is highly optimized for performance and can offer numerous capabilities outside its main requirement. Here are some useful features an ideal media server should have:

Simulcast
Video is served to the peers at different bitrates based on their configuration or network conditions. The peers send their video at multiple resolutions and bitrates to the media server and it chooses which version to send to each peer.

Recording
Call recording is made possible either by directly forwarding all incoming media from the server to storage or by connecting a custom peer to the server that receives all media streams and stores them.

Transcoding
Not all peers connected to the call might support the same audio/video codec. The media server should be able to resolve this issue by transcoding the audio/video to an appropriate codec supported by all, before sending the streams out.

Audio/Video Optimisations
Customized audio/video optimization should be possible. The server should be able to send only the media of active speakers to reduce bandwidth consumption, selectively mute audio from someone, or prioritize screen-share media over other videos.

Here are some of the widely used media servers:

- MCU (Multipoint Control Unit)
- SFU (Selective Forwarding Unit)
- SFU Relay (Distributed SFU)

Now, let’s discuss the architectures (Mixing and Routing) that use these media servers to solve issues commonly faced in the Mesh architecture — high bandwidth expenditure and heavy resource consumption.

Mixing Architecture (using the MCU Server)

In this setup, all peers send their media to a central media server. Then, the media server operates on the media gathered, packs it into a single stream, and sends it to all peers. Here, every peer sends a single media stream to and receives one media stream from the server. The media server used here is called Multipoint Control Unit (MCU).

Multipoint Control Unit (MCU)
An MCU server receives media from all peers and reworks it, performing the following functions:

1. Decoding: Upon gathering the primary media streams from all peers, the MCU decodes them.

2. Rescaling: The decoded videos are rescaled based on the peer’s network conditions.

3. Composing: The rescaled videos are combined into a single video stream in a layout requested by that peer.

4. Encoding: Finally, the video stream is encoded for delivery to the peer.

This process is done parallelly for every single, separate peer in the call. This makes it easy for peers to send and receive media as a single stream without spending too much bandwidth.

Advantages and Disadvantages of Mixing

Advantages

The server sends a single media stream to the peer, which makes it possible for devices with lower processing power to participate in the call.
Requires very little resource or bandwidth due to the peer having just a single uplink and downlink.
Server-side recording is possible.

Disadvantages

The server requires high processing power and is generally costly to maintain.
Peers may experience delays in receiving media packets, as they have to be processed before sending from the server.

While this sounds like a great option, the number of peers in a call directly depends on the performance of the MCU in the Mixing architecture. In reality, it is hard to maintain a WebRTC call with more than 30 peers in Mixing architecture, without the MCU heavily draining server resources.

Despite this, Mixing remained the most widely used WebRTC architecture until a few years back. However, it has been slowly replaced by the Routing architecture.

Routing Architecture (SFU Server)

In this setup, all peers send their media to a central media server. The server forwards the media streams to all other peers separately, without operating on them in any way. Here, every peer sends a single media stream to the server and receives N-1 media streams (where N is the number of peers present in the call) from the server. The media server used here is called the Selective Forwarding Unit (SFU).

Selective Forwarding Unit (SFU)
An SFU server receives media from all peers in a call. Then, all that media is routed “as is” to every other peer connected to the server. The peers can send more than one media stream to the server, making simulcast possible. The SFU can also be customized to automatically decide which media stream to send to a specific peer.

SFU Relay (Distributed SFU)
This is a fairly recent development in the domain. SFU Relay servers are simply SFU servers that can communicate with each other. One SFU server can relay media to another, creating a distributed SFU structure. This reduces the load on a single SFU and makes the whole network more scalable. In theory, any server can connect to the SFU relay via an API and receive the routed media.

Advantages and Disadvantages of Routing

Advantages

Less demanding on server resources compared to options like MCU.
Works with asymmetric bandwidth (lower upload rate than download rate) for a peer, as there is only a single uplink with N-1 downlinks.
Simulcast is supported for different resolutions.

Disadvantages

Server-side recording is not possible. But it is possible to route media to a peer that records the streams.
The peer device must be good enough to handle multiple downlinks, unlike in Mixing architecture.
It requires complex design and implementation on the server side.

Closing Notes

To wrap up, let’s quickly summarise the architectures and corresponding use cases discussed above:

Mesh Architecture with a STUN/TURN server is ideal for calls with 4 or fewer peers.
Mixing Architecture with an MCU server is good for calls with more than 4 peers. It is highly used in cases where support for legacy devices is a necessity.
Routing Architecture with an SFU server is the modern approach to WebRTC video conferencing. As of now, this is the ideal approach to connecting peers in a call when their number exceeds the limits of the Mesh architecture.

It is also possible to dynamically switch between different architectures based on the call size, so as to find a balance between app performance and server costs.

Choosing and implementing the appropriate WebRTC server and architecture is just one side of the coin. Much more is required to make your WebRTC service reliable — optimizing performance, reducing call failure rate, and handling edge cases.

If you’re planning on writing your own media server, you should also be aware of some basic problems that often show up in the process:

Managing peers with a bad network connection.
Helping peers that cannot support all the mixed codecs running in the call.
Handling peer reconnections as well as new peers joining in and existing peers leaving the call.
Algorithms to perform bandwidth estimation for a peer so that the server does not send more data than it can handle.

In short, even if you choose the right server-side architecture, you will still have a lot to deal with regarding the technicalities of WebRTC.

If you don’t want to deal with the intricacies of WebRTC but still want to host calls with a gold standard video solution (be video conferencing or streaming), you have options like 100ms to do the heavy lifting for you.

WebRTC with 100ms

100ms’ live video SDKs allow you to add live video capabilities to your application with just a few lines of code. With multiple highly-relevant features and a predictable pricing plan, you don’t have to worry about dealing with exorbitant server costs for your app.

If your application requires high-quality video capabilities but you’re unsure about building it from scratch, or you just don’t want to deal with fine-tuning the nitty-gritty of WebRTC, 100ms is your best bet.

A New Approach to Live Streaming

Vrushti — Tue, 16 Aug 2022 10:41:28 +0000

The long arc of tech progress has shown that user behavior and technology evolve together: changes in user behavior inspire technology, and then technology drives more of that behavior. The world of live streaming is going through one such change, where new user behavior is driving us to reevaluate the live streaming tech stack.

Since the first live stream in 1995 where the Yankees were playing the Mariners, live streaming has now become an important medium for users on the Internet to learn, play, shop, and work. Who gets to stream and how they interact with their audience is changing rapidly, and this change is informing our approach to building infrastructure for live streaming.

The present-day live streaming tech stack

Most live streaming apps today are built by combining RTMP encoded media streams at the streamer’s end and HLS streams at the viewer’s end. An industry of media servers in the middle exists to transcode the input format into the output stream.

RTMP is a mature protocol that was originally built to support Adobe Flash. Given its maturity, RTMP is widely supported by encoding software and hardware, which can ingest raw device streams and output RTMP streams. RTMP is also fast: it optimizes for reduced latency.

RTMP used to work well on the viewer’s end too, given that it was the preferred streaming protocol for Adobe Flash. However, as Flash usage went down and HTML5 emerged, HLS became a better fit for viewers. HLS is built over HTTP and is widely supported across all mobile and desktop devices.

The combination of RTMP and HLS has worked well given the asymmetry in live streaming personas: there are many more viewers who require frictionless viewing and there are only a few streamers who need to configure specialized encoding software (like OBS).

So what’s changing?

User behavior around live streaming is changing in 3 big ways: democratization, interactivity, and creator collaboration.

Democratization
Live streaming has been democratized and is no longer limited to professional streamers using sophisticated equipment connected to reliable broadband. Everyone is now streaming live with Instagram and YouTube, and from mobile devices connected to unreliable networks.

Interactivity
Live streams are no longer one-way broadcasts. Streamers and viewers are looking for ways to engage and interact with each other. Chat and emoji reactions running alongside live streams are now table-stakes.

More recently, we have come across scenarios where viewers get “promoted” into becoming streamers. This enables new stream formats and increases the engagement between streamers and viewers.

Creator collaboration
Streamers are also experimenting with newer formats that involve collaborating with other streamers. As Ping Labs puts it, video calls have now become video content.

The pandemic has accelerated these changes. Live streaming creation and viewership shot upwards, and that motivated more streams, more interactivity and more experimentation with stream formats.

What will happen to the live streaming tech?

Live streaming works well on the viewer’s end. HLS, and similar protocols like MPEG-DASH, have democratized viewership by building on top of HTTP. Anyone with a web browser or a smartphone can view live streams today.

It's time something similar happens on the streamer’s end. The solution to democratization, interactivity, and creator collaboration will be found in WebRTC becoming an alternative to RTMP in the live streaming tech stack.

Given the times it was designed in, RTMP is unsuitable for streaming from mobile devices. It is built over TCP and assumes a fixed encoding bitrate. When the device runs into network disruptions, an RTMP encoder keeps producing output, which further chokes the network. WebRTC is a more modern protocol: it’s built over UDP and can adjust encoding bitrates based on network feedback.

WebRTC is also more widely available. Any modern web browser today can encode WebRTC streams without requiring any additional software. Native apps on iOS and Android also support WebRTC well.

WebRTC is also built for interactivity, given that it was originally a solution to real-time video conferencing. Chat and other forms of interactivity are easy to achieve on top of WebRTC. It is also possible to invite HLS viewers as WebRTC participants, which makes it suitable for advanced interactivity scenarios, where the viewer is promoted into becoming a streamer.

Given its roots in conferencing, WebRTC also supports creator collaboration out of the box. Streamers can join in from different device platforms given that WebRTC is everywhere.

The worlds of live video will merge
Given the evolution in user behavior it is time these world began to merge. Streaming use-cases will leverage conferencing tech to introduce interactivity and other benefits. Conferencing will leverage streaming tech to scale video calls to many viewers in near real-time.

Get access
We are a team that has built live video products in companies like Disney+ and Facebook and are now applying that expertise to enable thousands of developers to build live video apps.

We are excited by the creativity of our customers who are imagining new use-cases of live video every day. Developers and product managers are building experiences that mix the worlds of conferencing and streaming, and we are building infrastructure to enable them to do more with less.

If this evolution in live video excites you and is relevant to your needs, try live streaming with 100ms. Sign up to get started and join our Discord community to connect with us. We look forward to seeing what you build.

How 100ms tests for Network Reliability

Vrushti — Tue, 09 Aug 2022 14:37:00 +0000

From virtual classrooms to business meetings, shopping to dating apps, video is quickly becoming the de-facto communication mode online.

Innovative developers and product thinkers are looking to create engaging live experiences in their applications. So naturally, it's critical that the audio-video SDK they build these experiences on top of provides a stable, extensible, and scalable bedrock.

Among the many factors to consider before purchasing an audio/video SDK, network reliability stands out. After all, nobody enjoys running a twenty-minute monolog on a video call only to realize your network was down the entire time…

Testing Network Reliability for Real-World Scenarios

In this article, we've downloaded, deployed, and tested the reliability of the 100ms React SDK. To do so, we designed a series of tests that simulate common scenarios in real-life. Of course, since that's not fun enough, we decided to unleash our “full crazy” by battle testing each round against extreme conditions.

The tests verify how the 100ms SDK fares across three parameters that define network reliability: low bandwidth, network blips & network switching.

Why Network Reliability Matters

In the real world, individuals often have to deal with unstable or less-than-ideal network conditions. This happens when:

moving from one network area to another while traveling
suddenly experiencing slow internet because of an expiring data pack
suddenly experiencing call disconnection for a few seconds due to issues in the larger infrastructure
Network connectivity issues occur more often than we think. Video SDKs need to, at best, be resilient to these issues, and, at worst, provide developers with tools to deal with them gracefully.

Deployment
100ms has a sample React app (100ms 2.0 Sample React App) meant to facilitate the testing of its SDK. We deployed it on Heroku and exposed it to a few commonly occurring end-user scenarios.

https://github.com/100mslive/100ms-web

We had to generate some credentials from the 100ms console and then deployed this example React app on Heroku.

The SDK was deployed and tested on the Chrome browser running on macOS Monterey.

Conditions and cases

All these tests were 1:1 calls, performed with 2 people in the room. A few details about each test before we get into the results:

Low Bandwidth Test

Network speed varies across devices. For instance, users operating on 4G mobile data often experience a volatile network, as it tends to vary in speed and stability. In this test, we checked how 100ms handles calls with varying connection speeds on low bandwidth.

Network Blip Test

Network crises can happen in the middle of a call. In this test, we checked how 100ms handles the sudden loss of network connectivity followed by automatic reconnection.

Network Switching Test

It is common for users to switch between networks inadvertently. For example, they might be on a call while moving between state lines or from a city to the countryside, which may affect network strength.

Network switching usually occurs when you move away from the range of one network to another or when you switch between your available networks for a higher speed. In this test, we checked how 100ms handles a network switch.

1. Low Bandwidth Handling/Management Test

Audio/Video applications need to handle usage across varying network bandwidths. In this section, we monitor how 100ms handles calls for users with low bandwidth.

Testing Methodology for the Low Bandwidth Test

We used Network Link Conditioner to emulate different network conditions. We set the ideal resolution to 640x360, and tested the app on 4 different configurations: 300 Kbps, 500 Kbps, 800 Kbps, and 1 Mbps, switching from one to another in the middle of a call.

Test Results
The 100ms SDK handles the drop in bandwidth by prioritizing audio/video upload for other peers instead of audio/video download.

If the network is adequate (800 Kbps), the video of active or recent speakers continues to be visible. The audio remains perfectly functional.
If the network is poor, only peer audio is functional while their video degrades.
On the device facing poor network conditions, the video is somewhat degraded but not entirely non-functional. At lower bandwidths (500Kbps and 300Kbps), audio quality remains functional for all other peers in the meeting and only sees a drop for the attendee experiencing bandwidth constraints.

2. Network Blip Test

In this section, we check how 100ms handles call connectivity when a user’s network connection gets switched off, or drops for several seconds.

Testing Methodology for the Network Blip Test
First, we check the call by switching off the internet connection for 10 seconds. This is done by toggling the connected wifi network from the menu bar and connecting back by re-toggling the same.

Then, we iteratively repeat the same test for 20, 30, 45, and 60 seconds. While doing so, we observe the state of the call connection and how the app behaves during disconnection.

Test Results
The 100ms SDK reconnects every time when internet is disabled for 10, 20 and 30 seconds. When switched off for 45 and 60 seconds, the app tries to reconnect for 35s before disconnecting entirely.

3. Network Switching Test

Apps are often exposed to different network conditions in the real world. In this case, we’ve tested how the 100ms SDK reacts when the app moves from one network strength to another.

Testing Methodology for the Network Switching Test
This test checks how 100ms handles connection when switching from one network to another. We tested the app in 3 Wi-Fi networks:
2.5G and 5G from the same router, and a mobile hotspot.

To start the call, we connected to the Wifi 2.5G network.
Then, we switched from Wifi 2.5G to Wifi 5G.
Then, we switched back to Wifi 2.5G.
Then, we repeated the same process, switching to and from Wifi 2.5G and the Mobile Hotspot.

We waited for the call to reconnect during every network switch and monitored the time (in seconds) it took for the reconnection to occur.

Some of the flawed behavior in the ‘Wifi 2.5G to Hotspot’ test section might be due to the unstable 4G network connection we experienced while testing.

Test Results
The 100ms SDK manages to reconnect after every network switch. Sometimes the video reconnects after the audio. The average reconnection time when switching within the same network is 9.1s for audio and 10s for video. The time for reconnection between 2 different networks is 19.2s for audio and 13.8s for video.

Closing Notes

Given the centrality of reliability when it comes to choosing an audio-video SDK, we decided to lay all our cards on the table and reveal exactly how we fare in diverse network, bandwidth and end-user circumstances. Across all tests 100ms fared well under regular usage conditions. In some cases, like bandwidth drops, the SDK allows for graceful handling of degradation issues.

Of course, as an SDK provider, we pride ourselves for making 100ms even more bullet-proof, so we can’t wait to elegantly solve across all these conditions and meet you again with even more aggressive scenarios.

HLS 101: What it is, How it works & When to use it

Vrushti — Tue, 09 Aug 2022 13:06:18 +0000

If you consume online content (and if you are alive in 2022, you probably do), chances are that you’ve watched quite a few live streams. Be it online classes, sporting events, fitness lessons, or celebrity interactions, live streaming has quickly become the go-to source of learning and entertainment.

Live streamers comprised over 1/3rd of all internet users in March and April 2020, with only 1 in 10 people in the US and UK streaming live content of their own. Just two years since, almost 82% of internet use is expected to be devoted to streaming video by 2022.

A vast majority of live streaming applications are built on a protocol called HTTP Live Streaming, or HLS. In fact, if you’ve ever watched an Instagram live stream or tuned into the Super Bowl on the NBC Sports App, chances are, you’ve been touched by the magical hands of HLS.

So if you are looking to build that kind of sophisticated live streaming experience inside your app, this article should give you a comprehensive understanding of the HLS protocol and everything in it.

Read on to learn the basics of HLS, what it is, how it works, and why it matters for live streamers, broadcasters, and app developers.

What is HLS?

HLS stands for HTTP Live Streaming. It is a media streaming protocol designed to deliver audio-visual content to viewers over the internet. It facilitates content transportation from media servers to viewer screens — mobile, desktop, tablets, smart TVs, etc.

Created by Apple, HLS is widely used for distributing live and on-demand media files. For anyone who wants to adaptively stream to Apple devices, HLS is the only option. In fact, if you have an App Store app that offers video content longer than 10 minutes or is heavier than 5MB, HLS is mandatory. You also have to provide one stream, at the very least, that is 64 Kbps or lower.

Bear in mind, however, that even though HLS was developed by Apple, it is now the most preferred protocol for distributing video content across platforms, devices, and browsers. HLS enjoys broad support among most streaming and distribution platforms.

HLS allows you to distribute content and ensure excellent viewing experiences across devices, playback platforms, and network conditions. It is the ideal protocol for streaming video to large audiences scattered across geographies.

A little history
HLS was originally created by Apple to stream to iOS and Apple TV devices, Mac on OS X in Snow Leopard, and later OSes.

In the early days of video streaming, Realtime Messaging Protocol (RTMP) was the de-facto standard video protocol for streaming video over the internet. However, with the emergence of HTML5 players that supported only HTTP-based protocols, RTMP became inadequate for streaming.

With the rising dominance of mobile and IoT in the last decade, RTMP took a hit due to its inability to support native playback in these platforms. The Flash Player has to give away ground to HTML5, which resulted in a decline in Flash support across clients. This further contributes to RTMP’s unsuitability for modern video streaming.

In 2009, Apple developed HLS, designed to focus on the quality and reliability of video delivery. It was an ideal solution for streaming video to devices with HTML5 players. Its rising popularity also had much to do with its unique features, listed below:

Adaptive Bitrate Streaming
Embedded closed captions
Fast forward and rewind
Timed metadata
Dynamic Ad insertion
Digital Rights Management(DRM)

How does HLS work?

HLS has become the default way to play video on demand. Here’s how it works: HLS takes one big video and breaks it into smaller segments (video files) whose length varies, depending on what Apple recommends.

Here’s an example:

Let’s say there is a one-hour-long video, which has been broken into 10-second segments. You end up with 360 segments. Each segment is a video file ending with .ts. For the most part, they are numbered sequentially, so you end up with a directory that looks as seen below:

The video player downloads and plays each segment as the user is streaming the video. The size of the segments can be configured to be as low as a couple of seconds. This makes it possible to minimize latency for live buffering use cases. The video player also keeps a cache of these segments in case it loses network connection at some point.

HLS also allows you to create each video segment at different resolutions/bitrates. Take the example above. In this, HLS lets you create:

Here’s what the directory looks like now:

Once these segments are created at different bitrates, the video player can actually choose which segments to download and play, depending on the network strength and bandwidth available. That means if you are watching the stream at lower bandwidth, the player picks and plays video segments at 360p. If you have a stronger internet connection, you get the segments at 1080p.

In the real world that means the video doesn’t get stuck, it just plays at different quality levels. This is called Adaptive Bitrate Streaming (ABR).

What Adaptive Bitrate Streaming means in the real world
Imagine you’re streaming the Super Bowl live on your phone (because you just had to drive out of town that day). Just as the Rams are racing towards their winning touchdown, you hit a spot of questionable network in the Nevada desert.

You’d think that means the livestream would basically stop working because your network strength has dropped. But, thanks to ABR, that wouldn’t be the case.

Instead of ceasing to work, the steam would simply adjust itself to the current network. Let’s say you were watching the stream at 720p. Now, you’d get the same stream at 240p. That means, even though there is a drop in video quality, you would still be able to see Cooper Kupp take his MVP-winning touchdown. HLS would enable this automatically, simply by just adjusting to a lower quality broadcast to match your network.

HLS Streaming Components
Three major components facilitate an HLS stream:

The Media Server,
The Content Delivery Network, and
The Client-side Video Player

HLS Server (Media Server)
Once audio/video has been captured by input devices like cameras and microphones, it is encoded into a format that video players can translate and utilize: H.264 for video and AAC or MP3 for audio.

The video is then sent to the HLS server (sometimes called the HLS streaming server) for processing. The server performs all the functions we’ve mentioned — segmenting video files, adapting segments for different bitrates, and packaging files into a certain sequence. It also creates index files that carry data about the segments and their playback sequence. This is information the video player will need to play the video content.

Content Delivery Network (CDN)
With the volume of video content to store, queue, and process, a single video server responding to requests from multiple devices would likely experience immense stress, slow down and possibly crash. This is prevented by using Content Deliver Networks (CDNs).

A CDN is a network of interconnected servers placed across the world. The main criteria for distributing cached content (video segments in this case) is the closeness of the server to the end-user. Here’s how it works:

A viewer presses the play button, and their device requests the content. The request is routed to the closest server in the CDN. If this is the first time that particular video segment has been requested, the CDN will push the request to the origin server where the original segments are stored. The origin server responds by sending the requested file to the CDN server.

Now, the CDN server will not only send the requested file to the viewer but also cache a copy of it locally. When other viewers (or even the same one) request the same video, the request no longer goes to the origin server. The cached files are sent from the local CDN server.

CDN servers are spread across the globe. This means requests for content do not have to travel countries and continents to the origin server every time someone wants to watch a show.

HTML5 Player
To view the video files, end-users need an HTML5 player on a compatible device. Ever since Adobe Flash passed into the tech graveyard, HLS has become the default delivery protocol. Getting a compatible player won’t be a challenge since most browsers and devices support HLS by default.

However, HLS does provide advanced features which some players may not support. For example, certain video players may not support captions, DRM, ad injection, thumbnail previews, and the like. If these features are important to you, make sure whichever player you choose supports them.

Resolving Latency Issues in HLS

Apple recommended batches of 10-second segments until 2016. That particular spec focused on loading three video segments before the player could start the video. However, with 10-second segments, content distributors suffered a 30-segment latency before playback could begin. Apple did eventually cut down the duration to 6 seconds, but that still left streamers and broadcasters with noticeable latency. Even since then, reducing segment size has been a popular way to drive down latency. By ‘tuning’ HLS with shorter chunks, you can accelerate download times, which speeds up the whole pipeline.

In 2019 Apple released its own extension of HLS called Low Latency HLS (LL-HLS). This is often referred to as Apple Low Latency HLS (ALHLS). The new standard not only came with significantly lower latency but was also compatible with Apple devices. Naturally, this made LL-HLS a massive success and has been widely adopted across platforms and devices.

LL-HLS comes with two major changes to its spec which are largely responsible for reducing latency. One is to divide the segments into parts and deliver them as soon as they’re available. The other is to ensure that the player has data about the upcoming segments even before they are loaded.

A detailed breakdown of LL-HLS is beyond the scope of this article. However, you can find a structured deep-dive into the protocol in this Introduction to Low Latency Streaming with HLS.

When to use HLS Streaming

When delivering high-resolution videos with over 3MB: Without HLS, viewing such content usually leads to sub-par user experiences, especially when the user is on an average internet and/or mobile connection.
When broadcasting live video from one to millions to reach the broadest audience possible: Not only is HLS supported by most browsers and operating systems, but it also offers ABR, which allows content to be viewed at different network speeds (Cellular, 3G, 4G, LTE, WIFI Low, WIFI High).
When reducing overall costs: HLS reduces CDN costs by delivering the video at optimal bitrate to viewers.
When using advanced features in your stream: With HLS, you can leverage ad insertion, DRM, closed captions, adaptive bitrate, and much more.
When you expect your audience to use Apple devices: HLS enjoys support across devices, but Apple devices support HLS over MPEG-DASH and other alternatives. Additionally, apps on the App Store with over 10-minute videos are required to use HLS.
When you are concerned with security: HLS power video on demand with encryption (DRM), which helps reduce and avoid piracy.

Major Advantages of HLS

Transcoding for Adaptive Bitrate Streaming

We’ve already explained what ABR is in a previous section. Transcoding here refers to altering video content from one bitrate to another. In the above example, video segments are converted to 1080p, 720p, and 360p from a single, high-resolution stream.

In the HLS workflow, the video travels from the origin server to a streaming server with a transcoder. The transcoder creates multiple versions of each video segment at different bitrates. The video player picks which version works best with the end-user’s internet and delivers the video accordingly.

Delivery and Scaling

With HLS, it is much easier to broadcast live video from one to millions. This is because most browsers and OSes support HLS.

Since HLS can use web servers in a CDN to push media, the digital load is distributed among HTTP server networks. This makes it easy to cache audio-video chunks, which can be delivered to viewers across all locations. As long as they are close to a web server, they can receive video content.

Does 100ms support HLS for live streaming?

100ms supports live streaming output via HLS. However, we use WebRTC as input for the stream, unlike services that offer the infrastructure for live streaming alone, which generally use RTMP.

The 100ms live streaming stack combines streaming with conferencing, enabling our customers to build more engaging live streams that support multiple broadcasters, interactivity between broadcaster and viewer, and easy streaming from mobile devices. Since the entire audio/video SDK is packaged into one product, broadcasters can freely toggle between HLS streams and WebRTC, thus allowing two-way interaction while live streaming.

Introduction to Low Latency Streaming with HLS

Vrushti — Tue, 09 Aug 2022 12:19:01 +0000

Whether it’s a World Cup match, the Super Bowl, or the French Open finals, watching it with your friends on a Saturday night is #goals. Sadly, not all of us can get tickets and travel across cities, countries, or continents to attend them. Thankfully, live streaming makes it possible to watch all the action, close to real-time.

But, the only question is “how close to real-time are we talking?”

Video streaming is largely facilitated on the back of a video protocol called HLS (HTTP Live Streaming). While the origins and fundamentals of HLS are explained in another piece on our blog, the current piece will focus on how HLS resolved one of its greatest shortcomings: latency.

To start with, let’s take a quick peek at how HLS works.

The Way of the HLS

We will first try to understand how HLS works, and makes live streaming possible. This is what the typical flow of an HLS streaming system looks like:

The audio/video stream captured by input devices is encoded and ingested into a media server.

The media server transcodes the stream into an HLS-compatible format with multiple ABR variants and also creates a playlist file to be used by the video players.

Then, the media server serves the media and the playlist file to the clients, either directly or via CDNs by acting as an origin server.

The players, on the client end, make use of the playlist file to navigate through the video segments. These segments are typically “slices” of the video being generated, with a definite duration (called segment size, usually 2 to 6 seconds).

The playlist is refreshed based on segment size and players can select the segments specified in them, based on the order of playback and the video quality they require.

Even though HLS offers a reliable way of video streaming, its high latency levels may pose obstacles and issues for many streamers or video distributors. According to the initial specification, a player should load the media files in advance before playing it. This makes HLS an inherently higher latency protocol with a latency of about 30 to 60 seconds.

Tuning HLS for Low Latency

Everyone was interested in implementing HLS but the high latency was a serious roadblock. So, devs and enthusiasts started to find workarounds to reduce latency and refine the protocol for effective usage. Some of these practices offered such positive results that they started becoming a silent standard along with the HLS specification. Two of these practices are listed below:

Reducing the default segment size
When Apple introduced HLS, the typical segment size was 10 seconds. Most HLS implementers found it too long because of which Apple decided to reduce it to 6 seconds. The overall latency can reduced by reducing segment size and the buffer size of the player.

However, this carries some issues. Some of them include increased overall bitrate, buffering or jitter for devices with inferior network conditions. The ideal segment size should be decided based on the target audience and could be in the range of 2 to 4 seconds.

Media Ingest with faster protocols
The main reason HLS is used for live streaming is the scalability, reliability and player compatibility it provides across all platforms, especially when compared to other protocols. This has made HLS irreplaceable for video delivery so far.

But the first mile contribution (also known as ingest) from the HLS stack can be replaced with lower latency protocols to reduce overall latency.

The HLS ingest is usually replaced by RTMP ingest, which enjoys wide support for encoders/services and has proved to be a cost-effective solution. The stream ingested with RTMP is then transcoded to support HLS with the help of a media server before serving the content. Even though there have been experiments with other protocols such as WebRTC, SRT for the ingest part, RTMP remains the most popular option.

The Evolution of HLS to LL-HLS

The latency in HLS started posing a significant hurdle, leading to less than stellar user experiences. This was becoming more frequent since HLS was being widely adopted around the world. Tuning HLS wasn’t enough and everyone was looking for better and more sustainable solutions.

It was in 2016 that Twitter’s Periscope engineering team made some major changes to their implementation in order to achieve low latency with HLS. This proprietary version of HLS, often referred to as LHLS, offered latency of 2 to 5 seconds.

DASH, the main competitor for HLS came up with a low latency solution based on chunked CMAF in 2017, following which a community-based low latency HLS solution (L-HLS) was drafted in the year 2018. This variant was heavily inspired from the Periscope’s LHLS and leveraged Chunked Transfer Encoding (CTE) to reduce latency. This variant is often referred to Community Low Latency HLS (CL-HLS).

While this version of HLS was gaining popularity, Apple decided to release their own extension of the protocol called Low Latency HLS (LL-HLS) in 2019. This is often referred to as Apple Low Latency HLS (ALHLS). This version of HLS offered low latency comparable to the CL-HLS and promised compatibility with Apple devices. Since then, LL-HLS has been merged into the HLS specification and has technically become a single protocol.

How LL-HLS reduces Latency

In this section, we’ll explore the changes LL-HLS brings to HLS, making low latency streaming possible. This protocol came with 2 main changes in spec, responsible for its low latency nature. One is to divide the segments into parts and deliver them as soon as they’re available. The other is to inform the player about the data to be loaded next before said data is even available.

Dividing Segments into Parts
The video segments are further divided into parts (similar to chunks used in CMAF). These parts are just “smaller segments” with a definite duration — represented with EXT-X-PART tag in the media playlist.

The players can fill up their buffer more efficiently by publishing the parts while the segment is being generated. Reducing the buffer size on the player side using this approach, results in reduced latency. These parts are then collectively replaced with their respective segments upon completion, which will remain available for a longer period of time.

Preload Hints

When LL-HLS was first introduced, it had HTTP/2 push specified as a requirement on the server side for sending new data to clients. Many commercial CDN providers were not supporting this feature at the time, which resulted in a lot of confusion.

This issue was addressed by Apple in a subsequent update, replacing the HTTP/2 push with preload hints. They decided to include support for preload hints by adding a new tag EXT-X-PRELOAD-HINT to the playlist, reducing overhead.

With the help of preload hint, a video player can anticipate the data to be loaded next and can send a request to URI from the hint to gain faster access to the next part/data. The servers should block all requests for the preload hint data and return them as soon as the data becomes available, thus reducing latency.

A look at the LL-HLS Media Playlist
Now, let’s take a look at how these tags are specified in the media playlist file, using an example. We will assume the segment size to be 6 seconds and the part size to be 200 milliseconds. We will also assume that 2 segments (segment A and B) have been completely played, while the 3rd segment (segment C) is still being generated. This segment is being published as a list of parts in the order of playback because it has not yet been completed.

The following is a sample media playlist (M3U8 file).

Players that don’t support LL-HLS yet tend to ignore tags like EXT-X-PART and EXT-X-PRELOAD-HINT, enabling them to treat the playlist with the traditional HLS and load segments at a higher latency.

**Low-Latency HLS on non-Apple devices

**The new and improved HLS has a latency of about 3 seconds or less. The only reasonable competition for this protocol is LL-DASH. But Apple does not support DASH on all of its devices. This makes LL-HLS the only low latency live streaming protocol that has wide client-side support including Apple devices.

One of the main advantages of using LL-HLS is its backward compatibility with legacy HLS players. The players that don’t support this variant may fall back to standard HLS and still work with higher latency. Since this protocol required players to start loading unfinished media segments instead of waiting until they become fully available, the changes in the spec made it difficult to adapt it quickly for all players.

It took a while for most non-Apple devices to start supporting LL-HLS. Now, it is widely supported across almost all platforms with relatively newer versions of players. Even though some of them have been planning the support for the protocol since its inception, most of them are new and are improving their compatibility at the moment.

Here are some popular players from different platforms that support LL-HLS in its entirety:

AVPlayer (iOS)
Exoplayer (Android)
THEOPlayer
JWPlayer
HLS.js
VideoJS
AgnoPlay

Comparing LL-HLS, LL-DASH and WebRTC

Here, we compare three protocols LL-HLS, LL-DASH and WebRTC on six parameters: compatibility, delivery method, support for ABR, security, latency, best use case.

Compatibility
LL-HLS provides good support for all Apple devices and browsers. It has been gaining support for most non-Apple devices.
LL-DASH supports most non-Apple devices and browsers but is not supported on any Apple device.
WebRTC is supported across all popular browsers and platforms.

Delivery Method

First, let’s go through a few relevant terms used with CMAF.

Chunked Encoding (CE) is a technique used for making publishable “chunks”. When added together, these chunks create a video segment. Chunks have a set duration and are the smallest unit that can be published.

Chunked Transfer Encoding (CTE) is a technique used to deliver the “chunks” as they are created in a sequential order. With CTE, one request for a segment is enough to receive all its chunks. The transmission ends once a zero-length chunk is sent. This method allows even small chunks to be used for transfer.

LL-HLS uses Chunked Encoding to create “parts” or “chunks” of a segment. But, instead of using Chunked Transfer Encoding, this protocol uses its own method of delivering chunks over TCP. The client has to make a request for every single part, instead of just requesting the whole segment and receiving it in parts.
LL-DASH uses Chunked Encoding for creating chunks and Chunked Transfer Encoding for delivering them over TCP.
WebRTC uses Real-time Transfer Protocol (RTP) for sending video and audio streams over UDP.

Support for Adaptive Bitrate (ABR)

Adaptive Bitrate (ABR) is a technique for dynamically adjusting the compression level and video quality of a stream to match bandwidth availability. It heavily impacts the video streaming experience for the viewer.

LL-HLS has support for ABR.
LL-DASH has support for ABR.
WebRTC doesn’t support ABR. But, a similar technique called Simulcast is used for dynamically adjusting video quality.

Security
Both LL-HLS and LL-DASH support media encryption and benefit from security features such as token authentication and digital rights management (DRM).

WebRTC supports end-to-end encryption of media for transfer, user, file, and round-trip authentication. This is often sufficient for DRM purposes.

Latency
Both LL-HLS and LL-DASH have a latency of 2 to 5 seconds.

WebRTC, on the other hand, has a sub-second latency of ~500 milliseconds.

Best Use Case
Both LL-HLS and LL-DASH are best suited for live streaming events that need to be delivered to millions of viewers. They are often used for streaming sporting events live.

WebRTC is very frequently used for solutions such as video conferencing that require minimal latency and are not expected to scale to a big number.

Now that the HLS supports low latency streaming, it is all set to conquer the video streaming space, ready to serve millions of fans watching their favourite team play a crucial match without any issues. Whether you want to start live streaming yourself or build an app that facilitates live streaming, LL-HLS remains your best friend.

Roles on 100ms: Mapping real-world interactions to live video with a few clicks

Vrushti — Tue, 09 Aug 2022 07:02:00 +0000

On this world's stage, we all have a role to play. Turns out, the same is true for the world of live, interactive video applications.

If you're building a live video app, a meeting room will contain people who will need to perform different functions. For example, in a virtual classroom, the teacher is able to display their video and audio, as well as share their screen. Depending on the app, they can also display a student's screen to the rest of the class, allow the student to address the class, and more.

All this while, the student may only be able to watch and listen to the teacher's video. They may only be able to share their screen or speak when the teacher allows them to, reducing interruptions or chaos in an ongoing class - especially if the class is large.

However, building these permissions (what participants can and cannot do within meetings) are difficult because they usually require some coding and implementation effort to set up. The more nuanced and varied the permissions in an app, the more effort devs have to expend to ensure that the final app offers the exact features required by end-users.

At 100ms, we call these permissions "roles". The teacher's role (in the above example) is to share audio/video, share their screen, allow students to ask questions or address the class, and more. The student's is to view the teacher's video, listen to their audio and perhaps share their screen, ask questions, and speak to the class - when the teacher enables them to.

This article will delve into what these "roles" are, how they make your life easier, and how they are an improvement upon the industry standard "publish/subscribe" logic.

But, let's start with the obvious question.

What is a role?

Before answering this question, we want to lay out 100ms' mission: bring the world closer by enabling real, life-like, live conversations virtually. We want our customers to be able to offer their users an online interactive experience that is as close to the real-world as possible.

This is where roles comes in. They allow the easy recreation of real-life interactions on video, as this article will demonstrate with an example. In 100ms terminology, a role is a collection of permissions that allow users to perform certain tasks while being part of the meeting room. Essentially, the role determines whether a user in the room has publish/subscribe permissions. It determines whether they can speak, mute other users, be muted, share their screen, etc.

Before moving on, let us understand roles better by diving into the composing features:

Publishing rights: The term "publish" here refers to a user's ability to share audio, video and if needed, their screen, when in a video call. The user's particular role decides whether they can share their audio/video/screen to the room.

Subscribing rights: The term "subscribe" here refers to a user's ability to view and listen to the video and audio being shared ("published") by others in the room. Depending on the use-case, they may only be able to subscribe to one person's (host) audio-video or to multiple users' streams.

Permissions and Power: Developers and specific users of a video app should be able to configure and manipulate roles (others' and their own). This would let them perform actions such as muting/unmuting others in the room, letting them share their screen or expelling them from the meeting altogether.

How Roles Work: An Example

Let’s take a closer look at roles through a simple example: a virtual events app for online concerts. The real-world experience this app is trying to replicate is denoted by the image below:

Built with 100ms, this hypothetical app has three roles in place: “stage” (where the artist performs), “audience” (where the viewers watch the performance online) and “backstage” (where the person/people handling tech/logistics keep the show going as expected).

And, these roles will have the following permissions:

Those in the “stage” role can do the following: sing (publish their audio-video), invite audience members onto the stage to interact with. Those in the “audience” role can only view the artist’s stream, unless they have been invited to the “stage”. Those in the backstage can interact with others in the same role, kick out audience members if required, change lights in the artist’s video and of course, view the artist on “stage”.

A real-world app would have more roles and each role would have more permissions. But this is what roles fundamentally do.

Now, be it Meet, Twitch, or your cousin's school app, the concept of roles exist within all video SDKs. For example, these are the default roles we see in a typical webinar setup: a host (sometimes more than one depending on the app customer) and multiple participants. The host publishes their audio-video streams, the participants subscribe to the host, and sometimes other participants. The host can share their screen, and so can the participants but only if allowed by the host.

However, most real-world scenarios need more than two roles to recreate real-world experiences. If you wanted more roles with varied permissions, you’d have to configure the same by coding it from scratch.

So, what’s the solution? How do we make the process of creating varied roles easier?

The answer: custom roles on 100ms.

Why are custom roles required?

As mentioned above, if you want to give users a nuanced, layered experience that matches their day-to-day life, two or three simple roles are not enough. Let’s take another example for this: a video consultation app for doctors and patients.

In the real world, patients are greeted by a nurse/receptionist who takes their information, they wait in the waiting room, and when the doctor is ready, they are called into the consultation room. With roles limited to host and audience, you can’t do this virtually. You would have to create custom roles for “waiting room”, “nurse” and the like, which is usually time, effort and resource-intensive.

However, using 100ms, you can create custom roles with far greater ease. Owing to our built-in customizability and extensibility, you can actually build such a consultation app with zero lines of code. This is demonstrated step-by-step below.

Custom roles in action

In the aforementioned video consultation app, we can customize a user’s journey by modifying who they are publishing to and subscribing to, i.e. whose audio-video they can watch/listen to, and vice-versa.

Here’s what the patient’s journey looks like: when a patient enters, they initially communicate with a nurse in a virtual “waiting room” where their information (name, gender, DOB, temperature, weight, symptoms) are noted by the nurse. After this, they wait until the doctor is able to communicate with them.

In the language of roles, they will initially publish and subscribe to someone in the “nurse” role. Then, their role will be changed so that they publish and subscribe to someone in the “doctor” role.

Using 100ms, devs can map out this exact journey without writing a single line of code—just a few clicks on the dashboard.

To demonstrate roles in action, let’s put them to a test and visualize a user’s journey to the online “clinic”.

Using Roles to recreate a Patient’s Consultation Experience Online

Go to the 100ms dashboard by creating a new account for free.
Click on Create New. There are multiple templates to choose from, depending on your use case. For a simple video conferencing app, select the ‘video conferencing’ template. But we’ll go with something else for our app.

Select the ‘Create Custom App’ option.

This will let us access the ‘Create Roles’ option so that we can customize roles for our use case.

For a clinic app, we require 4 roles: consultation-admin, consultation-area, reception-admin, reception-area (details of each role explained below). Click ‘Add a Role’ and name them accordingly.

As soon as a new person enters the clinic, they’ll be assigned the “reception-area” role. When the doctor is ready, the role will be changed to “consultation-area”.

Now, we have created the four roles we require: consultation-admin, consultation-area, reception-admin, and reception-area.

Handling Permissions for each Role

100ms enables users to quickly and easily modify the permissions of different roles and subscription strategies, right from the dashboard.

In the example, here’s what each role entails:

reception-area: The role assigned to the patient when they first enter the online clinic. This role can publish and subscribe only to the “reception-admin” role.
reception-admin: The role assigned to the “nurse” who greets the patient and takes their info. This role can be published and subscribed to the other three roles. They can also change roles from “reception-area” to “consultation-area”. If needed, they can remove the person in the “reception-area” role from the meeting room entirely.
consultation-area: The role assigned to the patient when the doctor is ready to see them. The nurse in the “reception-admin” role changes the patient’s role from “reception-area” to “consultation-area” when the doctor is ready.
consultation-admin: The role assigned to the doctor. This role can publish and subscribe to the “consultation-area” and “reception-admin” roles. If needed, they can expel the person in the “consultation-area” role completely.

On the 100ms dashboard, we can assign mute/unmute, screenshare, publish/subscribe permissions to each of these role with a couple of clicks. We can even give certain roles the ability to change other roles or every remove somebody from the meeting room entirely.

To go back to our example, we start by modifying the nurse role.

As mentioned, when a person walks into a clinic they will automatically be assigned the “reception-area” role. There, someone in the “reception-admin” role greets them.
The reception-admin role should be subscribed to the person in the “reception-area” role, and also have permissions to modify the “reception-area” roles, or even remove them if necessary.

A person in the “reception-area” role should be subscribed to the “reception-admin” role, but they will not have permission to modify their roles or add/remove them.

This will now serve as a waiting room/reception area where the nurse connects with the incoming patients.
Now, we assign permissions to the “consultation-admin” role. Since the “consultation-admin” can call in a patient, they need to have administrative permissions to modify user roles, mute/unmute them, share their own screen, and the like.

The “consultation-admin” will also be subscribed to the “consultation-area” role.

Lastly, we modify the “consultation-area” role. They will be subscribed to the “consultation-admin” role to enable consultation.

With that done, we have successfully modified user permissions for our use case. Now, we implement the app using nothing but the power of roles and customization provided on the 100ms dashboard.

And just a note, we have been able to do all this without writing a single line of code!
The last step now is to pick a domain. Let’s go ahead with “hospital.app.100ms.live” as the subdomain and click on ‘Set up app’.

100ms enables you to have a completely personalized subdomain for your app—for free. You can easily host these powerful video templates on your own domain URL with a click.

The app is now ready to use.

Testing the App

With that done, we are ready to test out the app.

Here’s what our newly set up telehealth app looks like:

https://youtu.be/webB4efxxyg

That’s it! 🚀

We have successfully implemented a clinic-like experience digitally using nothing but roles on the 100ms dashboard.

This is just the start.

With custom roles at your fingertips, the applications are limitless. Here are a few quick examples of virtual scenarios you can easily build with roles:

A digital classroom where a teacher will admit students when required. - The same waiting roles, as depicted above can be used there.
Online performances and events. It would be a simple task to create roles for “backstage”, “stage”, and “audience”—also exemplified above.
Online games: create roles for “dealer”, “player” and “spectator” for poker, for example.
Virtual interviews
Celebrity fanmeets … and so much more.

Roles are the Silver Bullet in your App-Building Arsenal
As mentioned before, 100ms seeks to enable easier, more human communication by allowing customers to create interactive video apps that match our regular interactions as closely as possible. This is the whole point of the 100ms Marketplace.

Our customers don’t have to worry about how to set up permissions for users of their apps. They don’t have to work on the basics: coding permissions, publish, and subscribe strategies for specific scenarios. They only have to imagine and conceptualize how user journeys will work, and using roles, developers can set them up with a few clicks—no coding involved.

In fact, our customers have already achieved this. Have a look at how Mingout, a dating app used roles on 100ms to reimage first dates online.

For the developers out there, here is a closer, more dev-centric dive into how roles work, have a look at this video on Building a Clubhouse clone from scratch using React. It starts off by examining the 100ms SDK, and demonstrating how roles ease the process of app building.

If you’re curious, try it out for yourself. Get Started with 100ms for free, and play around with roles to bring an imagined app to life with just a few clicks!

Introduction to Low Latency Streaming with HLS

Vrushti — Fri, 05 Aug 2022 10:28:08 +0000

But, the only question is “how close to real-time are we talking?”

To start with, let’s take a quick peek at how HLS works.

The Way of the HLS

We will first try to understand how HLS works, and makes live streaming possible. This is what the typical flow of an HLS streaming system looks like:

The audio/video stream captured by input devices is encoded and ingested into a media server.
The media server transcodes the stream into an HLS-compatible format with multiple ABR variants and also creates a playlist file to be used by the video players.
Then, the media server serves the media and the playlist file to the clients, either directly or via CDNs by acting as an origin server.
The players, on the client end, make use of the playlist file to navigate through the video segments. These segments are typically “slices” of the video being generated, with a definite duration (called segment size, usually 2 to 6 seconds).
The playlist is refreshed based on segment size and players can select the segments specified in them, based on the order of playback and the video quality they require.

Even though HLS offers a reliable way of video streaming, its high latency levels may pose obstacles and issues for many streamers or video distributors. According to the initial specification, a player should load the media files in advance before playing it. This makes HLS an inherently higher latency protocol with a latency of about 30 to 60 seconds.

Tuning HLS for Low Latency

But the first mile contribution (also known as ingest) from the HLS stack can be replaced with lower latency protocols to reduce overall latency.

The Evolution of HLS to LL-HLS

How LL-HLS reduces Latency

Dividing Segments into Parts
The video segments are further divided into parts (similar to chunks used in CMAF). These parts are just “smaller segments” with a definite duration - represented with EXT-X-PART tag in the media playlist.

Preload Hints
When LL-HLS was first introduced, it had HTTP/2 push specified as a requirement on the server side for sending new data to clients. Many commercial CDN providers were not supporting this feature at the time, which resulted in a lot of confusion.

The following is a sample media playlist (M3U8 file).

Players that don’t support LL-HLS yet tend to ignore tags like EXT-X-PART and EXT-X-PRELOAD-HINT, enabling them to treat the playlist with the traditional HLS and load segments at a higher latency.

Low-Latency HLS on non-Apple devices

The new and improved HLS has a latency of about 3 seconds or less. The only reasonable competition for this protocol is LL-DASH. But Apple does not support DASH on all of its devices. This makes LL-HLS the only low latency live streaming protocol that has wide client-side support including Apple devices.

Here are some popular players from different platforms that support LL-HLS in its entirety:

AVPlayer (iOS)
Exoplayer (Android)
THEOPlayer
JWPlayer
HLS.js
VideoJS
AgnoPlay

Comparing LL-HLS, LL-DASH and WebRTC
Here, we compare three protocols LL-HLS, LL-DASH and WebRTC on six parameters: compatibility, delivery method, support for ABR, security, latency, best use case.

Compatibility

LL-HLS provides good support for all Apple devices and browsers. It has been gaining support for most non-Apple devices.
LL-DASH supports most non-Apple devices and browsers but is not supported on any Apple device.
WebRTC is supported across all popular browsers and platforms.

Delivery Method
First, let’s go through a few relevant terms used with CMAF.

Chunked Encoding (CE) is a technique used for making publishable “chunks”. When added together, these chunks create a video segment. Chunks have a set duration and are the smallest unit that can be published.

Chunked Transfer Encoding (CTE) is a technique used to deliver the “chunks” as they are created in a sequential order. With CTE, one request for a segment is enough to receive all its chunks. The transmission ends once a zero-length chunk is sent. This method allows even small chunks to be used for transfer.

-LL-HLS uses Chunked Encoding to create “parts” or “chunks” of a segment. But, instead of using Chunked Transfer Encoding, this protocol uses its own method of delivering chunks over TCP. The client has to make a request for every single part, instead of just requesting the whole segment and receiving it in parts.

LL-DASH uses Chunked Encoding for creating chunks and Chunked Transfer Encoding for delivering them over TCP.
WebRTC uses Real-time Transfer Protocol (RTP) for sending video and audio streams over UDP.

Support for Adaptive Bitrate (ABR)
Adaptive Bitrate (ABR) is a technique for dynamically adjusting the compression level and video quality of a stream to match bandwidth availability. It heavily impacts the video streaming experience for the viewer.

LL-HLS has support for ABR.
LL-DASH has support for ABR.
WebRTC doesn’t support ABR. But, a similar technique called Simulcast is used for dynamically adjusting video quality.

Security
Both LL-HLS and LL-DASH support media encryption and benefit from security features such as token authentication and digital rights management (DRM).

WebRTC supports end-to-end encryption of media for transfer, user, file, and round-trip authentication. This is often sufficient for DRM purposes.

Latency
Both LL-HLS and LL-DASH have a latency of 2 to 5 seconds.

WebRTC, on the other hand, has a sub second latency of ~500 milliseconds.

Best Use Case

Both LL-HLS and LL-DASH are best suited for live streaming events that need to be delivered to millions of viewers. They are often used for streaming sporting events live.

WebRTC is very frequently used for solutions such as video conferencing that require minimal latency and are not expected to scale to a big number.

Agora vs Twilio vs Jitsi vs Zoom vs 100ms

Pro100ms — Thu, 05 May 2022 15:52:27 +0000

Unsurprisingly, the Covid-19 pandemic revealed exactly how useful audio-video conferencing software could be. Having to shelter in place and work remotely, people worldwide had to manage everything from doctor’s appointments, fitness classes, dating to Friday night drinks, and watching live concerts on video apps.

Naturally, video conferencing software has seen massive growth in usage during the pandemic. There was a 627% increase in downloads of video chat and online conference apps in North America. A 121% increase in daily active users was also observed. The explosion of Zoom and Google Meet is common knowledge by now. Even Skype for Business, GoToMeeting, and JoinMe apps saw downloads increase by 66%, 85%, and 43% in March 2020.

App Categories with Significant Growth

As it stands, users expect certain apps to have in-built audio-video communicative capabilities. The numbers attest to this. Video conferencing improves communication for 99% of people. Video meetings boost productivity by 50%. By every metric, video conferencing capabilities improve user experience.

However, embedding audio-video communication into an app from scratch requires time, effort, and investment. Instead, it's much easier to implement audio-video using an SDK designed for that purpose.

Since there are multiple products and SDKs aimed at helping devs build comprehensive video conferencing into their software, choosing the right platform is more complicated than it seems.

To help devs and product managers make a more informed decision, this article will break down the main features and limitations of five tools that provide the infrastructure for embedding in-app audio-video functionality: Agora, Twilio, Jitsi, Zoom, and 100ms.

Note: This piece will compare tools on six parameters: ease of integration, error handling, scalability, cost of support, plugins for easy feature development, and pricing.

Agora: Real-Time Engagement for Apps

What is Agora? What are its main offerings?

Agora is an API-first SaaS company that started by providing real-time audio and video broadcast APIs. Now, it has expanded to a platform that allows customers to create rich, in-app audio-video features, such as real-time recording and messaging, embedded video, and video chat as well as interactive live video streaming.

Main features:

Cross-platform SDKs that are highly customizable.
Using low code UIKit libraries, users can embed a real-time video UI with a few lines of code.
Agora uses its own Software Defined Real-time Network (SD-RTN™), a real-time transmission network. Unlike a traditional carrier network, the SD-RTN™ is not limited by device type, phone numbers, or a network provider’s coverage radius.
Multiple extensions (interactive whiteboard, cloud recording, Agora analytics) enable the easy addition of new and useful features to an app.
Offers official SDKs for React Native, Electron, Unity, Cocos, and Flutter.

Agora’s Key Functionality Metrics

1. Ease of integration: Agora offers both pre-built and custom SDKs. The pre-built version can be installed and activated in a few lines of code but is not customizable and extensible. It comes with two pre-defined permissions (roles) for peers within a call: host and participant.

Developers have to handle low-level publish-subscribe abstractions. This adds overhead in handling network exceptions, bandwidth management, and writing role-based infrastructure. Developers will also have to manually configure the permissions for different roles in the call (teacher vs. student, instructor vs. learner, etc.)

2. Error Handling: Agora does not support in-built disconnection handling and edge cases on devices like app background handling, switching of microphones, etc. Devs must write extra code to set it up.

3. Scalability: Within a single call, Agora allows a maximum of 17 hosts and a total of 10,000 participants, including the hosts.

4. Cost of Support: Agora’s free Starter plan offers Tickets/Email support, Online Documentation, and KB Access. Services like code review, guaranteed response times, live developer consultation, and training must be paid for (the lowest price being $1200/month and the highest being $4900/month).

5. Plugins for easy feature development: Agora offers multiple plugins for the easier development of feature-rich apps. Users can access Agora’s plugin marketplace for a large number of integrations with various functions. However, adding all plugins requires extra coding effort.

6. Pricing: Starts at $4, but Agora’s pricing policy is quite layered and will require close examination before purchase.

Agora’s Limitations

Pricing for Agora goes up exponentially as the number of video participants increases. Anytime the download quality on a call exceeds 720p (which can happen by screen-sharing between two peers), charges switch to HD prices. Agora’s pricing plans can also be complicated for users.
You cannot have more than 17 hosts in a meeting.
Integrating the SDK is quite complicated and time-consuming.

Twilio: Meaningful In-App Customer Engagement

What is Twilio? What are its main offerings?

Twilio originally offered an API for automating traditional phone calls and SMS text messages. Today, it provides programmable APIs to help developers build business communication (in-app and otherwise) across the customer journey. They allow devs to integrate audio and video interactions into multiple platforms.

Once the Twilio API has been integrated, interactions can take the form of SMS, WhatsApp, Voice, Video, email, and even IoT.

Main Features:

Communications APIs to implement messaging, voice chats, and video conversations within the software or outside its UI (SMS, WhatsApp messages, etc.)
Programmable connectivity features (Chat, Voice API, Video) for generating virtual phone numbers, initiating SIP trunking, and messaging.
Use case-based APIs that allow the abstraction required for tasks related to authentication, message control, and call routing.

Twilio’s Key Functionality Metrics

1. Ease of integration: Twilio provides web, iOS, and Android SDKs. The SDK does not support other frameworks like Flutter and React Native. When using multiple audio and video inputs, devs must manually configure them, for which they have to write extra code.

Much like Agora, devs have to handle low-level publish-subscribe abstractions to set up Twilio integrations. They must also manually define permissions for all actors within a call.

2. Error Handling: Manual configuration is required to build bandwidth management. Twilio offers extensive call insights to track and analyze errors.

3. Scalability: Twilio supports a maximum of 50 hosts within a call and a maximum of 50 participants, including hosts. You can switch to Twilio Live for HLS streaming and accommodate unlimited participants in a call. But, separate SDK integration is required to stream video via HLS.

4. Cost of Support: Twilio’s free support plan offers API status notifications and email support during business hours. Users have to pay for more services like 24/7 live chat support, support escalation line, quarterly status review, and guaranteed response times. The price depends on the support plan, usually a percentage of the monthly plan or a certain minimum amount (lowest being $250/month and highest being $5000/month).

5. Plugins for easy feature development: No plugins are available.

6. Pricing: Starts at $4, and has a relatively simple pricing policy.

Twilio’s Limitations

SDKs are available only for web, iOS, and Android.
It is not possible to stream video via RTMP.
Devs must manually compose all recordings.
Number of participants is limited to 50.

Jitsi: Open-Source Video Integration

What is Jitsi? What are its main offerings?

Jitsi is a collection of open-source projects designed to help users build and implement secure video conferencing options. Among its offerings, Jitsi Meet is best known for providing video conferencing services. Jitsi also comes with meet.jit.si which hosts a free-for-use Jitsi Meet instance and the Jitsi Videobridge which powers and sustains Jitsi’s multi-peer video features.

Main Features:

Jitsi is an open-source solution with impressive community support.
Setup is relatively easy and comes with one-click installation.
The process to set up video/audio calls and multi-user meeting rooms is quite user-friendly.
Jitsi uses industry-standard physical, administrative, and technical shields to safeguard the confidentiality of users' personal data.
Jitsi users have a decent choice of service providers across multiple geographies to choose from, in order to host the application locally.
Jitsi supports all available clients (Windows, Linux, Mac, iOS, Android).

Jitsi’s Key Functionality Metrics

1. Ease of integration: Offers both pre-built and custom SDKs. Like Twilio, there are no predefined roles. Devs have to manually configure permissions for peers within a call.

2. Error Handling: Manual configuration is required to build bandwidth and connection management into Jitsi's low-level API. Jitsi also has SDKs for less customizable versions with some in-built connection management.

3. Scalability: With the open-source SDK*,* Jitsi supports a maximum of 100 hosts in a call, and a maximum of 100 participants, including the hosts. The paid Jitsi SDK from 8x8 supports a maximum of 500 participants (including hosts) in a call.

4. Cost of Support: Users can ask questions to the Jitsi community. For paid support, they will have to approach 8x8, the company that acquired Jitsi in 2018.

5. Plugins for easy feature development: Since Jitsi is fully DIY, plugins may not be available for most features. However, there are some open-source plugins devs can use.

6. Pricing: Largely free of charge for the open-source SDK. However, some costs will be involved for the deployment infrastructure. Users can expect that to be 40-50% of the cost of other paid providers.

For the paid SDK from 8x8, prices are based on the number of active monthly active users. For example, JaaS (Jitsi as a Service) Dev supporting 25 active monthly users is free, JaaS Basics supporting 300 users is $99/month, JaaS Standard supporting 1500 users is $499/month, and so on.

Jitsi’s Limitations

Given its open-source nature, video apps with Jitsi have to be built from scratch. Therefore, it takes a significant amount of time to build a stable product.

Zoom: Quick and Easy Video Conferencing

What is Zoom? What are its main offerings?

Zoom is a cloud-based video conferencing app that offers SDKs for devs to implement audio-video communications and relevant capabilities within new or existing apps. Zoom SDKs are split into the Meeting SDK and the Video SDK.

Zoom SDKs allow for the setup and integration of numerous features such as live chat, webinars, screen sharing, changing background, and multiple collaborative functions. Zoom Meeting SDKs are a solid option for anyone looking to include a slew of video communication features into their software ecosystem.

Main features:

Zoom users can simply import libraries and packages for quick implementation of the Zoom meeting platform into applications.
Zoom supports seven major languages and provides open translation extensibility which opens any app to international growth and improved user experience.
The Zoom Video SDK comes with fully customizable UI features that developers can expand and modify according to the requirements of their app.

Zoom’s Key Functionality Metrics

1. Ease of integration: Zoom offers two predefined roles: host and participants. Permissions for these roles cannot be modified.

2. Error Handling: Zoom SDKs come with in-built error handling and bandwidth management, as built into their consumer offering. Developers have to handle only minimal reconnection/bandwidth management.

3. Scalability: Zoom supports a maximum of 300 hosts in a call and a maximum of 1000 participants including the hosts.

4. Cost of Support: Zoom offers three customer support plans: Access, Premier, and Premier+. Pricing for these plans has to be obtained by contacting Zoom.

5. Plugins for easy feature development: Currently, Zoom does not have a plugin marketplace for devs to use.

6. Pricing: Base pricing starts at $3.99. Zoom has a fairly straightforward, uncomplicated pricing policy.

Zoom’s Limitations

Zoom is best suited for basic use cases, as it only allows the use of predetermined roles: host and participant. For use cases that require modified permissions for peers, Zoom may pose difficulties.
The SDK’s footprint size is inordinately large.

100ms: Pre-built, low-code audio-video templates

What is 100ms? What are its main offerings?

100ms provides live audio-video SDKs that enable devs to add powerful, extensible, scalable, and resilient audio-video features into their apps with half a dozen lines of code. It abstracts the business logic of the conference room in Templates and Roles. The client-side SDKs include all edge cases within the SDK rather than leaving it to the application side.

The solution has been built by the team that powered live video infrastructure for some of the world’s largest live events & running billions of minutes a day at Facebook and Disney+Hotstar.

Main Features:

The 100ms SDK provides pre-built templates to build virtual events, audio rooms, classrooms, and a wide range of use-cases with a few lines of code.
The SDK is comprehensive, meaning that any piece of code which multiple application developers have to write repeatedly exists in the infrastructure layer.
By virtue of room templates, the business logic remains on the server-side rather than being burned into client apps.
The SDK is fully customizable and designed to be modified as required by any app’s UI.

100ms’ Key Functionality Metrics

1. Ease of integration: 100ms SDKs are fully customizable. The publish-subscribe logic has been abstracted in the concept of roles. Pre-defined roles are available, and users can customize roles as required with zero coding - right on the 100ms dashboard.

Devs can use roles to build complex video applications with minimal coding. All the code resides on the SDK side. Write a few lines of code on the application side and go live.

2. Error Handling: Pre-built disconnection handling is available in 100ms. Devs also benefit from edge case handling like granular device capture errors, in-built network degradation handling, automatic lowest latency server choice, and more.

3. Scalability: 100ms supports up to 10,000 participants and up to 100 hosts on regular video calls. It provides a single switch between WebRTC and HLS, which means that video calls can be streamed to millions via HLS with one click.

4. Cost of Support: No extra cost is charged for support. Customers can access support via private Slack channels as well as Discord.

5. Plugins for easy feature development: With the 100ms SDK, the most relevant features like whiteboard, hand-raising, media player, chat, and screen share are provided out of the box. Customers also get RTMP Streaming, HLS, and recording capabilities out of the box. Therefore, too many plugins may not be necessary.

6. Pricing: Base pricing starts at $4. Simple, easy-to-understand pricing policy.

100ms’ Limitations

No simulcast available
No hosting servers in ANZ and South America.

A Comprehensive Guide to Flutter-WebRTC

Nilay Jayswal — Mon, 21 Mar 2022 12:57:35 +0000

So, you want to establish real-time audio and video in your Flutter app. This is a common requirement for app developers, especially post-pandemic, since everyone wants to interact online almost as easily as they do in real life.

This content was originally published - HERE

One of the most effective ways to go about this is to use WebRTC.

This article will demonstrate how to use WebRTC and implement real-time audio-video communication in a Flutter app.

Note: This piece assumes that you are already familiar with app development in Flutter and will focus on WebRTC integration with a Flutter app.

How to build a Flutter WebRTC app

We’ll start by building a new Flutter project and then go deeper into how WebRTC works. But first, let’s take a moment to answer the question: “What is WebRTC?”

WebRTC is an HTML specification that enables real-time, audio-video communication between websites and devices. It comprises networking, audio, and video components standardized by the Internet Engineering Task Force and the World Wide Web Consortium.

Simply put, WebRTC is an open-source project that allows direct P2P communication without installing additional programs or plugins. It can be used on all modern browsers to establish peer-to-peer communication and can also be embedded into native applications using available libraries.

We’ll discuss WebRTC in more detail later in the article. For now, let’s start building a Flutter WebRTC app.

First of all, let’s create a new Flutter project

flutter create webrtc_flutter

We’ll refactor some code and get rid of the comments.

The first step is to add the Flutter WebRTC plugin based on Google WebRTC. Using this, we can develop the app in Flutter for mobile, desktop, and the web.

Run the following code in the terminal to add flutter_webrtc as a dependency in your pubspec.yaml file:

flutter pub add flutter_webrtc

Access Camera and Microphone on iOS

Add the following permission entry to your Info.plist file, located in <project root>/ios/Runner/Info.plist:

<key>NSCameraUsageDescription</key>
<string>$(PRODUCT_NAME) Camera Usage!</string>
<key>NSMicrophoneUsageDescription</key>
<string>$(PRODUCT_NAME) Microphone Usage!</string>

This entry allows your app to access the device’s camera and microphone.

Android Manifest File Changes

To enable user permissions on Android, add the following to the Android Manifest file, located in <project root>/android/app/src/main/AndroidManifest.xml:

<uses-feature android:name="android.hardware.camera" />
<uses-feature android:name="android.hardware.camera.autofocus" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.CHANGE_NETWORK_STATE" />
<uses-permission android:name="android.permission.MODIFY_AUDIO_SETTINGS" />

Add the following if you wish to use a Bluetooth device:

<uses-permission android:name="android.permission.BLUETOOTH" />
<uses-permission android:name="android.permission.BLUETOOTH_ADMIN" />

build.gradle Changes

Currently, the official WebRTC jar uses static methods in the EglBase interface. So, you will need to change your build settings to Java 8. To do so, add the following code to app-level build.gradle:

android {
    //...
    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
}

If necessary, in the same build.gradle, you may need to increase the minSdkVersion of defaultConfig to 23 (currently, the default Flutter generator has set it to 16).

Our initial objective is to show the local user's video in the app. We’ll start from there and go on to connect to a remote user and establish a P2P connection using WebRTC.

Let’s start by writing the Dart code.

Rendering a Local User

Inside the stateful MyHomePage() widget, we’ll initialize a localVideoRenderer for the same.

The RTCVideoRenderer lets us play video frames obtained from the WebRTC video track. Depending on the video track source, it can play videos from a local peer or a remote one.

final _localVideoRenderer = RTCVideoRenderer();

  void initRenderers() async {
    await _localVideoRenderer.initialize();
  }

The initRenderers function should now be called in the initState() of the stateful widget.

@override
  void initState() {
    initRenderers();
    super.initState();
  }

It also needs to be disposed of. Disposing of the renderer stops the video and releases the resources associated with it.

@override
  void dispose() async {
    await _localVideoRenderer.dispose();
    super.dispose();
  }

The next step is to get the user media as a stream.

The getUserMedia() function will prompt the user to allow an input device to be used. The said device will generate a Media Stream with the requested media types. This stream will have an audio track generated from an audio source such as a microphone (or others) and a video track from a camera, recording device, etc.

Moving on, we create a new function named getUserMedia() and call it in initState() as shown below:

_getUserMedia() async {
    final Map<String, dynamic> mediaConstraints = {
      'audio': true,
      'video': {
        'facingMode': 'user',
      }
    };

    MediaStream stream =
        await navigator.mediaDevices.getUserMedia(mediaConstraints);
    _localVideoRenderer.srcObject = stream;
  }

  @override
  void initState() {
    initRenderers();
    _getUserMedia();
    super.initState();
  }

The last step is to complete our initial objective: use this local renderer in the UI to display the user’s video.

Let us modify the build method of the stateful widget as:

@override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: Text(widget.title),
      ),
      body: Stack(
        children: [
          Positioned(
              top: 0.0,
              right: 0.0,
              left: 0.0,
              bottom: 0.0,
              child: RTCVideoView(_localVideoRenderer))
        ],
      ),
    );
  }

With that done, we can now run the app to check if the local video is being rendered.

Flutter WebRTC demo

First of all, the app will ask for permission when we first run it.

Once loading is complete, the following screen should show up:

And now, you have successfully rendered the local video of the user on-screen. The next step is to render a remote user. But first, we need to understand connection and communication using WebRTC.

How WebRTC works

WebRTC allows peer-to-peer communication over websites, apps, and devices, even though a peer may initially have no idea where the other peers are or how to connect or communicate with them.

To establish a connection, WebRTC needs clients to exchange metadata in order to coordinate with them - which is done by a process called signaling. Signaling is also required to bypass firewalls and work with network address translators (NATs).

Let’s dive a little further into the process.

NAT (Network Address Translation)

All devices have a unique IP address. WebRTC uses this unique address to connect two peers.

Due to the large number of connected devices in the modern world, unallocated IPv4 addresses are depleting. This has led to the deployment of a successor protocol, IPv6. But, because of classful network addressing to Classless Inter-Domain Routing, the exhaustion of these IPv4 addresses was substantially delayed.

In addition, NAT or Network Address Translation helps with this. Instead of allocating public addresses to each network device, it allows Internet service providers and enterprises to masquerade as private network address space. Hence they have only one publicly routable IPv4 address on the Internet interface of the main Internet router.

Because of NAT, we have two IP addresses. One is public, associated with the router, visible only from the outside. The other is a private address visible only to those connected to the same router.

NAT mapping is what makes the connectivity of WebRTC possible. WebRTC leverages it to allow two peers in completely different subnets to communicate.

Interactive Connectivity Establishment (ICE)

Now, it is not possible to connect two peers with just their IP addresses. To establish a connection, we’d have to first pass in the unique address to locate the peer on the internet. We’d also need a relay server for sharing media (if P2P connection is not allowed by the router), and finally, we’d have to get through the firewall.

A framework called Interactive Connectivity Establishment (ICE) is used to solve this.

ICE is used to find the optimum path for connecting the peers. It examines the different ways to do so and chooses the best one.

About the Flutter WebRTC Server

The ICE protocol tries to connect using the host address obtained from a device’s OS and network card. If that fails (which it will for devices behind NATs), ICE tries to get an external address using a STUN server. If that fails, it uses a TURN relay server to route traffic. Fundamentally, a TURN server is a STUN server with additional built-in relaying functionality.

But, let’s look a bit further into them.

STUN (Session Traversal Utilities for NAT)

An endpoint under a NAT with a local address is not reachable for other endpoints outside the local network. Hence a connection cannot be established. When this occurs, the endpoint can request a public IP address from a STUN server. Other endpoints can use this publicly reachable IP to establish a connection.

TURN (Traversal Using Relays around NAT)

As the name suggests, a TURN server is used as a relay or intermediate server to exchange data. If any endpoint under Symmetric NAT can contact a server on the public internet to establish a connection with another endpoint, it is called a TURN client.

But, a disadvantage of using a TURN server is that it is required throughout the whole session, unlike the STUN server, which is not needed after the connection is established. Therefore, in the ICE technique, STUN is used as default.

In other words, a STUN server is used to get an external network address. TURN servers are used to relay traffic if a direct (peer-to-peer) connection fails.

Connecting to a Remote User

To connect to a remote user, we’d have to make an offer from a local to a remote user and receive an answer to establish a connection.

First, we ascertain media conditions, such as resolution and codec capabilities, locally. The metadata obtained will be used for the offer-and-answer mechanism. We’ll also get potential network addresses, known as candidates, for the app’s host.

This local data then needs to be shared with the remote user. Following this, the steps below are executed to initiate a connection:

The local user creates an offer which is essentially an SDP session description.
This offer is stringified and sent over to a remote user.
The remote user sets its remote description to the obtained offer and sends back an answer.
The answer is used to set a remote description of the local user.
Connection is established between the local and the remote user.

Networking information also needs to be exchanged between the two parties. The ICE framework is used to find network interfaces and ports.

As we go along with the tutorial, we’ll understand this part better.

Let’s start the process by modifying the code in the build method of the stateful widget to create a UI for the remote user.

@override
  Widget build(BuildContext context) {
    return Scaffold(
        appBar: AppBar(
          title: Text(widget.title),
        ),
        body: Column(
          children: [
            videoRenderers(),
          ],
        ));
  }

The videoRenderers() function would render the media of both the local and the remote user.

SizedBox videoRenderers() => SizedBox(
        height: 210,
        child: Row(children: [
          Flexible(
            child: Container(
              key: Key('local'),
              margin: EdgeInsets.fromLTRB(5.0, 5.0, 5.0, 5.0),
              decoration: BoxDecoration(color: Colors.black),
              child: RTCVideoView(_localVideoRenderer),
            ),
          ),
          Flexible(
            child: Container(
              key: Key('remote'),
              margin: EdgeInsets.fromLTRB(5.0, 5.0, 5.0, 5.0),
              decoration: BoxDecoration(color: Colors.black),
              child: RTCVideoView(_remoteVideoRenderer),
            ),
          ),
        ]),
      );

Initialise a _remoteVideoRenderer as we did a _localVideoRenderer.

final _remoteVideoRenderer = RTCVideoRenderer();

We also need a few local variables and TextEditingController to be used later.

final sdpController = TextEditingController();
bool _offer = false;

RTCPeerConnection? _peerConnection;
MediaStream? _localStream;

Note: The TextEditingController needs to be disposed of. Therefore we modify the dispose() of the stateful widget as:

  @override
  void dispose() async {
    await _localVideoRenderer.dispose();
    sdpController.dispose();
    super.dispose();
  }

To connect with the peer, we create a new function named _createPeerConnection(). Inside this function, we’ll first add the configuration as shown below:

Map<String, dynamic> configuration = {
      "iceServers": [
        {"url": "stun:stun.l.google.com:19302"},
      ]
    };

The configuration has the URL to the STUN server. In our example, we’ve used a free one made available by Google. You can check out a list of options for STUN servers here.

Next, we add the SDP Constraints with the mandatory offer to receive audio/video set to true.

final Map<String, dynamic> offerSdpConstraints = {
      "mandatory": {
        "OfferToReceiveAudio": true,
        "OfferToReceiveVideo": true,
      },
      "optional": [],
    };

We need to create a late variable named _localStream of type MediaStream inside our widget.

late MediaStream _localStream;

Set this to the _getUserMedia() function we had created earlier.

 _localStream = await _getUserMedia();

With that done, we create an RTC peer connection with the configuration and SDP constraints as parameters and then add the local stream.

    RTCPeerConnection pc =
        await createPeerConnection(configuration, offerSdpConstraints);

    pc.addStream(_localStream);

    pc.onIceCandidate = (e) {
      if (e.candidate != null) {
        print(json.encode({
          'candidate': e.candidate,
          'sdpMid': e.sdpMid,
          'sdpMlineIndex': e.sdpMLineIndex
        }));
      }
    };

    pc.onIceConnectionState = (e) {
      print(e);
    };

    pc.onAddStream = (stream) {
      print('addStream: ' + stream.id);
      _remoteVideoRenderer.srcObject = stream;
    };

 return pc;

Lastly, we set the remoteVideoRenderer source object to the stream obtained and return pc.

Moving ahead, we make a createOffer function to generate an offer and set the local description to it.

  void _createOffer() async {
    RTCSessionDescription description =
        await _peerConnection!.createOffer({'offerToReceiveVideo': 1});
    var session = parse(description.sdp.toString());
    print(json.encode(session));
    _offer = true;

    _peerConnection!.setLocalDescription(description);
  }

Also, we create a createAnswer function.

  void _createAnswer() async {
    RTCSessionDescription description =
        await _peerConnection!.createAnswer({'offerToReceiveVideo': 1});

    var session = parse(description.sdp.toString());
    print(json.encode(session));

    _peerConnection!.setLocalDescription(description);
  }

We add the function to set a remote description and add a candidate.

  void _setRemoteDescription() async {
    String jsonString = sdpController.text;
    dynamic session = await jsonDecode('$jsonString');

    String sdp = write(session, null);

    RTCSessionDescription description =
        new RTCSessionDescription(sdp, _offer ? 'answer' : 'offer');
    print(description.toMap());

    await _peerConnection!.setRemoteDescription(description);
  }

  void _addCandidate() async {
    String jsonString = sdpController.text;
    dynamic session = await jsonDecode('$jsonString');
    print(session['candidate']);
    dynamic candidate = new RTCIceCandidate(
        session['candidate'], session['sdpMid'], session['sdpMlineIndex']);
    await _peerConnection!.addCandidate(candidate);
  }

The only thing left now is to create a user interface to call the above functions. We make a basic layout using a Row and a Column to arrange a TextField for input and buttons to call functions.

Modify the build function as shown below:

@override
  Widget build(BuildContext context) {
    return Scaffold(
        appBar: AppBar(
          title: Text(widget.title),
        ),
        body: Column(
          children: [
            videoRenderers(),
            Row(
              children: [
                Padding(
                  padding: const EdgeInsets.all(16.0),
                  child: SizedBox(
                    width: MediaQuery.of(context).size.width * 0.5,
                    child: TextField(
                      controller: sdpController,
                      keyboardType: TextInputType.multiline,
                      maxLines: 4,
                      maxLength: TextField.noMaxLength,
                    ),
                  ),
                ),
                Column(
                  crossAxisAlignment: CrossAxisAlignment.center,
                  children: [
                    ElevatedButton(
                      onPressed: _createOffer,
                      child: const Text("Offer"),
                    ),
                    const SizedBox(
                      height: 10,
                    ),
                    ElevatedButton(
                      onPressed: _createAnswer,
                      child: const Text("Answer"),
                    ),
                    const SizedBox(
                      height: 10,
                    ),
                    ElevatedButton(
                      onPressed: _setRemoteDescription,
                      child: const Text("Set Remote Description"),
                    ),
                    const SizedBox(
                      height: 10,
                    ),
                    ElevatedButton(
                      onPressed: _addCandidate,
                      child: const Text("Set Candidate"),
                    ),
                  ],
                )
              ],
            ),
          ],
        ));
  }

You’ll notice we have created elevated buttons to generate an offer, answer, set remote description, and set candidate.

Final Flutter WebRTC Project Demo

Now, we can go ahead and test our app features.

Refer to the complete code here.

Run the Flutter project. It should show up as seen below:

We can copy the URL and paste it into a new browser window to create two peers.

Click on the ‘Offer’ button in window 1 to generate an offer. This should output the offer to the console accessible using developer tools on Chrome.

Copy-paste this offer into the TextField of window 2 and click on the ‘Set Remote Description’ button. That should set the remote description by calling the underlying function.

Now, click on ‘Answer’ to generate an answer and copy-paste it in the TextField of window 1.

Lastly, copy a candidate from window 2, paste it in the window 1 TextField, and click on the ‘Set Candidate’ button.

This should create the WebRTC connection, and the remote video should show up as seen below:

And, that’s it. We have successfully created an application using the WebRTC plugin for Flutter. As you can see, the process of setting up a Flutter WebRTC app may seem a little complicated, but once you get the hang of it, it’s not too difficult to accomplish.

Using a Signaling Server

You must’ve noticed we had to manually copy and paste the offer and answer in the app. Alternatively, you can use a Signaling Server to help communicate between the parties and establish a connection.

A signaling server helps coordinate communication by exchanging information. WebRTC needs some exchange of information for clients to set up calls. Signaling allows clients to pass messages between each other.

The exchanged information is then used to overcome challenges in communication. WebRTC uses the ICE Framework to cope with NATs and firewalls, but the information of candidates, the offer, and the answer need to be exchanged via signaling.

For our example, we demonstrated a simple Flutter app using a free STUN Server and manual signaling by copying-pasting the offer, answer, and a candidate. In production, you may want to set up your own Flutter WebRTC servers, but scaling and maintaining them can get tedious really fast.

As a developer, I would not want to deal with the intricacies of server-side code. It’s extra effort, and I’d rather focus on what I do best - building UIs with Flutter and providing my users with a better experience. That is what 100ms helps me with.

Integrate WebRTC with Flutter via 100ms

100ms makes it really easy to integrate audio/video into my Flutter app without sacrificing the customization that’d I have with WebRTC. I can continue building my apps without worrying about scalability or having to set up or manage servers of my own.

The 100ms SDK helps make app development more efficient and requires far less coding effort than usual. If you’re curious, use the 100ms plugin instead of the Flutter WebRTC plugin, and you’ll see how it makes your life easier.

Or, you could also try this quickstart guide on creating a Demo Flutter Project with the 100ms SDK.

DEV Community: 100ms Inc.

Engage Your Audience with Interactive Polls and Quizzes - A Step-by-Step Guide

How to enable polls and quizzes for your rooms on the 100ms dashboard?

How to add Polls to React sample app?

Buying Guide: Agora, Twilio, Jitsi (JaaS), Zoom, 100ms

Functionality and Features

Compliance and Security

Support

Pricing

Introducing 100ms Starter Kits

Server-side Considerations for your WebRTC Infrastructure

The Server in WebRTC

Signaling Server

WebRTC multi-peer architecture

Mesh Architecture (STUN/TURN Servers)

WebRTC Media Servers

Mixing Architecture (using the MCU Server)

Advantages and Disadvantages of Mixing

Routing Architecture (SFU Server)

Advantages and Disadvantages of Routing

Closing Notes

WebRTC with 100ms

Further Reading

A New Approach to Live Streaming

The present-day live streaming tech stack

So what’s changing?

What will happen to the live streaming tech?

How 100ms tests for Network Reliability

Testing Network Reliability for Real-World Scenarios

Why Network Reliability Matters

1. Low Bandwidth Handling/Management Test

2. Network Blip Test

3. Network Switching Test

HLS 101: What it is, How it works & When to use it

What is HLS?

How does HLS work?

Resolving Latency Issues in HLS

When to use HLS Streaming

Major Advantages of HLS

Does 100ms support HLS for live streaming?

Introduction to Low Latency Streaming with HLS

The Way of the HLS

Tuning HLS for Low Latency

The Evolution of HLS to LL-HLS

How LL-HLS reduces Latency

Preload Hints

**Low-Latency HLS on non-Apple devices

Comparing LL-HLS, LL-DASH and WebRTC

Support for Adaptive Bitrate (ABR)

Roles on 100ms: Mapping real-world interactions to live video with a few clicks

What is a role?

How Roles Work: An Example

Why are custom roles required?

Custom roles in action

Using Roles to recreate a Patient’s Consultation Experience Online

Handling Permissions for each Role

Testing the App

Introduction to Low Latency Streaming with HLS

The Way of the HLS

Tuning HLS for Low Latency

The Evolution of HLS to LL-HLS

How LL-HLS reduces Latency

Low-Latency HLS on non-Apple devices

Agora vs Twilio vs Jitsi vs Zoom vs 100ms

Agora: Real-Time Engagement for Apps

What is Agora? What are its main offerings?

Agora’s Key Functionality Metrics

Agora’s Limitations

Twilio: Meaningful In-App Customer Engagement

What is Twilio? What are its main offerings?

Twilio’s Key Functionality Metrics

Twilio’s Limitations

Jitsi: Open-Source Video Integration

What is Jitsi? What are its main offerings?

Jitsi’s Key Functionality Metrics

Jitsi’s Limitations

Zoom: Quick and Easy Video Conferencing

What is Zoom? What are its main offerings?

Zoom’s Key Functionality Metrics

Zoom’s Limitations