DEV Community: Tim

Using Ant Media Server's JavaScript SDK to Live Stream with a Virtual Background

Tim — Wed, 16 Aug 2023 10:39:25 +0000

Original blog post at https://antmedia.io/ant-media-server-virtual-background/

How to use Virtual Backgrounds with Ant Media Server’s JavaScript SDK?

Using a virtual background with Ant Media Server’s JavaScript SDK is very easy! Just follow these simple steps to apply a virtual background on your publishing page.

The two main WebRTCAdaptor methods that are used to apply the background replacement are:

Set the background image: webRTCAdaptor.setBackgroundImage()
Apply the background image: webRTCAdaptor.enableEffect()

There are only a few basic changes needed to apply the logic to add the feature.

Here is a very basic publish page that has two buttons, one for publishing and one for unpublishing and as you can see, a very boring background.

and here is the code inside the <body> tag:

<div class="container">

    <video id="localVideo" autoplay muted controls playsinline width="800px"> </video>

    <div class="d-block">
      <button id="start_publish">Publish</button>
      <button id="stop_publish">Unpublish</button>
      <p id="status">Status:Offline</p>
    </div>


   </div>

   <script type="module">

   import {WebRTCAdaptor, VideoEffect} from "./node_modules/@antmedia/webrtc_adaptor/dist/es/index.js";

    const webRTCAdaptor = new WebRTCAdaptor({
      websocket_url: "wss://ant-media-server:5443/LiveApp/websocket",
      localVideoElement: document.getElementById("localVideo"),
      mediaConstraints: {
        video: true,
        audio:true
      },
      callback: (info, obj) => {
        console.log("callback info: " + info);
        if (info == "publish_started") {
            console.log("publish started");
            statusInfo.innerHTML = "Status:Broadcasting"
        }
        else if (info == "publish_finished") {
            console.log("publish finished")
            statusInfo.innerHTML = "Status:Offline"
        }
        else if( info == "available_devices"){
          console.log(obj)
        }
      },
      callbackError: function (error, message) {
        //some of the possible errors, NotFoundError, SecurityError,PermissionDeniedError
        console.log("error callback: " + JSON.stringify(error));
      }

    });

    //get random streamId
    const streamId = "stream" + parseInt(Math.random()*999999);
    const startPublishButton = document.getElementById("start_publish")
    const stopPublishButton = document.getElementById("stop_publish")
    const statusInfo = document.getElementById("status");

    startPublishButton.addEventListener("click", () => {
      console.log("start publish is clicked " + streamId);
      webRTCAdaptor.publish(streamId);
    });

    stopPublishButton.addEventListener("click", () => {
      console.log("stop publish is clicked");
      webRTCAdaptor.stop(streamId);
    });

</script>

As you can see, it's a very basic WebRTC publish page implementation using the WebRTC Adaptor. To use this publish page, remember to update the websocket_url URL with your own Ant Media Server instance.

It’s pulling the WebRTC Adaptor from NPM and importing the WebRTCAdaptor and VideoEffect classes which is what we’ll need to publish WebRTC.

Then apply an event handler to the two buttons to publish and unpublish the live stream which calls the SDK methods webRTCAdaptor.publish(streamId) and webRTCAdaptor.stop(streamId).

Step 1: Make Virtual Background Images Available

The images that you want to have available as virtual images, need to be added into a directory accessible by your publish page.

In this example, all the images are located in the images directory. Lets create the HTML that will display all the images under the video player so we can select the desired background.

Add the following HTML under the buttons.

<div class="col-sm-12 text-center" id="virtualBackgrounds">

   <h3>Select a virtual background Image</h3>
   <div class="d-flex" id="virtualbackgroundimages">

     <img src="./images/noeffect-background.png"  id="noeffect" class="backgroundImages selected" />
     <img src="./images/blur-background.png" id="blur" class="backgroundImages " />
     <img src="./images/virtual-background.png" id="antMediaBackground" class="backgroundImages "/>
     <img src="./images/cloud-background.png" class="backgroundImages"/>

     <img src="./images/unsplash.jpg" class="backgroundImages"/>
     <img src="./images/huy.jpg" class="backgroundImages"/>
     <img src="./images/mokry.jpg" class="backgroundImages"/>
     <img src="./images/mitchell.jpg" class="backgroundImages"/>
   </div>

   <div class="d-block">
   <h3>Upload a Virtual Background Image</h3>
   <input type="file" class="custom-file-input" id="customFile" style="height:100%;" accept=".jpg, .png, .jpeg">
   </div>
 </div>

Make sure to update this code with the names of the images you have in your directory. You’ll also notice there is also a section to upload a background image. We’ll get to that later.

It now looks like this:

Notice that we also have two images that do not look like backgrounds on the far left. These will function to either remove the virtual background or blur the background.

Right now, clicking these images do nothing! So its time to use the SDK to help us add the virtual background functionality.

Step 2: Add a Click Handler to the Images

To make things easy and quick, we’ll use the JavaScript library jQuery to apply event handlers, for example, when you click on the desired background image, a Javascript method is called to apply the virtual background.

Lets start by adding that click handler.

All the images have a class name of backgroundImages. So lets use that class and register a click handler.

$(".backgroundImages").click(enableVirtualBackground);
function enableVirtualBackground(){
 // add the magic here
}

As you can see, we are registering a click handler to each image and calling a method enableVirtualBackground thats directly underneath.

This is where all the magic will happen.

Add this bit of JavaScript right under the import statement that imports the WebRTCAdaptor

Lets also not forget about uploading custom backgrounds.

There is also a upload section, so we’ll need to hook this up with an event handler to capture the image thats uploaded.

Here is the code sample that does that:

$('input[type="file"]').change(function(e) {
     const url = URL.createObjectURL(e.target.files[0]);
     const newBackgroundImg = $("<img src=\"" + url + "\" class=\"backgroundImages\"/>");
     $("#virtualbackgroundimages").append(newBackgroundImg);
     $(".backgroundImages").click(enableVirtualBackground);
     newBackgroundImg.click();
 });

What this is doing is listening for when a file is selected. Once thats been done, its using URL.createObjectURL JavaScript function to create a special object URL (Uniform Resource Locator) that represents the uploaded image.

Then a new image is created and appended to the list of current background images in the HTML.

Once it has been appended, apply the click handler as before to all the images and simulate a click on the newly uploaded image to apply the virtual background.

Step 3: Enable Virtual Backgrounds

Now its time to get our virtual backgrounds working!

You might have noticed that the two images that either blur the background or remove the virtual background have an id of blur or noeffect respectively.

We’ll be using VideoEffect to configure the background image and webRTCAdaptor to apply the virtual background.

Here is the full working code sample for enableVirtualBackground method:

function enableVirtualBackground() {
     var effectName;
     if ($(this).attr("id") == "blur") {
         effectName = VideoEffect.BLUR_BACKGROUND;
     }
     else if ($(this).attr("id") == "noeffect") {
         effectName = VideoEffect.NO_EFFECT;
     }
     else {
         effectName = VideoEffect.VIRTUAL_BACKGROUND;
         webRTCAdaptor.setBackgroundImage(this);
     }
     webRTCAdaptor.enableEffect(effectName).then(() => {
         console.log("Effect: "+ effectName+" is enabled");
     }).catch(err => {
         console.error("Effect: "+ effectName+" is not enabled. Error is " + err);
         $.notify(err.name, {
             autoHideDelay:5000,
             className:'error',
             position:'top center'
         });
     });
     $(this).addClass("selected");
 }

The first part is detecting whether or not we want to remove the virtual background or blur the background and assigning the necessary VideoEffect property (VideoEffect.NO_EFFECT or VideoEffect.BLUR_BACKGROUND) based on the ID of the image that has been clicked.

Otherwise, we want to apply the image as a virtual background, so assigning VideoEffect.VIRTUAL_BACKGROUND is necessary followed by calling the webRTCAdaptor.setBackgroundImage to set the image as the virtual background image.

Once thats been done, all thats left is to call the webRTCAdaptor.enableEffect to enable the virtual background.

Step 4: Test the Virtual Background

Now its time to test that this works.

You should now be able to apply a virtual background from one of the images available or upload your own.

I’ve decided to be on the beach for this example.

Now try live streaming and changing your background on the fly. You should see it change almost immediately.

Full Code Sample:

Here is the full source code of the page. Remember to update the websocket_url of your server.

<!DOCTYPE html>

<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta http-equiv="X-UA-Compatible" content="IE=edge">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Document</title>
  <script src="https://code.jquery.com/jquery-3.6.0.min.js"></script>

  <style>
    body{
      font-family: Tahoma;
    }
    #virtualBackgrounds img{
      height: 50px;
    }
    .container{
      margin:auto;
      width: 800px;
      text-align:center;
    }
  </style>

</head>
<body>

  <div class="container">

    <video id="localVideo" autoplay muted controls playsinline width="800px"> </video>

    <div class="d-block">
      <button id="start_publish">Publish</button>
      <button id="stop_publish">Unpublish</button>
      <p id="status">Status:Offline</p>
    </div>


   <div class="col-sm-12 text-center" id="virtualBackgrounds">
      <h3>Select a virtual background Image</h3>
      <div class="d-flex" id="virtualbackgroundimages">

        <img src="./images/noeffect-background.png"  id="noeffect" class="backgroundImages selected" />
        <img src="./images/blur-background.png" id="blur" class="backgroundImages " />
        <img src="./images/virtual-background.png" id="antMediaBackground" class="backgroundImages "/>
        <img src="./images/cloud-background.png" class="backgroundImages"/>

        <img src="./images/unsplash.jpg" class="backgroundImages"/>
        <img src="./images/huy.jpg" class="backgroundImages"/>
        <img src="./images/mokry.jpg" class="backgroundImages"/>
        <img src="./images/mitchell.jpg" class="backgroundImages"/>
      </div>

      <div class="d-block">
      <h3>Upload a Virtual Background Image</h3>
      <input type="file" class="custom-file-input" id="customFile" style="height:100%;" accept=".jpg, .png, .jpeg">
      </div>
    </div>

   </div>

   <script type="module">

   import {WebRTCAdaptor, VideoEffect} from "./node_modules/@antmedia/webrtc_adaptor/dist/es/index.js";

   $(".backgroundImages").click(enableVirtualBackground);

   $('input[type="file"]').change(function(e) {
        const url = URL.createObjectURL(e.target.files[0]);
        const newBackgroundImg = $("<img src=\"" + url + "\" class=\"backgroundImages\"/>");
        $("#virtualbackgroundimages").append(newBackgroundImg);
        $(".backgroundImages").click(enableVirtualBackground);
        newBackgroundImg.click();
    });

   function enableVirtualBackground() {
        var effectName;
        if ($(this).attr("id") == "blur") {
            effectName = VideoEffect.BLUR_BACKGROUND;
        }
        else if ($(this).attr("id") == "noeffect") {
            effectName = VideoEffect.NO_EFFECT;
        }
        else {
            effectName = VideoEffect.VIRTUAL_BACKGROUND;
            webRTCAdaptor.setBackgroundImage(this);
        }
        webRTCAdaptor.enableEffect(effectName).then(() => {
            console.log("Effect: "+ effectName+" is enabled");
        }).catch(err => {
            console.error("Effect: "+ effectName+" is not enabled. Error is " + err);
            $.notify(err.name, {
                autoHideDelay:5000,
                className:'error',
                position:'top center'
            });
        });
        $(this).addClass("selected");
    }

    const webRTCAdaptor = new WebRTCAdaptor({
      websocket_url: "wss://ant-media-server:5443/LiveApp/websocket",
      localVideoElement: document.getElementById("localVideo"),
      mediaConstraints: {
        video: true,
        audio:true
      },
      callback: (info, obj) => {
        console.log("callback info: " + info);
        if (info == "publish_started") {
            console.log("publish started");
            statusInfo.innerHTML = "Status:Broadcasting"
        }
        else if (info == "publish_finished") {
            console.log("publish finished")
            statusInfo.innerHTML = "Status:Offline"
        }
        else if( info == "available_devices"){
          console.log(obj)
        }
      },
      callbackError: function (error, message) {
        console.log("error callback: " + JSON.stringify(error));
      }

    });

    const streamId = "stream" + parseInt(Math.random()*999999);
    const startPublishButton = document.getElementById("start_publish")
    const stopPublishButton = document.getElementById("stop_publish")
    const statusInfo = document.getElementById("status");

    startPublishButton.addEventListener("click", () => {
      console.log("start publish is clicked " + streamId);
      webRTCAdaptor.publish(streamId);
    });

    stopPublishButton.addEventListener("click", () => {
      console.log("stop publish is clicked");
      webRTCAdaptor.stop(streamId);
    });

</script>
</body>
</html>

There is also a sample page in the default applications located at the following URL https://ant-media-server:5443/{application}/publish_webrtc_virtual_background.html.

That sample page is also available on Github and you can see a live demo on our samples page

What Next?

Explore the JavaScript SDK reference guide and other popular SDKs, the documentation or join the community discussions to learn more about the Ant Media Server.

Deployment options include other 1-click apps, cloud marketplaces, Docker/Kubernetes/Scripts on various cloud providers such as AWS, Microsoft Azure, Digital Ocean, Linode, Google, and Alibaba.

A 30 day free trial is available to try out the Enterprise Edition of the product and support packages are available for both editions if you need some extra help getting started.

Original blog post at https://antmedia.io/ant-media-server-virtual-background/

WebRTC Cheatsheet

Tim — Sun, 21 May 2023 22:37:41 +0000

🌐 WebRTC Basics:

WebRTC stands for Web Real-Time Communication.
It is an open framework for enabling real-time communication in web browsers and mobile applications.
WebRTC allows peer-to-peer audio, video, and data sharing without the need for additional plugins or software.

📹 Video Communication:

getUserMedia() API captures video from the user's camera.
MediaStream represents the video and audio streams.
RTCPeerConnection establishes a direct peer-to-peer connection for video communication.
RTCDataChannel enables real-time data sharing alongside video communication.

🔊 Audio Communication:

getUserMedia() API captures audio from the user's microphone.
MediaStream represents the audio stream.
RTCPeerConnection establishes a direct peer-to-peer connection for audio communication.
RTCDataChannel enables real-time data sharing alongside audio communication.

💡 Useful JavaScript APIs:

getUserMedia(): Grants access to the user's camera and microphone.
RTCPeerConnection: Handles peer-to-peer communication.
RTCDataChannel: Enables real-time data sharing between peers.
MediaStream: Represents audio and video streams.

🔒 Security and Encryption:

WebRTC uses Secure Real-Time Transport Protocol (SRTP) for encryption.
Signaling is required to exchange session information and establish a connection.
Signaling servers help coordinate the communication process but are not part of the WebRTC standard.

🌐 WebRTC Frameworks and Libraries:

🚀 Deploying WebRTC:

Cloud hosting platforms like Firebase, AWS, or Heroku can be used to deploy WebRTC applications.
Consider the server-side requirements for signaling, TURN, and STUN servers.

📚 Resources:

WebRTC API documentation: https://webrtc.org/
WebRTC samples and demos: https://antmedia.io/webrtc-samples/
WebRTR Live Streaming Software: https://github.com/ant-media

3 Great Canvas Manipulations in WebRTC Live Streaming

Tim — Fri, 19 May 2023 10:55:57 +0000

This blog post is originally published at antmedia.io

How to manipulate live stream content in WebRTC is one of the most asked questions at Ant Media.

This can be achieved by using the HTML canvas element as a live stream source in WebRTC.

Here are some common questions live streamers tend to ask:

How can I add my logo to the live stream?
How can I add a watermark to my live stream?
My live stream doesn’t have video, can I display a static image instead of video?

In this blog post, we will show you 3 use cases of using Canvas as a live stream source:

Adding a logo to live stream
Canvas with audio
Putting a background image on a live stream

This tutorial uses Ant Media Server, but the technique can be applied to other WebRTC stream sources.

Adding a Logo to Live Stream

We will use the sample page that comes with Ant Media Server to publish a live stream.

You can access it at https://ANT-MEDIA-SERVER:PORT/APPLICATION/ on the server it's located at /usr/local/antmedia/webapps/LiveApp/index.html

The idea is to use the canvas as a stream source. We will draw two things on the canvas. The first one is the content of the video component and the second one is the logo image.

The logo image in our sample is a PNG file.

Lets start by adding a canvas component above the video component:

<div class="col-sm-12 form-group">
     <canvas id="canvas" width="200" height="150"></canvas>
     <p>
        <video id="localVideo"  autoplay muted controls playsinline></video>
    </p>
 </div>

Then use this canvas component for merging the logo image and video frame. This is what our method will do:

Create an Image instance from the PNG file
Create a draw method where you merge the video frame and logo image.

We'll call this method every 25ms with the help of JavaScripts setInterval.

Finally, capture the canvas content at 25fps using the captureStream method:

var canvas = document.getElementById('canvas');

var vid = document.getElementById('localVideo');

  var image=new Image();

  image.src="antmedia.png";

    function draw() {

    if (canvas.getContext) {

      var ctx = canvas.getContext('2d');

      ctx.drawImage(vid, 0, 0, 200, 150);

      ctx.drawImage(image,50, 10, 100, 30)
    }

}

//update canvas for every 25ms

setInterval(function() { draw(); }, 25);

//capture stream from canvas

var localStream = canvas.captureStream(25);

Finally, initialise the Ant Media Servers Javascript SDK's WebRTC adaptor with the localStream created from the canvas.

We need to pass the localStream to the WebRTCAdaptor:

navigator.mediaDevices.getUserMedia({video: true, audio:true}).then(function (stream) {

        var video = document.querySelector('video#localVideo');

        video.srcObject = stream;

        video.onloadedmetadata = function(e) {

        video.play();

      };

      //initialize the webRTCAdaptor with the localStream created.

      //initWebRTCAdaptor method is implemented below

      initWebRTCAdaptor(localStream);

  });

When publishing starts, the canvas content will be published and you can view the live stream with the added logo.

Canvas with Audio

In this sample, we will show how to publish only canvas with audio. There will not be video content.

This might be useful when you only want to publish audio, i.e a radio station website.

In order to achieve this, we will skip drawing the video content and just draw a rectangle on the canvas instead of a logo image file.

We will use drawings to show that you can do more than putting a logo. You can also write text on the canvas.

For this example, we need to change the content of the draw method. Instead of drawing an image, we draw a rectangle:

function draw() {
   if (canvas.getContext) {
   var ctx = canvas.getContext('2d');
   ctx.fillStyle = 'rgba(0, 0, 200, 0.5)';
   ctx.fillRect(30, 30, 100, 50);
   }
 }

Then we need to disable video and add only audio to localStream:

//get audio with getUserMedia

navigator.mediaDevices.getUserMedia({video: false, audio:true}).then(function (stream) {

  //add audio track to the localstream which is captured from canvas

  localStream.addTrack(stream.getAudioTracks()[0]);

  //initialize the webRTCAdaptor with the localStream created.

  //initWebRTCAdaptor method is implemented below

  initWebRTCAdaptor(localStream);

});

When this stream is played back, there will be a rectangle and audio. No video.

Putting a Background Image on a Live Stream

This use case is also known as the green-screen effect. You can live stream using a green screen and replace that green screen with something else.

For example, you can replace that green screen with a background image of your choice. Let’s see how we can do this.

Start by adding two more canvas components. The first canvas is for a video frame, the second one is for a background image that we will use in our live stream and the third one is for merging these canvas together :

<canvas id="canvas" width="200" height="150"></canvas>

<canvas id="canvas2" width="200" height="150"></canvas>

<canvas id="canvas3" width="200" height="150"></canvas>

Then we find the green pixels and replace them with the background image’s pixel colours:

var canvas = document.getElementById('canvas');

var canvas2 = document.getElementById('canvas2');

var canvas3 = document.getElementById('canvas3');

var vid = document.getElementById('localVideo');

var image=new Image();

image.src="antmedia.png";

  function draw() {

  if (canvas.getContext) {

    var ctx = canvas.getContext('2d');

    var ctx2 = canvas2.getContext('2d');

    var ctx3 = canvas3.getContext('2d');

    ctx2.drawImage(image,0, 0, 200, 150)

    let frame2 =  ctx2.getImageData(0, 0, 200, 150);

    ctx.drawImage(vid, 0, 0, 200, 150);

    let frame = ctx.getImageData(0, 0, 200, 150);

    let l = frame.data.length / 4;

    for (let i = 0; i < l; i++) {

      let r = frame.data[i * 4 + 0];

      let g = frame.data[i * 4 + 1];

      let b = frame.data[i * 4 + 2];

      if (g > 100 && r > 100 && b < 43){

        frame.data[i * 4 + 0] = frame2.data[i * 4 + 0]

        frame.data[i * 4 + 1] = frame2.data[i * 4 + 1]

        frame.data[i * 4 + 2] = frame2.data[i * 4 + 2]

        }

      }

    ctx3.putImageData(frame, 0, 0);

  }

}

Finally, we use the canvas3 for capturing the stream:

//update canvas for every 40ms

setInterval(function() { draw(); }, 25);

//capture stream from canvas

var localStream = canvas3.captureStream(25);

//get audio with getUserMedia

navigator.mediaDevices.getUserMedia({video: true, audio:true}).then(function (stream) {

var video = document.querySelector('video#localVideo');

      video.srcObject = stream;

      video.onloadedmetadata = function(e) {

        video.play();

      };

//initialize the webRTCAdaptor with the localStream created.

//initWebRTCAdaptor method is implemented below

initWebRTCAdaptor(localStream);

});

This now completes the tutorial. Hope you enjoyed the post. You might want to check How to Embed WebRTC Live Streaming into Your Website in 2 Easy Ways?

This blog post is originally published at antmedia.io

How to Create a Kubernetes Cluster on DigitalOcean

Tim — Thu, 27 Apr 2023 10:07:17 +0000

Kubernetes Cluster Creation on DigitalOcean

To create a Kubernetes cluster:

From the Create menu in the control panel, click Kubernetes.
Select a Kubernetes version. The latest version is selected by default and is the best choice if you have no specific need for an earlier version.
Choose a data center region.
Customize the default node pool, choose the node pool names, and add additional node pools.
Name the cluster, select the project you want the cluster to belong to, and optionally add a tag. Any tags you choose will be applied to the cluster and its worker nodes.
Click Create Cluster. Provisioning the cluster takes several minutes.

Verify the Cluster was created

Download the cluster configuration file by clicking Actions, then Download the Config from the cluster home page.

run the following kubectl command to check nodes:

kubectl –kubeconfig={CONFIG FILE PATH} get nodes

You should get nodes as in the following image:

Deploying An Application on Kubernetes

After this point, you have a running Kubernetes Cluster and it is time to use it!

Now it's time to deploy an application on the Kubernetes cluster. As a suggestion, you can use Ant Media Server an Ultra-low latency WebRTC live streaming engine.

Simply follow the documentation here

Build a custom live streaming engine with Ant Media Server plugins

Tim — Mon, 24 Apr 2023 13:47:40 +0000

When content management solutions introduced a plugin ecosystem, it revolutionised how we can customise and extend websites.

The ability to easily install a plugin that provides some specific functionality for a specific use case feels like a dream and it still is.

WordPress has taken it to another level with enough plugins to totally transform a WordPress blog into a fully-fledged social media network, video hosting platforms like Youtube or a highly scalable e-commerce store.

With the rise of live streaming, the large players in the space such as Wowza, Agora and Castr are offering services to allow influencers, live auction houses and live shopping services to name just a few use cases to integrate live streaming into their applications.

But, you are still restricted to what the service has to offer. For example, what about integrating ad insertion, AI or a unique, amazing effect you want to apply to a live stream. It's not possible unless the service offers an API or a way of interacting with the live stream. At least, not until now!

Ant Media Server Streaming Engine

If you haven't heard of Ant Media Server, let me briefly explain it.

Ant Media Server is a live streaming engine built from Java that can be hosted on your own servers to enable a fully featured live streaming engine service.

Features include:

Multiple ingest formats like SRT, RTMP, WebRTC
WebRTC ~0.5 second latency
Restreaming to multiple endpoints e.g. social networks
Live stream recording locally or directly to Amazon S3
Multiple SDKs: Android, iOS, Flutter, React, React Native
Scaling and clusting to 500K viewers+

The software can support all of the most popular use cases you can think of and has an extremely reasonable licensing package. There is also a 1-click installation in the most popular cloud marketplaces like Amazon, Digital ocean and Vultr.

Ant Media Server Plugins, Applications and Marketplace

But Ant Media have now introduced a full-blown plugin architecture that can enable the customisation of the system.

They have launched a marketplace and there are already a few unique plugins that can be downloaded for free to extend the default offering.

A few plugins/applications from the marketplace include:

Portmeet - a Zoom-like application that enables a conferencing solution
Filter Plugin - add filters to the ongoing stream
OBS Castify - WebRTC for OBS (20MS P2P latency)
Stamp Plugin - add overlays to the video stream
DuetMaster - play a piano duet in Real-time

These are a few plugins and applications that are currently available in the marketplace ready to download and extend your Ant Media Server instance.

How to get started building plugins and applications

While the documentation for plugins and applications is not yet the best, there are a few resources available to get started.

There is a blog post plugin development guide and also a Youtube video Introducing the Plugin Mechanism but not much else yet.

What's the difference between plugins and Applications

A plugin is what will interact with the live stream and perform some pre-processing or capture some kind of data.

An application is a way to build a custom user interface to interact with the live streaming engine.

An example is the DuetMaster application in the marketplace.

It's a custom interface that uses the live streaming engine to send and process the data sent and received from the Ant Media Server instance to enable you to play the virtual piano with someone.

I think the marketplace and the flexibility of Ant Media Server are going to become more and more essential for the ever-growing demand for live streaming.