3 Great Canvas Manipulations in WebRTC Live Streaming

This blog post is originally published at antmedia.io

How to manipulate live stream content in WebRTC is one of the most asked questions at Ant Media.

This can be achieved by using the HTML canvas element as a live stream source in WebRTC.

Here are some common questions live streamers tend to ask:

• How can I add my logo to the live stream?
• How can I add a watermark to my live stream?
• My live stream doesn’t have video, can I display a static image instead of video?

In this blog post, we will show you 3 use cases of using Canvas as a live stream source:

1. Adding a logo to live stream
2. Canvas with audio
3. Putting a background image on a live stream

This tutorial uses Ant Media Server, but the technique can be applied to other WebRTC stream sources.

Adding a Logo to Live Stream

We will use the sample page that comes with Ant Media Server to publish a live stream.

You can access it at https://ANT-MEDIA-SERVER:PORT/APPLICATION/ on the server it's located at /usr/local/antmedia/webapps/LiveApp/index.html

The idea is to use the canvas as a stream source. We will draw two things on the canvas. The first one is the content of the video component and the second one is the logo image.

The logo image in our sample is a PNG file.

Lets start by adding a canvas component above the video component:

<div class="col-sm-12 form-group">
     <canvas id="canvas" width="200" height="150"></canvas>
     <p>
        <video id="localVideo"  autoplay muted controls playsinline></video>
    </p>
 </div>

Then use this canvas component for merging the logo image and video frame. This is what our method will do:

1. Create an Image instance from the PNG file
2. Create a draw method where you merge the video frame and logo image.

We'll call this method every 25ms with the help of JavaScripts setInterval.

Finally, capture the canvas content at 25fps using the captureStream method:

var canvas = document.getElementById('canvas');

var vid = document.getElementById('localVideo');

  var image=new Image();

  image.src="antmedia.png";

    function draw() {

    if (canvas.getContext) {

      var ctx = canvas.getContext('2d');

      ctx.drawImage(vid, 0, 0, 200, 150);

      ctx.drawImage(image,50, 10, 100, 30)
    }

}

//update canvas for every 25ms

setInterval(function() { draw(); }, 25);

//capture stream from canvas

var localStream = canvas.captureStream(25);

Finally, initialise the Ant Media Servers Javascript SDK's WebRTC adaptor with the localStream created from the canvas.

We need to pass the localStream to the WebRTCAdaptor:

navigator.mediaDevices.getUserMedia({video: true, audio:true}).then(function (stream) {

        var video = document.querySelector('video#localVideo');

        video.srcObject = stream;

        video.onloadedmetadata = function(e) {

        video.play();

      };

      //initialize the webRTCAdaptor with the localStream created.

      //initWebRTCAdaptor method is implemented below

      initWebRTCAdaptor(localStream);

  });

When publishing starts, the canvas content will be published and you can view the live stream with the added logo.

Canvas with Audio

In this sample, we will show how to publish only canvas with audio. There will not be video content.

This might be useful when you only want to publish audio, i.e a radio station website.

In order to achieve this, we will skip drawing the video content and just draw a rectangle on the canvas instead of a logo image file.

We will use drawings to show that you can do more than putting a logo. You can also write text on the canvas.

For this example, we need to change the content of the draw method. Instead of drawing an image, we draw a rectangle:

function draw() {
   if (canvas.getContext) {
   var ctx = canvas.getContext('2d');
   ctx.fillStyle = 'rgba(0, 0, 200, 0.5)';
   ctx.fillRect(30, 30, 100, 50);
   }
 }

Then we need to disable video and add only audio to localStream:

//get audio with getUserMedia

navigator.mediaDevices.getUserMedia({video: false, audio:true}).then(function (stream) {

  //add audio track to the localstream which is captured from canvas

  localStream.addTrack(stream.getAudioTracks()[0]);

  //initialize the webRTCAdaptor with the localStream created.

  //initWebRTCAdaptor method is implemented below

  initWebRTCAdaptor(localStream);

});

When this stream is played back, there will be a rectangle and audio. No video.

Putting a Background Image on a Live Stream

This use case is also known as the green-screen effect. You can live stream using a green screen and replace that green screen with something else.

For example, you can replace that green screen with a background image of your choice. Let’s see how we can do this.

Start by adding two more canvas components. The first canvas is for a video frame, the second one is for a background image that we will use in our live stream and the third one is for merging these canvas together :

<canvas id="canvas" width="200" height="150"></canvas>

<canvas id="canvas2" width="200" height="150"></canvas>

<canvas id="canvas3" width="200" height="150"></canvas>

Then we find the green pixels and replace them with the background image’s pixel colours:

var canvas = document.getElementById('canvas');

var canvas2 = document.getElementById('canvas2');

var canvas3 = document.getElementById('canvas3');

var vid = document.getElementById('localVideo');

var image=new Image();

image.src="antmedia.png";

  function draw() {

  if (canvas.getContext) {

    var ctx = canvas.getContext('2d');

    var ctx2 = canvas2.getContext('2d');

    var ctx3 = canvas3.getContext('2d');

    ctx2.drawImage(image,0, 0, 200, 150)

    let frame2 =  ctx2.getImageData(0, 0, 200, 150);

    ctx.drawImage(vid, 0, 0, 200, 150);

    let frame = ctx.getImageData(0, 0, 200, 150);

    let l = frame.data.length / 4;

    for (let i = 0; i < l; i++) {

      let r = frame.data[i * 4 + 0];

      let g = frame.data[i * 4 + 1];

      let b = frame.data[i * 4 + 2];

      if (g > 100 && r > 100 && b < 43){

        frame.data[i * 4 + 0] = frame2.data[i * 4 + 0]

        frame.data[i * 4 + 1] = frame2.data[i * 4 + 1]

        frame.data[i * 4 + 2] = frame2.data[i * 4 + 2]

        }

      }

    ctx3.putImageData(frame, 0, 0);

  }

}

Finally, we use the canvas3 for capturing the stream:

//update canvas for every 40ms

setInterval(function() { draw(); }, 25);

//capture stream from canvas

var localStream = canvas3.captureStream(25);

//get audio with getUserMedia

navigator.mediaDevices.getUserMedia({video: true, audio:true}).then(function (stream) {

var video = document.querySelector('video#localVideo');

      video.srcObject = stream;

      video.onloadedmetadata = function(e) {

        video.play();

      };

//initialize the webRTCAdaptor with the localStream created.

//initWebRTCAdaptor method is implemented below

initWebRTCAdaptor(localStream);

});

This now completes the tutorial. Hope you enjoyed the post. You might want to check How to Embed WebRTC Live Streaming into Your Website in 2 Easy Ways?

This blog post is originally published at antmedia.io