DEV Community: Braden Riggs

How-to Broadcast a WebRTC stream to Twitch

Braden Riggs — Tue, 01 Aug 2023 16:03:31 +0000

Recently, while exploring syndicating Dolby.io WebRTC streams, I learned that Twitch has added support for WebRTC Ingest or WHIP as it is known in the industry.

WebRTC for streaming is an exciting choice because it can decrease stream latency compared to traditional protocols such as RTMP and HLS. When ingested, Twitch will transmux the WebRTC stream into something the platform supports (HLS), so that adds latency, slowing down the feed.

With that said, WHIP support is a great step for the community and with OBS now adding support for WebRTC, I thought I'd have to try it out.

In this guide, we'll showcase how to stream WebRTC from OBS into Twitch.

Setting up OBS for WebRTC

The core OBS project is working to add WebRTC, however, at the moment it is still an experimental build. You can try out this build by downloading the version relevant to your system here.

Once downloaded, extract the project and install it.

Streaming WebRTC from OBS to Twitch

With the project installed and launched, navigate to:
Settings -> Stream

Inside of Stream select WHIP as your service:

To start a WebRTC stream to Twitch you need the Server path and your Stream key.

The Twitch WHIP server

The server is (currently) the same for everyone:
https://g.webrtc.live-video.net:4443/v2/offer

Note: This server currently only supports H264 and Opus encoded streams.

Getting Your Twitch Stream Key

Your Twitch Stream Key can be found on your dashboard once you've logged in, under:
settings -> stream

Copy both the Server URL and the Stream Key into the Server and Bearer Token inputs within OBS.

Click Apply, set up OBS as usual, and click Start Stream to begin your WebRTC broadcast to Twitch.

Learn More

Broadcasting a WebRTC stream to Twitch is an great feature for the site as it allows people to easily syndicate their WebRTC streams to a popular platform. Because Twitch transmuxes the WebRTC stream, some delay is added, so if you're looking for an end-to-end white-label real-time streaming solution, check out Dolby.io Real-time Streaming.

A special shout out to Sean DuBois for his work on both the OBS project and on Twitch's WHIP support.

A Low-Latency Live Stream React App

Braden Riggs — Mon, 03 Apr 2023 18:36:01 +0000

Original Article Published Here

When building a streaming app or platform it is important to consider how the end user experiences and engages with the content being streamed. If your users need to engage with the content creator, the delay between capture and consumption should be minimal. To achieve this, many developers rely on WebRTC, a content-over-internet transfer protocol that boasts exceptionally low delays for video and audio. By leveraging WebRTC, developers can quickly build a low-delay immersive experience, leaving plenty of time to make the UI look outstanding using front-end libraries such as ReactJS.

In this guide, we're going to showcase a WebRTC ReactJS streaming app powered by Dolby.io Streaming and NodeJS.

The WebRTC React Example Code

The WebRTC React Streaming example app can be found on the dolbyio-samples GitHub.

To set up the project you need four things:

A cloned copy of the sample app.
Node v16 or greater installed.
Yarn package manager v1.22.19 or greater installed.
A Dolby.io account.

Once you've cloned the repo and set up the Node and Yarn, navigate to the main directory via the terminal and run the following command to install all dependencies:

yarn

While your project is installing we can briefly discuss how to set up your Dolby.io account. Once you've created an account you'll be dropped off on the Dolby.io Dashboard where you can create and manage tokens required for leveraging Dolby.io Streaming servers.

Click the purple and white + Create button to create a new token. Give the token a label and your stream name a unique identifier, then switch to the Advanced tab to enable "Multisource" as shown in the image below:

Enabling Multisource allows you to leverage Dolby.io Streaming to capture and deliver multiple low-delay streams at once. With your Token created, we can now click on the newly created token and gather your stream name, stream account id, and stream publishing token as shown in the image below:

Now that we have all the credentials required to connect to the Dolby.io servers, let's update the project credentials. To do this we need to edit the .env.example, which can be found inside of apps/publisher/ and apps/viewer/, by renaming the file to .env and populating the file with the respective credentials.

Additionally inside of the apps/publisher/.env.example there is a parameter to adjust the viewer URL. For testing locally this can be set to a local host URL, however, in production, this should be a web-accessible endpoint.

VITE_RTS_VIEWER_BASE_URL=http://localhost:7070/

With all the credentials set up, we can now run the React streaming app. The app can be split into two functions, a publisher and a viewer. The publisher app, which is what a content creator would use, serves content to the end user who participates via the viewer app.

To start the publisher app experience:

yarn nx serve publisher

To start the viewer app experience:

yarn nx serve viewer

With both the viewer and the publisher running we now have a live streaming app with Node.js and React powered by Dolby.io WebRTC Streaming. This experience can be deployed on a cloud service such as Netlify to share publicly, just remember to add your branding and styling.

Building your own React WebRTC streaming app

With your Dolby.io account already created, building your own bespoke viewer and publisher experience is easy. Dolby.io Streaming has a React Native SDK allowing developers to quickly and easily build a streaming solution. If you are interested in learning more about Dolby.io Streaming check out some of our other blogs including building a Multiview web app or about our Dolby.io Streaming OBS integration.

Feedback or Questions? Reach out to the team on Twitter, LinkedIn, or via our support desk.

5 Different ways to Broadcast SRT Streams

Braden Riggs — Mon, 09 Jan 2023 19:16:36 +0000

Originally published here

SRT, or Secure Reliable Transport, is a type of streaming protocol that provides enhanced security and reliability for video streaming. SRT is becoming increasingly popular among broadcasters and streamers including industry stalwarts such as ESPN because of its ability to deliver high-quality content over challenging network conditions and for its ability to make contribution and stream ingestion easy. SRT streams provide improved security, low latency, and flexibility and is supported by a global community of developers all contributing to the open-source project. Because of the power of SRT streams, Dolby.io Real-Time Streaming has decided to launch support with an SRT open beta program.

In this guide, we'll cover a few different ways you can start broadcasting SRT streams with Dolby.io such as OBS, vMix and many more:

Streaming SRT with OBS
Streaming SRT with vMix
Streaming SRT with your iPhone
Streaming SRT with Avid Media Composer
Streaming SRT Directly from an Osprey Talon Encoder

Streaming SRT with OBS

For readers familiar with the Dolby.io platform you might know about our custom forked version of OBS designed to stream WebRTC natively. Although you can use our WebRTC-enabled OBS fork, you can actually publish SRT streams to the Dolby.io servers from the original OBS project. To do this you must have an active Dolby.io account, which you can create for free and the latest version of OBS installed on your system. To start publishing SRT streams with OBS follow the steps below:

1. Login or create a Dolby.io account and download OBS.

2. Navigate to your Dolby.io streaming dashboard and create a new token. You can leave all the token settings to default.

3. Open the API tab on your newly created token dashboard and navigate to the bottom where you'll see the SRT publish path, the SRT stream ID, and the SRT publish URL. Copy SRT publish URL.

The Dolby.io Streaming Token API tab. Highlighted box indicates the SRT publish URL used in OBS.

4. Open OBS and navigate to settings, then the Stream tab.

5. Inside of the Stream tab, set Service to Custom and Server to the SRT publish URL.

OBS stream settings page. Remember to set Service to "Custom" and Server to "Your SRT Publish URL".

6. Apply the changes and exit settings. You are now all set up to stream with OBS. When publishing, your SRT stream will be delivered to the Dolby.io Streaming Viewer, which can be found at the Hosted Player Path.

The Dolby.io Streaming Token API tab, with hosted player path highlighted. Opening this path in a browser will launch the stream.

Although the hosted player path is a great way to view the stream, you can use the Dolby.io Streaming JavaScript SDK to build a bespoke solution.

Note: If you are using the NVIDIA NVENC H.264 encoder that comes included with OBS you must set Max B-Frames to 0. This setting can be found in Output, then Advanced Output Mode, then the Streaming tab, where Encoder is set to NVIDIA NVENC H.264 and then Max B-frames is set to 0.

If you are using the NVIDIA NVENC H.264 encoder that comes included with OBS you must set Max B-Frames to 0. This setting can be found in Output, then Advanced Output Mode, then the Streaming tab, where Encoder is set to NVIDIA NVENC H.264 and then Max B-frames is set to 0.

Streaming SRT with vMix

vMix is a paid windows-only remote production tool used for vision mixing. It allows users to juggle input and outputs for live broadcasts and productions and includes support for publishing SRT streams. To publish an SRT stream with vMix follow the steps below:

1. Login or create a Dolby.io account.

2. Download and open vMix.

3. Navigate to your Dolby.io streaming dashboard and create a new token. You can leave all the token settings to default.

4. Open the API tab on your newly created token dashboard and navigate to the bottom where you'll see the SRT publish path, SRT stream ID, and the SRT publish URL. Copy the SRT publish path and the SRT stream ID.

5. Inside of vMix open settings and switch to Output / NDI / SRT.

The vMix mixing stage. Navigate to "Settings" and click on "Outputs / NDI / SRT" to open up the SRT settings menu.

6. Once you've switched to Output / NDI / SRT open the gear icon next to an output source.

Inside the SRT settings, select the gear icon highlighted in red.

7. Inside the output settings enable SRT, set the Hostname to the Dolby.io Millicast endpoint and the Port to the appropriate port (typically 10,000). Additionally, include the Stream ID and make sure the Quality settings match the limitations of Dolby.io SRT streaming.

When creating the SRT stream define Hostname, port, Stream ID, and Quality.

8. Press OK and exit settings. You are now all set up to stream with vMix. When streaming, your SRT stream will be delivered to the Dolby.io Streaming Viewer, which can be found at the Hosted Player Path.

The Dolby.io Streaming Token API tab, with hosted player path highlighted. Opening this path in a browser will launch the stream.

Although the hosted player path is a great way to view the stream, you can use the Dolby.io Streaming JavaScript SDK to build out a bespoke solution.

Streaming SRT with your iPhone

Softvelum's Larix Broadcaster is a tool available for iOS, Android, and React Native that allows you to push SRT streams directly from your mobile device. To set up a Larix SRT stream on an iOS device:

1. Login or create a Dolby.io account.

2. Download the Larix Broadcaster from the App Store.

3. Navigate to your Dolby.io streaming dashboard and create a new token. You can leave all the token settings to default.

4. Open the API tab on your newly created token dashboard and navigate to the bottom where you'll see the SRT publish path, the SRT stream ID, and the SRT publish URL. Copy the SRT publish path and the SRT stream ID.

5. Open the Larix Broadcaster and then Settings. From Settings, go to Connections and add a new connection.

Create a new connection with the plus icon in the top right corner.

6. Inside the connection, set the URL parameter to your Dolby.io Real-Time Streaming SRT publish path and set streamid to your SRT stream ID.

When adding a new connection in the Larix Broadcaster make sure to assign "streamid" and "URL".

7. From here you can exit your settings and start the stream by pressing the record button on the broadcaster.

Press the record button on the left to start an SRT stream.

8. Like the OBS and vMix examples, your SRT stream will be delivered to the Dolby.io Streaming Viewer, which can be found at the Hosted Player Path.

The Dolby.io Streaming Token API tab, with hosted player path highlighted. Opening this path in a browser will launch the stream.

Dolby.io Real-time Streaming supports a number of SDKs for creating viewer apps including a Flutter 3 SDK for creating viewer apps for Android, iOS, and Web.

Streaming SRT directly from an Osprey Talon Encoder

OBS, vMix, and Larix Broadcaster are examples of software tools that you can leverage for streaming SRT into the Dolby.io Streaming service, but what about hardware options? Depending on the scale of live production you might have access to cameras with built-in encoders that can directly egress SRT, which we can also connect to the servers. For cameras that don't have built-in encoders, you can connect the camera to an external encoder, some of which support SRT. One example of this is the Osprey Talon 4K-SC, which is not only the first WHIP encoder but can also encode SRT streams that we can connect to the Dolby.io servers.

1. Login or create a Dolby.io account.

2. Connect your Osprey Encoder to your camera and power it up.

3. Download the Osprey BOSS PRO application, which will allow you to discover the encoder on your local network. Alternatively, follow this in-depth guide by the Osprey team for setting up your encoder.

4. Click on the appropriate encoder, launch the web interface and sign in. Information regarding signing into Osprey equipment can be found here. Once signed in you will now be in the Osprey Dashboard.

5. Navigate to your Dolby.io streaming dashboard and create a new token. You can leave all the token settings to default.

6. Open the API tab on your newly created token dashboard and navigate to the bottom where you'll see the SRT publish path, the SRT stream ID, and the SRT publish URL. Copy the SRT publish path and the SRT stream ID.

The Dolby.io Streaming Token API tab. Highlighted box indicates the SRT publish URL used in OBS.

7. Inside the Osprey Dashboard, set SRT Dest Address to the SRT publish path excluding the port. Set SRT Port to the port number at the end of your SRT publish path (usually 10000) and set SRT Stream ID to your SRT Stream ID.

Set the SRT Dest Address to the SRT publish path, and SRT Port to 10000, and the SRT Stream ID to your Dolby.io Streaming Token Stream ID.

8. From here press start and the encoder will begin streaming content through the Dolby.io servers.

Limitations of Publishing SRT Streams to Dolby.io

Streaming SRT is just one part of the equation, Dolby.io Real-time Streaming also supports a number of SDKs for building streaming into your platforms and apps. If you are interested in learning more about how to use our SDKs check out our blog and let us know what you're building next.\
Feedback or Questions? Reach out to the team on Twitter, LinkedIn, or via our support desk.

Building a Livestream App with Flutter 3

Braden Riggs — Mon, 31 Oct 2022 20:19:09 +0000

Originally published here.

Streaming, especially the low latency kind, has become a popular medium to engage with an audience, host live events, and connect people virtually. For developers building streaming apps, however, there is just one issue. If we are interested in connecting to a wide audience we need to develop for a wide range of platforms such as Android, iOS, Web, and even desktop native apps, which can quickly become a heavy lift for any team. This is where Flutter 3 comes in, released in May of 2022, Flutter 3 takes cross-platform to the next level allowing users to "build for any screen" from a single code base. Hence, rather than building 3 separate apps for iOS, Android, and Web, you can build just one. To further sweeten the deal, Dolby.io has recently released their WebRTC real-time streaming SDK for Flutter, allowing users to build cross-platform streaming apps that combine scalability and ultra-low delay.

In this guide, we'll be exploring how to build a cross-platform real-time streaming app that works on Android, iOS, Desktop Native, and Web using the Dolby.io Streaming SDK for Flutter.

Getting Started with the Real-Time Streaming SDK

Before we begin you need to make sure you have the latest version of Flutter installed and set up on your machine. To get started with building a streaming app we need to install the Dolby.io Streaming SDK for Flutter 3 via the terminal.

flutter pub add millicast_flutter_sdk

Then run the following command in terminal to download the dependencies:

flutter pub get

With the Flutter Streaming SDK installed, you can start by creating a vanilla Flutter app and add the most recent version of flutter_webrtc to your project's pubspec.yaml. You should also see that the Dolby.io Millicast flutter SDK has been automatically added.

flutter_webrtc: ^x.x.x
millicast_flutter_sdk: ^x.x.x

Then inside main.dart you just import flutter_webrtc alongside any other dependencies your project may have.

import 'package:flutter_webrtc/flutter_webrtc.dart';

In addition to installing the SDK, you'll also need to create a free Dolby.io Account. The free account offers 50 Gigabytes of data transfer a month, which will be plenty for building and testing out the real-time streaming app.

Interested in following along with a project that already has the SDK installed and set up? Check out this GitHub repository which contains a completed version of this app.

Building the Real-Time Streaming App with Flutter

Building a WebRTC Flutter streaming app can be complicated, so to get started we first need to divide the app into a series of features that come together to support a real-time streaming experience. In order for the app to connect to the Dolby.io servers, we must include a way for the user to input their streaming credentials and tokens in order to authenticate and use the Dolby.io servers.

Taking in the WebRTC Stream Credentials

To publish and view a WebRTC stream with the Dolby.io Flutter SDK we need three things: an account ID, a stream name, and a publishing token. These credentials can be found on your Dolby.io dashboard and need to be input by the user which we can capture with the TextFormField widget, where the widget, on change, updates a TextEditingController variable.

Container(
    width: MediaQuery.of(context).size.width,
    constraints: const BoxConstraints(
        minWidth: 100, maxWidth: 400),
    child: TextFormField(
      maxLength: 20,
      controller: accID,
      decoration: const InputDecoration(
        labelText: 'Enter Account ID',
      ),
      onChanged: (v) => accID.text = v,
    )),

Note: In production, you don't need to have users input these credentials, instead you could use a custom login and serve the users a temporary login token. For learning more about Dolby.io tokens check out this blog on creating and securing tokens.

Because we need three inputs to publish a WebRTC stream to the Dolby.io server, we can repeat this code for each input.

Container(
     width: MediaQuery.of(context).size.width,
     constraints: const BoxConstraints(
         minWidth: 100, maxWidth: 400),
     child: TextFormField(
       maxLength: 20,
       controller: accID,
       decoration: const InputDecoration(
         labelText: 'Enter Account ID',
       ),
       onChanged: (v) => accID.text = v,
     )),
Container(
     width: MediaQuery.of(context).size.width,
     constraints: const BoxConstraints(
         minWidth: 100, maxWidth: 400),
     child: TextFormField(
       maxLength: 20,
       controller: streamName,
       onChanged: (v) => streamName.text = v,
       decoration: const InputDecoration(
         labelText: 'Enter Stream Name',
       ),
     )),
 // Publishing Token Input
 Container(
     width: MediaQuery.of(context).size.width,
     constraints: const BoxConstraints(
         minWidth: 100, maxWidth: 400),
     child: TextFormField(
       controller: pubTok,
       maxLength: 100,
       onChanged: (v) => pubTok.text = v,
       decoration: const InputDecoration(
         labelText: 'Enter Publishing Token',
       ),
     )),

Additionally, we can add an ElevatedButton for the user to press once they have added their credentials.

ElevatedButton(
  style: ElevatedButton.styleFrom(
    primary: Colors.deepPurple,
  ),
  onPressed: publishExample,
  child: const Text('Start Stream'),
),

Authentication and Publishing Streams from Flutter

You'll notice that the Elevated button triggers a function via its onPressed parameter. This function, called publishExample, checks if the credentials are valid and authenticates the stream. First, the function checks that a user has input a value for each input.

void publishExample() async {
    if (pubTok.text.isEmpty || streamName.text.isEmpty || accID.text.isEmpty) {
      ScaffoldMessenger.of(context).showSnackBar(const SnackBar(
        backgroundColor: Colors.grey,
        content: Text(
            'Make sure Account ID, Stream Name, and Publishing Token all include values.'),
      ));
    }

Then the function calls publishConnect, an asynchronous function that takes in streamName, pubTok, and a third object called localRenderer. localRendered is a RTCVideoRenderer object included with the flutter.webrtc package.

final RTCVideoRenderer localRenderer = RTCVideoRenderer();
publish = await publishConnect(localRenderer, streamName.text, pubTok.text);

Using these three parameters we have everything we need to authenticate and begin publishing a stream. Inside of the publishConnect function, we need to generate a temporary publishing token using the streamName and pubTok:

Future publishConnect(RTCVideoRenderer localRenderer, String streamName, String pubTok) async {
  // Setting subscriber options
  DirectorPublisherOptions directorPublisherOptions =
      DirectorPublisherOptions(token: pubTok, streamName: streamName);

  /// Define callback for generate new token
  tokenGenerator() => Director.getPublisher(directorPublisherOptions);

...
}

With the temporary publishing token created, we then can use it to create a publish object. Using this publish object we could start the stream, however, we wouldn't be able to see or hear anything, this is because we haven't specified what kind of stream we are creating or which devices we will connect to. To do this we need to specify if the stream will include audio, video, or audio and video, then we need to pass these constraints into the getUserMedia function which will map the constraints to the default audio capture device and the default video capture device.

{
...
Publish publish =
      Publish(streamName: 'your-streamname', tokenGenerator: tokenGenerator);

  final Map<String, dynamic> constraints = <String, bool>{
    'audio': true,
    'video': true
  };

  MediaStream stream = await navigator.mediaDevices.getUserMedia(constraints);

...
}

Using this stream object, we can also provide a feed to the user in the form of a viewer. To do this we need to assign our input devices to localRender as sources.

{
...

localRenderer.srcObject = stream;

...
}

Finally, we can map the stream object and pass it as an option to the connect function, which is inherited from publish.

{
...
//Publishing Options
  Map<String, dynamic> broadcastOptions = {'mediaStream': stream};

  /// Start connection to publisher
  await publish.connect(options: broadcastOptions);
  return publish;
}

With our stream connected, we can now look at setting up the viewer using localRender.

In-App WebRTC Stream Viewer

Now that our stream is authenticated and publishing we need to add a viewer object so the streamer can see themselves streaming. This can be done with a RTCVideoView object which takes in our localRender object and is constrained by a container.

Container(
  margin: const EdgeInsets.all(30),
  constraints: const BoxConstraints(
      minWidth: 100, maxWidth: 1000, maxHeight: 500),
  width: MediaQuery.of(context).size.width,
  height: MediaQuery.of(context).size.height / 1.7,
  decoration:
      const BoxDecoration(color: Colors.black54),
  child: RTCVideoView(localRenderer, mirror: true),
)

Sharing the Real-time Stream

With the stream authenticated and live, we want to share our content with the world. We can do this via a URL formatted with our streamName and our accountID which we collected as inputs. Using the example app as a template we can create a function called shareStream which formats the URL to share and copies it to the clipboard.

void shareStream() {
    Clipboard.setData(ClipboardData(
        text:
            "https://viewer.millicast.com/?streamId=${accID.text}/${streamName.text}"));
    ScaffoldMessenger.of(context).showSnackBar(const SnackBar(
      backgroundColor: Colors.grey,
      content: Text('Stream link copied to clipboard.'),
    ));
  }

Unpublishing a WebRTC Stream

To unpublish the stream we can call the publish object returned from our asynchronous publishConnect function to stop, killing the connection with the Dolby.io server.

publish.stop();

Flutter 3 is Truly Cross Platform

The power of Flutter is taking one code base and having it work across multiple platforms. Here we can see examples of the app working for Android, Windows, and Web:

Building in this cross-platform framework saves both time and resources, allowing you to get started building real-time streaming apps without having to worry about which platform works for your users. These apps are perfect for streaming live events and virtual events to the widest range of audiences allowing for high-quality interactive experiences. If you are interested in learning more about our Flutter streaming SDK check out our documentation and play around with the full project on this GitHub repository.

Feedback or Questions? Reach out to the team on Twitter, LinkedIn, or via our support desk.

Installing Unreal Engine Plugins from GitHub or Source Code

Braden Riggs — Tue, 26 Jul 2022 17:14:00 +0000

Whether you are just getting started with the Unreal Engine or have been using it for years, Plugins are a great way to enhance your creation and reduce the development time across a project. Although many useful plugins come preinstalled on your system, your project may require more bespoke and advanced tools that aren’t included by default and need to be installed from a 3rd party host like GitHub.

How to install 3rd Party Plugins from GitHub or Source Code

To get started make sure you have the Unreal Engine installed along with Visual Studio. Make sure that you also have Visual Studio set up for game development with C++.
Create a project within the Unreal Engine editor, then save and close the program.
Locate the directory of your newly created project. For example C:\Users\User\Unreal Engine\MyProject
Create a “Plugins” folder and place the source code into the new directory. For example C:\Users\User\Unreal Engine\MyProject\Plugins
Navigate back to the top level of your project and right-click on the .uproject file. Select Generate Visual Studio Project.
Reopen the project with the Unreal Engine editor and Navigate to Edit>Plugins and enable (if already enabled don’t worry) your new plugin.

From there the engine should restart and you should have your newly installed plugin ready to use. If you are interested in an example of a plugin your might want to install from a 3rd party, check out the Dolby.io Millicast WebRTC plugin for creating WebRTC pixel streams.

Building a Livestream Platform

Braden Riggs — Thu, 30 Jun 2022 15:58:12 +0000

With competition rising in the live streaming space, companies are battling it out to offer more compelling streaming experiences, whether that is for live events, e-learning, or remote post-production. In order to stand out, solutions must balance offering high-quality audio and video in addition to offering optimal content delivery speeds, leading platforms such as Twitch and YouTube Live to sacrifice real-time (10-15 seconds of latency) in favor of quality. But what if you didn't have to sacrifice at all, and could have both quality and speed? In this guide, we'll explore building a webRTC low latency streaming platform capable of delivering 4k streams with just a few lines of JavaScript.

Building a Low Latency Livestreaming platform with Dolby.io Millicast and JavaScript.

To get started building your live stream platform, we first have to set up a free Dolby.io Millicast account here. Dolby.io Millicast is an ultra-low latency platform that provides APIs, SDKs, and integrations for content delivery with a delay of 500 milliseconds or less to anywhere in the world using a technology called WebRTC. The free account is hard-capped at 50 Gigabytes of data transfer a month, which will be plenty for building and testing your JavaScript Livestream platform.

The Millicast webRTC Javascript SDK

To get started, clone the GitHub repo, which contains an example app demonstrating how to implement everything you'll need for building a Millicast WebRTC Livestream viewing platform.

The landing page for the Livestream viewer where users are required to input the stream name and account ID.

Once users log into the Livestream viewer they connect to a low-latency webRTC stream powered by Dolby.io Millicast.

To test out this app, we need to navigate to the dashboard of your newly created Millicast account. There you'll see a header, *Stream Tokens, *which includes a list of your streaming tokens. If you've just created an account you'll only have one token, which you can click to open its settings.

The Millicast dashboard landing page.

Clicking on the token name brings you into the token settings where you can create a stream.

From Settings switch from Token Details to the API tab. In this tab, you'll see lots of different tokens, endpoints, and information. We can disregard most of the info for this app and instead just copy the Account ID value and the Stream Name. Next, switch back to the Token Details tab and click on the bright green Broadcast button. This will launch a WebRTC streaming tool where you can adjust settings and start streaming, although no one will be able to join when you do until you have a working stream viewer. For now, click on the start stream button to begin a Livestream. Finally, we can launch our cloned sample app by either using the Live Server extension for VS Code or just opening the *index.html *file in browser. With the sample app launched, you can connect to your Livestream by entering the Account ID and the Livestream Name.

Note: it's important to follow the best security practices when sharing or exposing any IDs or authentication parameters on the web.

To Summarize

Clone the Livestream sample project.
From your Millicast account locate your Stream Token Settings and get your Account ID and Livestream Name from the API tab.
Start broadcasting a Livestream from the Millicast dashboard.
Launch your sample app by opening index.html in browser.
Enter the Account ID and Livestream Name to view your stream.
Watch your Livestream!

What's cool about this sample app is, although it is simple to understand, the app could be hosted on a server and shared with anyone across the web, allowing them to watch what you're streaming from the Millicast dashboard.

So How Does the Millicast Livestream Viewer Work?

Millicast supports a JavaScript SDK which you can utilize as either an NPM package or with the JSDELIVR content delivery network (CDN). For this guide, we will be using the CDN option which we import into the head of our HTML file.

<script src="https://cdn.jsdelivr.net/npm/@millicast/sdk@latest/dist/millicast.umd.js"></script>

This import allows us to utilize the tools offered in the SDK. We also need to include a video *tag and another script tag linking our HTML file to millicast_viewer.js*, otherwise, the rest of the HTML file is set dressing.

<video width="640" height="360" hidden="True" id="videoPlayer" controls class="vidBox"></video>
<script src="src/millicast_viewer.js"></script>

With the Millicast SDK imported, our video player placed, and our HTML and JavaScript files linked, we can change over to millicast_viewer.js to learn about how we authenticate and connect to a Livestream.

There are three main steps for authenticating and connecting to a Millicast broadcast inside of millicast_viewer.js.

Step #1: Token Generator

The first step in connecting to a Millicast stream is to define our token generator. Since we imported the SDK via a CDN we need to preface Millicast SDK functions with window.millicast, then the function, in this case, Director.getSubscriber. Additionally, for the token generator, we also have to include the Stream Name and the Account ID. In our example we utilize user input to get these values, however, there are a variety of different options for generating a Millicast token. Using our token generator, we can create a Millicast View option which we will use to connect to the broadcast.

const tokenGenerator = () =>
        window.millicast.Director.getSubscriber({
            streamName: streamName,
            streamAccountId: accID,
        });
const millicastView = new window.millicast.View(streamName, tokenGenerator);

Step #2: Adding the Stream to Video

Next, we need to add the stream to our video tag, defined in the HTML. To do this we define a function called addStreamToYourVideoTag and listen for a track event, such as a broadcast updating, to add to our video tag.

millicastView.on("track", (event) => {
        addStreamToYourVideoTag(event.streams[0]);
    });

As for our function addStreamToYourVideoTag, it just takes in the stream element and sets our <video> tag's srcObject equal to it.

function addStreamToYourVideoTag(elem) {
    //Adds Stream to the <video> tag.
    let video = document.getElementById("videoPlayer");
    video.srcObject = elem;
    video.autoplay = true;
}

Step #3: Connecting to the Stream

The final step in building our Livestream viewer is to specify a few stream settings and connect to the broadcast. First, we define an options object which contains three main parameters:

disableVideo: set to false because we want video-enabled.
disableAudio: set to false because we want audio enabled.
bandwidth: used to set data transfer limits, however, we set it to zero in this case which means unlimited bandwidth, since we are only testing the app.

const options = {
        disableVideo: false,
        disableAudio: false,
        bandwidth: 0,
    };

Next, we can call the connect function using our millicastView object and our options object. We surround this function with a try-catch statement in case bandwidth issues prevent the broadcast from connecting immediately.

try {
        await millicastView.connect(options);
    } catch (e) {
        console.log("Connection failed, handle error", e);
        millicastView.reconnect();
    }

With the connect step complete, we are now able to join a Millicast broadcast. We can see all these steps put together here and the result is a Livestream viewing platform powered by WebRTC.

Once users log into the Livestream viewer they connect to a low-latency webRTC stream powered by Dolby.io Millicast.

Final Thoughts on Building the Millicast Livestream App

In this guide, we've outlined the fundamentals for building a low latency Livestream viewer in JavaScript that's powered by WebRTC. If you've been following along with the sample project code you'll notice a few extra parts and tools. For the most part, the additional code serves as set dressing, helping make the app look and function in a way you might expect a Livestream viewer to function, however, it is not core to connecting to a Millicast broadcast. As you play around and build with the SDK more you might find better ways to create and style your app for your personal goals so don't be afraid to play around with the code a bit. If you have any questions or are interested in learning more about Dolby.io Millicast feel free to reach out to our support or check out our documentation guides, which include many more SDKs and tools such as a Millicast OBS integration or an Unreal Engine Plugin.

3 Things To Know Before Building with PyScript

Braden Riggs — Thu, 26 May 2022 16:57:13 +0000

For anyone who hasn't already heard PyScript, which debuted at PyCon 2022, is a browser-embedded python environment, built on top of an existing project called Pyodide. This project, to the shock of long-term Pythonistas and web developers, seamlessly blends (well almost) JavaScript and Python in a bi-directional environment allowing developers to utilize Python staples such as NumPy or Pandas in the browser.

After playing with the project for a few days I wanted to share a few learnings and gotchya's that tripped me up on my journey to master PyScript.

Prelude: A Crash Course in PyScript
1. Package Indentation Matters!
2. Local File Access
3. DOM Manipulation

A Crash Course in PyScript

To get started using PyScript we first have to link our HTML file with the PyScript script as we would for any ordinary javascript file. Additionally, we can link the PyScript style sheet to improve usability.

<head>
    <link rel="stylesheet" href="https://pyscript.net/alpha/pyscript.css" />
    <script defer src="https://pyscript.net/alpha/pyscript.js"></script>
</head>

With PyScript imported in the head of our HTML file, we can now utilize the tag in the body of our HTML to write python code.

<body>
    <py-script>
        for i in ["Python", "in", "html?"]:
            print(i)
    </py-script>
</body>

Yep! It really is just that simple to get started. Now, where do things get tricky?

Package Indentation Matters

One of the big advantages of using PyScript is the ability to import Python libraries such as NumPy or Pandas, which is first done in the Head using the tag and then inside of the tag just like you would in regular Python.

<head>
    <link rel="stylesheet" href="https://pyscript.net/alpha/pyscript.css" />
    <script defer src="https://pyscript.net/alpha/pyscript.js"></script> <py-env>
- numpy
- pandas
    </py-env>
</head><body>
    <py-script>
        import pandas as pd
    </py-script>
</body>

On the surface, this may seem straightforward but note the indentation of the packages within .

 <py-env>
- numpy
- pandas
    </py-env>

Turns out that if there is any indentation you'll receive a ModuleNotFoundError: No module named 'pandas' *or*ModuleNotFoundError*: No module named 'numpy' ) *for PyScript. This error caught me off guard initially since indentation in Python is so important.

Local File Access

JavaScript handles file access very differently compared to Python... as it should given the relationship between web development and privacy and security. Hence Vanilla JavaScript does not have direct access to local files. Since the PyScript project is built on top of JavaScript your Python code won't be able to access local files like you might be used to.

PyScript does offer a solution to file access in the tag. In addition to importing packages, you can also import files such as CSVs or XLSXs.

 <py-env>
- numpy
- pandas
- paths:
    - /views.csv
    </py-env>

Again note the indentation as in this case the CSV must be indented in relation to the paths.

With the file included in the path, you can read it within your code.

<py-script>
    import pandas as pd
    df = pd.read_csv("views.csv")
</py-script>

DOM Manipulation

For anyone who has worked in web development, you should be familiar with the DOM or Document Object Model. DOM Manipulation is common in most web applications as developers typically want their websites to interact with users, reading inputs and responding to button clicks. In the case of PyScript this raises an interesting question how do buttons and input fields interact with the Python code?

Again PyScript has a solution to this, however, it mightn't be what you expect. Here are a few (of many) examples where PyScript has functionality:

For buttons, you can include pys-onClick="your_function" parameter to trigger python functions when clicked.
For retrieving user input from within the tag *document.getElementById('input_obj_id').value *can retrieve the input value.
And Finally pyscript.write("output_obj_id", data) *can write output to a tag from within the * tag.

We can see these three DOM manipulation techniques put together into one web application that lets users check if a CSV has been added to the PyScript path:

<body>
   <form onsubmit = 'return false'>
   <label for="fpath">filepath</label>
   <input type="text" id="fpath" name="filepath" placeholder="Your name..">
   <input pys-onClick="onSub" type="submit" id="btn-form" value="submit">
    </form><div id="outp"></div> <py-script>
        import pandas as pd def onSub(*args, **kwargs):
            file_path = document.getElementById('fpath').value
            df = pd.read_csv(file_path)
            pyscript.write("outp",df.head())
    </py-script>
</body>

These examples aren't comprehensive as the project also supports visual component tags.

Conclusion

PyScript is a wonderful step in the right direction for bringing some excellent Python packages into the web development space. With that said it still has a bit of growing to do and there are many improvements that need to be made before the project sees widespread adoption.

Show some support to the team working on this awesome project: https://github.com/pyscript

Leave a comment with any other insights or gotchya's that you might have experienced working with PyScript and I'll make a part 2.

Searching Media to Find Loudness and Music Sections

Braden Riggs — Thu, 19 May 2022 17:01:08 +0000

Ever since the inception of cinema, Scores, the musical composition within a film, have become synonymous with the medium and a crucial staple in the experience of enjoying film or TV. As the industry has grown and matured so too has the score, with many productions having hundreds of tracks spanning many genres and artists. These artists can be anyone from an orchestra drummer all the way up to a sellout pop star sensation, each composing, producing, or performing a variety of tracks. The challenge with this growing score complexity is ensuring that every artist is paid for their fair share and contribution to the overall film.

The industry presently tackles this challenge with a tool known as a "Cue Sheet", a spreadsheet that identifies exactly where a track is played and for how long. The issue with Cue Sheets is that their creation and validation is an immensely manual process, constituting hundreds of hours spent confirming that every artist is accounted for and compensations are awarded accordingly. It was this inefficiency that attracted Dolby.io to help support the Cue Sheet Palooza Hackathon, a Toronto-based event that challenged musicians and software engineers to work and innovate together to reduce the time spent creating Cue Sheets. The event was sponsored by the Society of Composers, Authors and Music Publishers of Canada or SOCAN for short, which is an organization that helps ensure Composers, Authors, and Music Publishers are correctly compensated for their work.

Many of the hackers utilized the Dolby.io Analyze Media API to help detect loudness and music within an audio file and timestamp exactly where music is included. In this guide, we will highlight how you can build your own tool for analyzing music content in media, just like the SOCAN hackathon participants.

So what is the Analyze Media API?

Before we explained how hackers used the API we need to explain what Analyze Media is and what it does. The Dolby.io Analyze Media API generates insight based on the underlying signal in audio for creating data such as loudness, content classification, noise, and musical instruments or genre classification. This makes the API useful for detecting in what section music occurs in a media file and some qualities of the music at that instance.

The Analyze Media API adheres to the Representational state transfer (REST) protocol meaning that it is language-agnostic and can be built into an existing framework that includes tools to interact with a server. This is useful as it means the API can adapt depending on the use case. In the Cue Sheet example many teams wanted to build a web application as that was what was most accessible to the SOCAN community, and hence relied heavily on HTML, CSS, and JavaScript to build out the tool.

In this guide, we will be highlighting how the participants implemented the API and why it proved useful for video media. If you want to follow along you can sign up for a free Dolby.io account which includes plenty of trial credits for experimenting with the Analyze Media API.

A QuickStart with the Analyze Media API in JavaScript:

There are four steps to using the Analyze Media API on media:

Store the media on the cloud.
Start an Analyze Media job.
Monitor the status of that job.
Retrieve the result of a completed job.

The first step, storing media on the cloud depends on your use case for the Analyze Media API. If your media/video is already stored on the cloud (Azure AWS, GCP) you can instead move on to step 2. However, if your media file is stored locally you will first have to upload it to a cloud environment. For this step, we upload the file to the Dolby.io Media Cloud Storage using the local file and our Dolby.io Media API key.

async function uploadFile() {
    //Uploads the file to the Dolby.io server
    let fileType = YOUR_FILE_TYPE;
    let audioFile = YOUR_LOCAL_MEDIA_FILE;
    let mAPIKey = YOUR_DOLBYIO_MEDIA_API_KEY;

    let formData = new FormData();
    var xhr = new XMLHttpRequest();
    formData.append(fileType, audioFile);

    const options = {
        method: "POST",
        headers: {
            Accept: "application/json",
            "Content-Type": "application/json",
            "x-api-key": mAPIKey,
        },
        // url is where the file will be stored on the Dolby.io servers.
        body: JSON.stringify({ url: "dlb://file_input.".concat(fileType) }),
    };

    let resp = await fetch("https://api.dolby.com/media/input", options)
        .then((response) => response.json())
        .catch((err) => console.error(err));

    xhr.open("PUT", resp.url, true);
    xhr.setRequestHeader("Content-Type", fileType);
    xhr.onload = () => {
        if (xhr.status === 200) {
            console.log("File Upload Success");
        }
    };
    xhr.onerror = () => {
        console.log("error");
    };
    xhr.send(formData);
    let rs = xhr.readyState;

    //Check that the job completes
    while (rs != 4) {
        rs = xhr.readyState;
    }
}

For this file upload, we have chosen to use XMLHttpRequest for handling our client-side file upload, although packages like Axios are available. This was a deliberate choice as in our Web App we add functionality for progress tracking and timeouts during our video upload.

With our media file uploaded and stored on the cloud we can start an Analyze Media API job which is done using the location of our cloud-stored media file. If your file is stored on a cloud storage provider such as AWS you can use the pre-signed URL for the file as the input. In this example, we are using the file stored on Dolby.io Media Cloud Storage from step 1.

async function startJob() {
    //Starts an Analyze Media Job on the Dolby.io servers
    let mAPIKey = YOUR_DOLBYIO_MEDIA_API_KEY;
    //fileLocation can either be a pre-signed URL to a cloud storage provider or the URL created in step 1.
    let fileLocation = YOUR_CLOUD_STORED_MEDIA_FILE;

    const options = {
        method: "POST",
        headers: {
            Accept: "application/json",
            "Content-Type": "application/json",
            "x-api-key": mAPIKey,
        },
        body: JSON.stringify({
            content: { silence: { threshold: -60, duration: 2 } },
            input: fileLocation,
            output: "dlb://file_output.json", //This is the location we'll grab the result from.
        }),
    };

    let resp = await fetch("https://api.dolby.com/media/analyze", options)
        .then((response) => response.json())
        .catch((err) => console.error(err));
    console.log(resp.job_id); //We can use this jobID to check the status of the job
}

When startJob resolves we should see a job_id returned.

{"job_id":"b49955b4-9b64-4d8b-a4c6-2e3550472a33"}

Now that we've started an Analyze Media job we need to wait for the job to resolve. Depending on the size of the file the job could take a few minutes to complete and hence requires some kind of progress tracking. We can capture the progress of the job using the JobID created in step 2, along with our Media API key to track the progress of the job.

async function checkJobStatus() {
    //Checks the status of the created job using the jobID
    let mAPIKey = YOUR_DOLBYIO_MEDIA_API_KEY;
    let jobID = ANALYZE_JOB_ID; //This job ID is output in the previous step when a job is created.

    const options = {
        method: "GET",
        headers: { Accept: "application/json", "x-api-key": mAPIKey },
    };

    let result = await fetch("https://api.dolby.com/media/analyze?job_id=".concat(jobID), options)
        .then((response) => response.json());
    console.log(result);
}

The checkJobStatus function may need to be run multiple times depending on how long it takes for the Analyze Media job to resolve. Each time you query the status you should get results where progress ranges from 0 to 100.

{
  "path": "/media/analyze",
  "status": "Running",
  "progress": 42
}

Once we know the job is complete we can download the resulting JSON which contains all the data and insight generated regarding the input media.

async function getResults() {
    //Gets and displays the results of the Analyze job
    let mAPIKey = YOUR_DOLBYIO_MEDIA_API_KEY;

    const options = {
        method: "GET",
        headers: { Accept: "application/octet-stream", "x-api-key": mAPIKey },
    };

    //Fetch from the output.json URL we specified in step 2.
    let json_results = await fetch("https://api.dolby.com/media/output?url=dlb://file_output.json", options)
        .then((response) => response.json())
        .catch((err) => console.error(err));

    console.log(json_results)
}

The resulting output JSON includes music data which breaks down by section. These sections contain an assortment of useful data points:

Start (seconds): The starting point of this section.
Duration (seconds): The duration of the segment.
Loudness (decibels): The intensity of the segment at the threshold of hearing.
Beats per minute (bpm): The number of beats per minute and an indicator of tempo.
Key: The pitch/scale of the music segment along with a confidence interval of 0.0-1.0.
Genre: The distribution of genres including confidence intervals of 0.0-1.0.
Instrument: The distribution of instruments including confidence intervals of 0.0-1.0.

Depending on the complexity of the media file there can sometimes be 100s of music segments.

"music": {
    "percentage": 34.79,
    "num_sections": 35,
    "sections": [
        {
            "section_id": "mu_1",
            "start": 0.0,
            "duration": 13.44,
            "loudness": -16.56,
            "bpm": 222.22,
            "key": [
                [
                    "Ab major",
                    0.72
                ]
            ],
            "genre": [
                [
                    "hip-hop",
                    0.17
                ],
                [
                    "rock",
                    0.15
                ],
                [
                    "punk",
                    0.13
                ]
            ],
            "instrument": [
                [
                    "vocals",
                    0.17
                ],
                [
                    "guitar",
                    0.2
                ],
                [
                    "drums",
                    0.05
                ],
                [
                    "piano",
                    0.04
                ]
            ]
        },

This snippet of the output only shows the results as they relate to the Cue Sheet use case, the API generates even more data including audio defects, loudness, and content classification. I recommend reading this guide that explains in-depth the content of the output JSON.

With the final step resolved we successfully used the Analyze Media API and gained insight into the content of the media file. In the context of the Cue Sheet Palooza Hackathon, the participants were only really interested in the data generated regarding the loudness and music content of the media and hence filtered the JSON to just show the music data similar to the example output.

Building an app for creating Cue Sheets

Of course not every musician or composer knows how to program and hence part of the hackathon was building a user interface for SOCAN members to interact with during the Cue Sheet creation process. The resulting apps used a variety of tools including the Dolby.io API to format the media content data into a formal Cue Sheet. These web apps took a variety of shapes and sizes with different functionality and complexity.

It's one thing to show how the Analyze Media API works but it's another thing to highlight how the app might be used in a production environment like for a Cue Sheet. Included in this repo here is an example I built using the Analyze Media API that takes a video and decomposes the signal to highlight what parts of the media contain music.\
Here is a picture of the user interface, which takes in your Media API Key and the location of a locally stored media file.

The starting screen of the Dolby.io Analyze API Music Data Web app, found here:https://github.com/dolbyio-samples/blog-analyze-music-web

For showcasing the app I used a downloaded copy of a music review podcast where the host samples a range of songs across a variety of genres. The podcast includes 30 tracks which are played over 40% of the 50-minute podcast. If you want to try out the App with a song you can use the public domain version of "Take Me Out to the Ball Game" originally recorded in 1908, which I had used for another project relating to music mastering.

The Dolby.io Analyze API Music Data Web app after running the analysis on a 50-minute music podcast.

Feel free to clone the repo and play around with the app yourself.

Conclusion:

At the end of the hackathon, participating teams were graded and awarded prizes based on how useful and accessible the Cue Sheet tool would be for SOCAN members. The sample app demoed above represents a very rudimentary version of what many of the hackers built and how they utilized the Analyze Media API. If you are interested in learning more about their projects the winning team included a GitHub repo with their winning entry where you can see how they created a model to recognize music and how they used the Dolby.io Analyze Media API to supplement the Cue Sheet creation process.

If the Dolby.io Analyze Media API is something you're interested in learning more about check out our documentation or explore our other tools including APIs for Algorithmic Music Mastering, Enhancing Audio, and Transcoding Media.

Beginner’s Guide to Diagnosing Audio Issues as Part of an Azure Serverless Media Workflow

Braden Riggs — Wed, 13 Apr 2022 15:14:48 +0000

Transcribing media is a resource-intensive process that is dependent on the quality of the audio and background noise, meaning it can often produce inconsistent results as a product of the media quality. Depending on the scale of the media you are transcribing this process can be both computationally and monetarily expensive, being a costly endeavor especially if the results end up being inaccurate and noisy. To alleviate some of the risks of transcribing audio that might be low quality, we can instead turn our transcription tool into a workflow that gauges the quality of the audio and only transcribes it if the audio is high enough quality to produce accurate results. Furthermore, because media data can vary dramatically in size we can develop this workflow on the cloud to ensure our processing requirements are dynamically adjusted, and avoid storing large quantities of data among other things.

Part #1: Getting your environment ready for Serverless development.

To get started we first need to set up our development environment. For users unfamiliar with Azure services it is worth noting that Azure has a pay-as-you-go policy with some resources costing more than others, so we recommend that you check out the pricing guide here before leveraging any of their services. Follow the steps below to get the appropriate credentials and environment set up:

Create a free Dolby.io account here.
Additionally, you'll need to create a free Azure account
1. With this free Azure account create a container and a storage blob for your media file here.
2. Also with your free Azure account create a cognitive services account here.
To get your Azure environment set up, Microsoft has a handy guide for building your first project. Make sure you are using Python 3.9 and Azure Functions Core Tools 3.x.

With the environment correctly set up, you can now clone our sample project from GitHub. The sample project, presented at the 2021 Azure Serverless Conference, is a basic example of an asynchronous and fully serverless media to transcription workflow. To best understand how serverless media workflows work we will use this sample project as a template.

Part #2: Why Azure Serverless?

Before we dive into the workflow let's briefly outline what Azure Serverless Functions are. Serverless functions are a service provided by Azure that allows for event-triggered code to be run without the need for personal servers or infrastructure. Instead, users can develop code such as a media workflow and allow it to run on the cloud in response to an event. These events are referred to as *Triggers *which are the starting line for any Serverless process. The most basic trigger to imagine is an *HTTP *trigger, which would kickstart a Serverless event if a user navigated to a specific URL from their local machine.

Serverless functions are perfect for building a media workflow as they can be developed to function asynchronously, meaning multiple jobs can be run at once, with the associated costs scaling according to usage. This means you can build a workflow that you only use once a month or a thousand times a day, only paying for what you use.

Part #3: Understanding media workflows with Azure Serverless.

Now that we understand the basics of Azure Serverless Functions let's take a look at the sample project we cloned from GitHub. What does this workflow look like?

A depiction of the serverless media workflow available at https://github.com/dolbyio-samples/media-azure-serverless-workflow

As outlined above our workflow must begin with a trigger, in this case, it is a basic *HTTP *trigger. There are two main paths the workflow follows dependent on the quality of the audio in the media provided. In this example, there are three main tools that the workflow relies on to ensure that the media will produce a sufficiently accurate transcription. In the event that the media will not produce an accurate transcription, the audio for the media is cleaned and returned to allow for manual review.

Dolby.io Audio Diagnose: A lightweight audio analysis tool that can return information relating to the audio such as its quality or audio defects.
Azure Cognitive Services Speech-to-Text: A transcription tool that converts spoken audio to text.
Dolby.io Enhance: An audio enhancement tool that can help reduce noise and level speakers.

Before we test out the sample project let's briefly discuss how we connect each tool together in a serverless fashion.

Part #4: Making things asynchronous.

One of the challenges of developing a Serverless media workflow is planning for asynchronous deployment, meaning developing in a way that allows multiple instances of the workflow to run without them interfering or slowing each other down. We already discussed triggers which are great for starting the workflow but what are some other useful tricks we can include to keep things running smoothly. One useful trick is including callbacks *in our API calls. Because we are using REST APIs, the actual processing is done on a different server. In the case of the Dolby.io APIs, this processing is done on the Dolby.io Servers, whereas the Speech-to-Text processing is done on the Azure Cognitive Services server. When we send the job to these servers we can include a *callback parameter that signals where the API should send a request once the job is done. Since we are using *HTTP *triggers, we can specify that the callback directs to the trigger. We don't want the workflow to begin at the start, so when we callback to the *HTTP *trigger we include a job= tag in the request destination. An example of this process for diagnosis looks can be seen below:

import requests                               #Python module for making http requests
url = "https://api.dolby.com/media/diagnose"  #Location to send the request

body = {"input" : presignedURL,
    'on_complete': {'url': httpTriggerURL + "?job=diagnose_success", "headers": ["x-job-id"]}
}

headers = {"x-api-key":API_KEY,"Content-Type": "application/json","Accept": "application/json", "x-job-id":"True"}
response = requests.post(url, json=body, headers=headers)
response.raise_for_status()

Included in the body of the call is an on_complete tag which includes the URL location of the trigger, plus a tag informing the function that the next step it should take is evaluating the results of diagnose. This process allows the function to be freed up as soon as it submits a diagnose job, sitting idle until it has to start another diagnose job or until that diagnose job completes. Structured like this the function is able to handle many requests sending media through the workflow.

The second trick for keeping things asynchronous and efficient is by not moving the media. This means we don't have to change where files are stored, rather we can keep them stored on the Azure cloud in the blob storage and pass directions to each of our tools so they can access where the files are stored. This is done through the use of the pre-signed URLs a useful concept that you can read about in more detail here. Pre-signed URLs allow the appropriate credentials and access to be passed in one simple URL which the Dolby.io server or the Cognitive Services server can use to find the media. An example of creating this URL can be seen below:

from datetime import datetime, timedelta
from azure.storage.blob import generate_blob_sas, BlobSasPermissions

input_sas_blob = generate_blob_sas(account_name= AZURE_ACCOUNT_NAME,
                                    container_name= AZURE_CONTAINER,
                                    blob_name= AZURE_BLOB_INPUT_NAME,
                                    account_key= AZURE_PRIMARY_KEY,

                                    #Since we aren't editing the file read access is sufficient
                                    permission=BlobSasPermissions(read=True),
                                    expiry=datetime.utcnow() + timedelta(hours=5)) #SAS will expire in 5 hours

input_url = 'https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_BLOB_INPUT_NAME+'?'+input_sas_blob

The generate_blob_sas function creates a Shared Access Signature, which when formatted correctly into a URL, creates a direct path to the media.

The three tools briefly mentioned earlier each are connected to create our workflow.

Dolby.io Audio Diagnose: A lightweight audio analysis tool that can return information relating to the audio such as its quality or audio defects.
Azure Cognitive Services Speech-to-Text: A transcription tool that converts spoken audio to text.
Dolby.io Enhance: An audio enhancement tool that can help reduce noise and level speakers.

We start with diagnose, which using the pre-signed URL can access the file stored on the cloud to evaluate its audio quality. It then returns via a callback a score out of 10 with 1 being low-quality audio and 10 being high-quality audio. If the score is at or above our threshold of 7 we will transcribe it, otherwise, we will clean up the audio with enhance for manual review. We settled on this threshold of 7 after some testing as audio that scores above this threshold usually performs the best with Speech-to-Text.

Now that we understand the workflow, and the pieces that connect to make it work, let's test out the sample project.

Part #5: Testing out the workflow.

With the workflow set up, we can now test it out one of two ways. We recommend testing locally before deploying it with Serverless as when developing locally you can make simple and fast changes to your code without having to worry about uploading it to the Azure servers. Additionally, when developing locally you don't run the risk of exposing API keys. When deploying to the Azure cloud make sure you review the Authentication Guide to be sure you are following best practices and protecting your keys.

Locally:

To test locally we need to create a HTTP tunnel to port forward to localhost. There are many options available, for this example I used a free ngrok account. In my case my local function was deployed on 7071, so I initialized ngrok to port forward on LocalHost:7071.
Once you have launched your HTTP tunnel its time to update the params.json file. Include the correct API keys along with the appropriate names for your Azure account and container want to search for the file in. Additionally you can adjust the score threshold and the output suffix. With our HTTP tunnel launched we also need to update the tunneling URL with the appropriate Forwarding address.
In Visual Studio Code, press F5 to launch the project and navigate to: https://localhost:"YOUR_SERVER"/api/MediaProcessingWorkflow?input_file="YOUR_INPUT_FILE"

You can monitor your console in Visual Studio for output logs and your container on the Azure portal for the transcribed results.

Serverless:\
Deploying the functions to an Azure Server is a great choice once the code has been fully debugged and tested, just remember to protect your API keys with the appropriate authentication strategy.

Adjust the params.json file, this time set the tunneling parameter to https://"YOUR_FUNCTION_APP_NAME".azurewebsites.net/api/"YOUR_FUNCTION_NAME"
Next deploy the function app through the Azure extension.
Once successfully deployed you can test your function by navigating to: https://"YOUR_FUNCTION_APP_NAME".azurewebsites.net/api/"YOUR_FUNCTION_NAME"?input_file="YOUR_INPUT_FILE"

Part #6 Building your own workflow.

With the function successfully deployed our transcription workflow tool is complete.

A depiction of the serverless media workflow available at https://github.com/dolbyio-samples/media-azure-serverless-workflow

To review, our media follows a two-step workflow. From the Azure container, the file is diagnosed to return an audio quality score. If the audio scores below a quality threshold of 7 we opt not to transcribe the audio and instead enhance the audio. If the media scores at or above that threshold we then pass the file onto the next stage for transcription. Once the transcription is complete the file is then deposited on the container.

This example serves as an introduction to building a Serverless media workflow on Azure. By using the tips described above it is possible to add any number of extra steps into the workflow including APIs that check for profanity or APIs that translate foreign languages, the possibilities are endless. If you are interested in learning more about the workflow outlined above to check out the team's presentation at the 2021 Azure Serverless Conference where we go into more detail otherwise feel free to explore some of our other great projects over at the Dolby.io Samples Github.

"Take Me Out to the Ball Game" Algorithmically Remastered in Python

Braden Riggs — Sat, 09 Apr 2022 19:54:55 +0000

The 1908 Jack Norworth and Albert Von Tilzer song "Take Me Out to the Ball Game" has been a classic staple across ballparks becoming synonymous with America's favorite pastime. At over 114 years old, the original version of "Take Me Out to the Ball Game" was recorded on a two-minute Edison Wax Cylinder by singer and performer Edward Meeker, quickly becoming a beloved classic. With the baseball season getting underway this April 7th we thought it about time to dust off the gloves, pick up our bats, step up to our Python environments, and get to work algorithmically remastering the classic anthem with the Dolby.io Music Mastering API.

Typically performed by audio engineers, Mastering is a labor-intensive post-production process usually applied as the last step in creating a song and is the final polish that takes a track from good to great. Because "Take Me Out to the Ball Game" was recorded and produced in 1908 the mastering and post-production technology was very limited and hence it could be interesting to explore the impact of applying a music mastering algorithm on the original recording, and the effect that has on the palatability of the track.

Picking a Version:

Before we can get started remastering "Take Me Out to the Ball Game", we first need to pick a version of the song. Whilst we often hear the catchy tune played during the middle of the seventh inning, that version isn't the original and is subject to copyright protection. For this project, we will be using the 1908 version found here, as it is now available in the public domain and free to use. Unfortunately, the highest-quality version of the 1908 song is stored as an MP3. Whilst this works with the API, Free Lossless Audio Codec (FLAC) or other lossless file types are preferred as they produce the best results during the mastering post-production process.

The Music Mastering API:

With our song in hand, it's time to introduce the tool that will be doing the majority of the heavy lifting. The Dolby.io Music Mastering API is a music enhancement tool that allows users to programmatically master files via a number of sound profiles specific to certain genres and styles. The API isn't free, however, the company offers trial credits if you sign up and additional trial credits if you add your credit card to the platform.

For this project, the trial tier will be sufficient which is available if you sign up here.

Once you have created an account and logged in, navigate over to the applications tab, select "my_first_app", and locate your Media APIs API Key.

An example screenshot of the Dolby.io dashboard.

It's important to note that all Dobly.io media APIs adhere to the[ REST framework,](https://www.redhat.com/en/topics/api/what-is-a-rest-api#:~:text=A%20REST%20API%20(also%20known,by%20computer%20scientist%20Roy%20Fielding.) meaning they are language agnostic. For the purposes of this project, I will be using the tool in Python, however, it works in any other language.

Adding it to the Dolby.io Server

To utilize the Music Mastering API we first need to store the MP3 file on the cloud. This can be done with either a cloud service provider such as AWS, or you can use the Dolby.io Media Storage platform. For simplicity, we will use the Dolby.io platform which can be accessed via a REST API call.

To get started we need to import the Python "Requests" package and specify a path to the MP3 file on our local machine.

import requests #Requests is useful for making HTTP requests and interacting with REST APIs
file_path = "Take-Me-Out-to-the-Ball-Game.mp3"

Next, we need to specify the URL we want the Requests package to interact with, specifically the Dolby.io Media Input address. In addition to the input URL, we also need to format a header that will authenticate our request to the Dolby.io server with our API key.

url = "https://api.dolby.com/media/input"
headers = {
    "x-api-key": "YOUR DOLBY.IO MEDIA API KEY",
    "Content-Type": "application/json",
    "Accept": "application/json",
}

Finally, we need to format a body that specifies the name we want to give our file once it is added to the server.

body = {
    "url": "dlb://input-example.mp3",
}

With the URL, Head, and Body all formatted correctly we can use the Requests package to create a pre-signed URL to which we can upload our MP3 file to.

response = requests.post(url, json=body, headers=headers)
response.raise_for_status()
presigned_url = response.json()["url"]

print("Uploading {0} to {1}".format(file_path, presigned_url))
with open(file_path, "rb") as input_file:
    requests.put(presigned_url, data=input_file)

Starting a Mastering Job

Once the audio file has been moved to the cloud we can begin calling a mastering job. The Music Mastering API includes a number of predefined "profiles" which match up to a selection of audio genres such as Hip Hop or Rock. For the best results, a Rock song should be mastered with the Rock profile, however, the process of picking a profile can require a bit of experimentation.

Because matching creative intent with different sound profiles can take a few trials the API offers a "preview version" where you can master a 30 seconds segment of a song with 3 different profiles. We format the body of this request to include this information as well as when we want the segment to begin.

body = {
    "inputs": [
        {"source": "dlb://input-example.mp3", "segment": {"start": 36, "duration": 30}} #36 seconds is the start of the iconic chorus.
    ],
    "outputs": [
        {
            "destination": "dlb://example-master-preview-l.mp3",
            "master": {"dynamic_eq": {"preset": "l"}} #Lets master with the Vocal profile
        },
        {
            "destination": "dlb://example-master-preview-m.mp3",
            "master": {"dynamic_eq": {"preset": "m"}} #Lets master with the Folk profile
        },
        {
            "destination": "dlb://example-master-preview-n.mp3",
            "master": {"dynamic_eq": {"preset": "n"}} #Lets master with the Classical profile
        }

    ]
}

The Header stays the same as the one we used to upload the file to the Dolby.io Server and the URL changes to match the Music Mastering endpoint.

url = "https://api.dolby.com/media/master/preview"
headers = {
    "x-api-key": "YOUR DOLBY.IO MEDIA API KEY",
    "Content-Type": "application/json",
    "Accept": "application/json",
}

We can use the Requests package to deliver our profile selections and start the mastering job.

response = requests.post(url, json=body, headers=headers)
response.raise_for_status()
print(response.json())
job_id = response.json()["job_id"]

This process can take a minute to complete. To check the status of the job we can format another request to the same URL with the Job_ID included in the body to check the progress of the master.

url = "https://api.dolby.com/media/master/preview"
headers = {
        "x-api-key": "YOUR DOLBY.IO MEDIA API KEY",
        "Content-Type": "application/json",
        "Accept": "application/json",
    }
params = {"job_id": job_id}
response = requests.get(url, params=params, headers=headers)
response.raise_for_status()
print(response.json())

The response from the request outputs the progress of the job between 0% and 100%.

Downloading the Mastered File

With our file mastered it's time to download the three master previews so we can hear the difference. The workflow for downloading files mirrors that of how the rest of the Dolby.io APIs work. Much like uploading a file or starting a job, we format a header with our API key and a body that points to the mastering output on the Dolby.io server.

import shutil #File operations package useful for downloading files from a server.

url = "https://api.dolby.com/media/output"
headers = {
        "x-api-key": api_key,
        "Content-Type": "application/json",
        "Accept": "application/json",
    }

for profile in ["l","m","n"]:

    output_path = "out/preview-" + profile + ".mp3"

    preview_url = "dlb://example-master-preview-" + profile + ".mp3"
    args = {"url": preview_url}

    with requests.get(url, params=args, headers=headers, stream=True) as response:
        response.raise_for_status()
        response.raw.decode_content = True
        print("Downloading from {0} into {1}".format(response.url, output_path))
        with open(output_path, "wb") as output_file:
            shutil.copyfileobj(response.raw, output_file)

Summary of mastering job workflow from locating the file to downloading results.

With the mastered files downloaded locally, we can listen to both and hear the difference between the original and one of our Masters.

The original version of "Take Me Out to the Ball Game", sung by Edward Meeker in 1908.

Mastered version of "Take Me Out to the Ball Game" with the Dolby.io Classical music profile (Profile n) focusing on wide dynamics, and warm full tones for orchestral instruments.

We can also hear the subtle differences between the Masters.

Mastered version of "Take Me Out to the Ball Game" with the Dolby.io Vocal music profile (Profile l) focusing on the mid-frequencies to highlight vocals.

Mastered version of "Take Me Out to the Ball Game" with the Dolby.io Folk music profile (Profile m) focusing on light touch with ample mid-frequency clarity to let acoustic instruments shine in the mix.

Mastered version of "Take Me Out to the Ball Game" with the Dolby.io Classical music profile (Profile n) focusing on wide dynamics, and warm full tones for orchestral instruments.

For the purposes of this demo, we only mastered with the last three profiles, however, there are 14 different music mastering profiles to pick from. From my testing, I like the "Classical" profile (Profile n) the best, but everyone is different, try it out yourself.

A More Modern Example

Whilst the classic still doesn't sound modern, remastering the track does make it a little more clear and hence more enjoyable to listen to. Typically the Dolby.io Music Mastering API is built for contemporary samples recorded on more modern equipment in lossless formats such as FLAC and is not designed to be an audio restoration tool. For the purposes of this investigation, we wanted to see the impact post-production mastering would have on the track rather than attempting to outright "fix" the original.

Currently, the Dolby.io team has a demo hosted here that lets you listen to before and after examples of licensed contemporary tracks which better exemplifies the use case of the API. Because Dolby.io owns the licenses to those songs they are allowed to host the content, whereas for this project I wanted to pick a track in the public domain so anyone with an interest can test it out for themselves without fear of infringing on copyright law.

The Dolby.io Music Mastering demo, available here.

If the Music Mastering API is something you are interested in further exploring check out the dolby.io documentation around the API or the live demo mentioned above, otherwise let's get excited for an awesome Baseball season ahead and "root, root, root for the home team".

Generating Pre-Signed URLs for Azure Cloud Storage with Python

Braden Riggs — Thu, 07 Apr 2022 20:31:31 +0000

Introduction

Media files stored through Microsoft's Azure cloud platform can be easily integrated with Dolby.io to create a pipeline for media enhancement and analysis, allowing Azure users to enrich and understand their audio, all from the cloud. In this guide, we will explore how to integrate Dolby.io's Media Processing APIs with Azure Blob Storage to help users enhance their audio in a simple and scalable way.

What do we Need to Get Started

Before we get started there are five parameters you need to make sure you have ready for integration with Azure:

Azure Account Name: The name of your Azure account.
Azure Container Name: The name of the Container where your file is located.
Azure Blob Name: The name of the Blob where your file is stored.
Azure Primary Access Key: The Primary Access Key for your Azure storage account.
Dolby.io API Key: The Dolby.io mAPI key found on your Dolby.io dashboard.

These parameters help direct Dolby.io to the appropriate location for finding your cloud-stored file, as well as handling necessary approval for accessing your privately stored media.

Getting Started with Python

To begin building this integration we first need to install Azure.Storage.Blob v12.8.1. It is important to note that installing the Azure.Storage.Blob Python package differs from just installing the base Azure SDK, so make sure to specifically install Azure.Storage.Blob v12.8.1 as shown in the code below. Once installed we can import the Python datetime, requests, and time packages, along with generate_blob_sas and BlobSasPermissions functions from Azure.Storage.Blob.

#!pip install azure.storage.blob==12.8.1

from datetime import datetime, timedelta #For setting the token validity duration
import requests                          #For using the Dolby.io REST API
import time                              #For tracking progress of our media processing job

#Relevent Azure tools
from azure.storage.blob import (
    generate_blob_sas,
    BlobSasPermissions
)

Next, we define our input parameters. We specify both an input and an output for our Blob files. The input represents the name of the stored file on the server and the output represents the name of the enhanced file that will be placed back onto the server.

# AZURE
AZURE_ACC_NAME = 'your-account-name'
AZURE_PRIMARY_KEY = 'your-account-key'
AZURE_CONTAINER = 'your-container-name'
AZURE_BLOB_INPUT='your-unenhanced-file'
AZURE_BLOB_OUTPUT='name-of-enhanced-output'

We also need to define some Dolby.io parameters including the server and the function applied to your files. In this case, we pick enhance and follow up by defining our Dolby.io Media Processing API key.

# DOLBY
server_url = "https://api.dolby.com"
url = server_url +"/media/enhance"
api_key = "your Dolby.io Media API Key"

With all our variables defined, we can now create the Shared Access Signatures (SAS) the Dolby.io API will use to find the files. To do this we use the generate_blob_permissions function in conjunction with the BlobSasPermissions function and the datetime function.

input_sas_blob = generate_blob_sas(account_name=AZURE_ACC_NAME,
                                container_name=AZURE_CONTAINER,
                                blob_name=AZURE_BLOB_INPUT,
                                account_key=AZURE_PRIMARY_KEY,
                                permission=BlobSasPermissions(read=True),
                                expiry=datetime.utcnow() + timedelta(hours=1))

output_sas_blob = generate_blob_sas(account_name=AZURE_ACC_NAME,
                                container_name=AZURE_CONTAINER,
                                blob_name=AZURE_BLOB_OUTPUT,
                                account_key=AZURE_PRIMARY_KEY,
                                permission=BlobSasPermissions(read=True, write=True, create=True),
                                expiry=datetime.utcnow() + timedelta(hours=1))

Note this in the code sample above we define both an input and an output SAS Blob. For the input SAS, we only need to give the signature the ability to read in files, however, for the output SAS, we need to give it the ability to create a file on the Azure server and then write to that file. We also specify how long we want our signatures to be valid. In this case, the links are valid for one hour, however, for larger jobs, we may need to increase this window of validity.

With our SAS tokens created we now need to format the tokens into URLs. Again we need to create two URLs, one for the input and one for the output.

'https://'+AZURE_ACC_NAME+'.blob.core.windows.net/'+AZURE_CONTAINER+'/'+AZURE_BLOB_OUTPUT+'?'+output_sas_blob

Using both SAS URLs we plug the values into the Dolby.io API and initiate the media processing job. The unique identifier for the job is captured in the job_id parameter which we can use to track progress.

body = {
  "input" : input_sas,
  "output" : output_sas
}

headers = {
  "x-api-key":api_key,
  "Content-Type": "application/json",
  "Accept": "application/json"
}

response = requests.post(url, json=body, headers=headers)
response.raise_for_status()
job_id = response.json()["job_id"]

Note how the input and output of the body have their corresponding URLs assigned.

Our job has now begun. To track the progress of the job we can create a loop that reports job status.

while True:
    headers = {
      "x-api-key": api_key,
      "Content-Type": "application/json",
      "Accept": "application/json"
    }

    params = {"job_id": job_id}

    response = requests.get(url, params=params, headers=headers)
    response.raise_for_status()
    print(response.json())

    if response.json()["status"] == "Success" or response.json()["status"] == "Failed":
        break

    time.sleep(20)

print("response.json()["status"]")

Once the job has been completed the loop will exit and our enhanced file will be visible on the Azure Blob Storage server. Alternatively, instead of looping through such as that seen in the example above, Dolby.io offers functionality for webhooks and callbacks which can be used to notify users of a jobs competition.

Conclusion

In summary, once we have created an Azure valid SAS, the process for integrating Azure with Dolby.io is simple, allowing for seamless integration between the two services. If you are interested in learning more about how to integrate Azure with Dolby.io or explore examples in alternative coding languages check out our documentation here.

Creating A Fixed Place Spatial Environment for Video Conferencing

Braden Riggs — Thu, 07 Apr 2022 00:47:12 +0000

Being in a virtual meeting where the audio comes at you can be a jarring and awkward experience as audio tracks overlap and speakers become indiscernible. Spatial audio, also known as 3D audio is a more natural approach to solving this problem. Traditionally non-spatial audio is subject to the walkie-talkie effect, where sound is flattened and output to the user via one or two channels of data in an experience that can feel like the audio is coming at you all at once. This limitation is corrected with spatial audio as sound instead comes from around you, helping create an environment where conversations blend more naturally like they would in the physical world.

In this blog post, we'll show how you can set up a 2D virtual video conference where participants will speak from fixed spatial perspectives using Dolby.io Spatial Audio and the Dolby.io Web SDK.

Account Setup

To get started with creating a static spatial audio scene you first need to have signed up for a free Dolby.io account. The free tier of Dolby.io awards you trial credit and doesn't require a credit card.

Once signed up and logged in, scroll down to the "applications" section and create a new app titled "Static_spatial". After naming the application your page will open to a list of API keys, take note of the Communications API Consumer Key and Consumer Secret.

Dolby.io Web SDK

With your account set up and your API keys on hand, we can get started with creating a static spatial video conference using the Dolby.io Web SDK.

If you haven't worked with our Web SDK before I recommend first building a basic web application by following our Web SDK Getting Started guide. If you are already familiar with the guide we can start by cloning the Fixed Place Spatial Demo repository. This project largely builds off of the original Web SDK getting started guide by adding user interface adjustments for creating a spatial experience. Whilst this blog will mainly focus on implementing a fixed spatial audio experience, all of the code is included in the repository in case you are interested in reviewing the interface changes as well.

Getting Started with a Spatial Environment

Once you have cloned the Fixed Place Spatial Demo project, the next step to adding spatial to your app is enabling it in the creation of your conference. To do this we assign an alias that can be user-defined or static depending on the scope of your web app and enable Dolby Voice, a tool that optimizes bandwidth utilization and suppresses background noise and audio defects in real-time.

let conferenceOptions = {
    alias: "spatialTestConf", // Can be user defined
    params: {dolbyVoice: true}, //Required for spatial audio
};

VoxeetSDK.conference.create(conferenceOptions)

With a Dolby Voice enabled conference created, we now activate spatial audio. It is important to note that these API calls are asynchronous operations, meaning they can vary slightly in execution time and hence rely on promises to resolve, so including the await operator or a .then() function is required.

//Start conference with audio and video turned off
VoxeetSDK.conference.join(conference, {
                    constraints: { audio: false, video: false },
                    spatialAudio: true,
                })

Setting a Spatial Scene

Part of integrating spatial audio into a web application is defining the spatial "scene", or rather how the audio renderer interprets what is "Forward" or what is "Right".

We define these directions on an "x, y, z-axis", where the larger "x" is the further to the right noise is heard, and the smaller "y" is the more participants at the top of the screen are heard from the front. In this case, the 3rd direction is irrelevant as our conference is only represented on a 2-dimensional plane, however, three-dimensional projects would also have to define an "up" axis as "z".

// Negative Y axis is heard in the forwards direction, so tell the SDK forward has -1 Y
const forward = { x: 0, y: -1, z: 0 };

// Upwards axis is unimportant for this case, we can set it to either Z = +1 or Z -1,
// we never provide a Z position
const up  = { x: 0, y: 0, z: 1 };

// Positive X axis is heard in the right-hand direction, so tell the SDK right has +1 X
const right  = { x: 1, y: 0, z: 0 };

In addition to directions, we also define a scale that mimics the physical world. In real life we define the hearing limit as the furthest possible distance you can hear someone. Whilst a variety of factors influence this limit in the real world, in the Dolby.io virtual world this limit is capped at 100 meters, so a person further than 100 meters away wouldn't be heard. This raises a question though, what is a meter in the virtual space? We define this as the "scale" parameter which can be converted from user defined units.

// scale for Z axis doesn't matter as we never provide a Z position, set it to 1
// We set the scale as 1:10, so as we move one unit in the virtual world, our hearing changes as // if we have moved 10 meters in the physical world.
 const scale = {
                    x: 0.1,
                    y: 0.1,
                    z: 1,
                };

For the purposes of our virtual conference, we want the scale to be defined by a 1:10 ratio, meaning that a guest who is assigned an "x" position 5 units greater than you would sound 50 meters further away in the "x" direction.

Setting Spatial Position

With the scale set and spatial audio enabled we now need to make sure everyone is assigned their spatial location as they join.

How people's spatial location is selected will depend on the layout of your web app, in our example code, we will be using a 3x3 square grid allowing for a maximum of 9 participants to join the web conference.

To appropriately assign spatial location we must track who joined and when. One way to do this is to define a posList object composed of 9 different arrays each containing an undefined participant ID and different position combinations. With both these lists created we next need to assign spatial positions to the attendees in the order of left to right, top to bottom as they arrive.

let posList = [
    [undefined, 1, 1],
    [undefined, 2, 1],
    [undefined, 3, 1],
    [undefined, 1, 2],
    [undefined, 2, 2],
    [undefined, 2, 3],
    [undefined, 3, 1],
    [undefined, 3, 2],
    [undefined, 3, 3],
];

There are a variety of different ways we can work to associate a particular participant with a spatial location. For example, our documentation maps the audio locations to the participant's center video. In our case, we will take a different approach that works by using a for loop to iterate over an array that records the participant’s ID. For each participant ID, the loop will then assign the corresponding spatial position according to the count. I.e. the 1st person would be assigned the array [personOneID, 1, 1] which corresponds to the first square along the top row and would sound 10 meters away in the x-direction when heard by someone who was assigned the second spatial position [personTwoID, 2, 1].

Once the position is assigned we can use the setSpatialPosition function. This function takes in a newly joined participant and assigns them to the next available cell. This means the first person will be assigned the top left square, the second person would be assigned the top center square, the third person will be assigned the top right square, and so on.


//Function for altering spatial positions as guests join
const setSpatialPosition = (participant) => {
    let spatialPosition = { x: 0, y: 0, z: 0 }; //default spatial position

    //loop over posList
    for (let i = 0; i < posList.length; i++) {
        //If posList[i] has no assigned participantID, assign one
        if (!posList[i][0] || participant.id == posList[i][0]) {
            posList[i][0] = participant.id;

            //Assigned spatial position based on join order
            spatialPosition = {
                x: posList[i][1],
                y: posList[i][2],
                z: 0, // Only 2d so "z" is never changed
            };

            break;
        }
    }
    VoxeetSDK.conference.setSpatialPosition(participant, spatialPosition);
};

In the sample code provided we included a banner that displays the spatial position of the user in terms of "x, y, and z".

Try it out yourself

Now that we have the theory out of the way we can boot up the sample app and try it out. the first step to getting the spatial app up and running is to update the last two rows of the "scripts/client.js" file with your Communications API Consumer Key and Consumer Secret.

//Update scripts/client.js with your API Keys
main(
    "Insert your Communications APIs Consumer Key here",
    "Insert your Communications APIs Consumer Secret here"
);

Now, simply open the file "index.html" in your web browser and start playing with the application.

It is important to note that hard coding your API keys into the client.js file is only for testing and should not be used for production as the key are not secure and could be stolen. Instead we we recommend using a token to initialize the SDK. For more information, see Initializing or learn about security best practices.

Next Steps

Spatial audio opens the door to a range of possibilities when building your web conferencing app, such as virtual events, meeting spaces, and collaboration tools. For this blog we kept it simple with a fixed place example, however, the tools work just as well for building a dynamically updating web app that adjusts spatial audio as the users move around in a 2D or 3D environment.

Whatever your next spatial project is, the Dolby.io team is here to help. Connect with us here or check out a few of our helpful resources to dive deeper into the awesome world of communication and spatial audio: