DEV Community: Spherical Flying Kat

WebRTC 102: #3 Getting Real with RTP and RTCP

Spherical Flying Kat — Mon, 20 Mar 2023 13:56:11 +0000

Introduction

Real-time communication over the internet has become increasingly popular in recent years, and WebRTC has emerged as one of the leading technologies for enabling it over the web. WebRTC uses a variety of protocols, including Real-Time Transport Protocol (RTP) and Real-Time Control Protocol (RTCP).

RTP is responsible for transmitting audio and video data over the network, while RTCP is responsible for monitoring network conditions and providing feedback to the sender. RTP and RTCP communicate over the same network, with RTP using even-numbered ports and RTCP using odd-numbered ports. This allows both protocols to use the same network resources without interfering with each other.

In this post, we will discuss what RTP and RTCP are and how they work together to enable real-time communication in WebRTC.

This is the third part of our ongoing WebRTC 102 blog series — in the second one we covered libWebRTC and in the first one, we tackled ICE and understood how those work under the hood.

Let’s dive in!

Real-Time Transport Protocol (RTP)

Real-Time Transport Protocol (RTP) is a protocol designed for transmitting audio and video data over the internet. RTP is used to transport media streams, such as voice and video, in real-time.

RTP is responsible for packetizing media data into small packets and transmitting it over the network. Each RTP packet contains a sequence number and timestamp, which are used to ensure that the packets are delivered in the correct order and at the correct time. RTP packets are transmitted over UDP, which provides low latency and is ideal for real-time communication.

Real-Time Control Protocol (RTCP)

Real-Time Control Protocol (RTCP) is a protocol designed to provide feedback on the quality of service (QoS) of RTP traffic. RTCP is used to monitor network conditions, such as packet loss and delay, and to provide feedback to the sender. RTCP packets are sent periodically to provide feedback on the quality of the RTP stream. They contain statistical information about the RTP stream, including the number of packets sent and received, the number of packets lost, and the delay between packets. This information can be used to adjust the RTP stream to improve the quality of the audio or video.

Understanding video compression

We will not delve deeply into video compression, but we will cover enough to understand why RTP is designed as it is. Video compression encodes video into a new format that requires fewer bits to represent the same video.

Lossy and Lossless Compression

Video can be encoded to be lossless (no information is lost) or lossy (information may be lost). RTP usually uses lossy compression to prevent high latency streams and more dropped packets, even if the video quality is not as good.

Intra and Inter-Frame Compression

Video compression comes in two types: intra-frame and inter-frame. Intra-frame compression reduces the bits used to describe a single video frame. The same techniques are used to compress still pictures, like the JPEG compression method. On the other hand, inter-frame compression looks for ways to not send the same information twice, since video is made up of many pictures.

Inter-Frame types

There are three frame types in inter-frame compression:

I-Frame - A complete picture that can be decoded without anything else.
P-Frame - A partial picture that contains only changes from the previous picture.
B-Frame - A partial picture that is a modification of previous and future pictures.

It is clear that video compression is a stateful process, posing challenges when transferred over the internet. It leads us to wonder, what happens if part of an I-Frame is lost? How does a P-Frame determine what to modify? As video compression methods become more intricate, these issues become even more pressing. Nonetheless, there is a solution offered by RTP and RTCP.

RTP packet structure

Every RTP packet has the following structure, as defined in the RFC

0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|X|  CC   |M|     PT      |       sequence number         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                           Timestamp                           |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|           Synchronization Source (SSRC) identifier            |
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
|            Contributing Source (CSRC) identifiers             |
|                             ....                              |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Payload                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Version (V)

The Version is always set to 2.

Padding (P)

Padding is a boolean that determines whether the payload has padding or not.

The last byte of the payload indicates the number of padding bytes that were added.

Extension (X)

If set, the RTP header will contain extensions.

CSRC Count (CC)

The CSRC Count refers to the number of CSRC identifiers following the SSRC and preceding the payload.

Marker

The Marker bit has no predetermined meaning and can be used as desired by the user.

In some cases, it may indicate when a user is speaking, or it may denote a keyframe.

Payload Type (PT)

The Payload Type is a unique identifier for the codec being carried by this packet.

For WebRTC, the Payload Type is dynamic, meaning that the Payload Type for VP8 in one call may differ from that of another. The offerer in the call determines the mapping of Payload Types to codecs in the Session Description.

Sequence Number

The Sequence Number is used for ordering packets in a stream. Every time a packet is sent, the Sequence Number is incremented by one.

RTP is designed to be useful over lossy networks. This gives the receiver a way to detect when packets have been lost.

Timestamp

The Timestamp is the sampling instant for this packet. It is not a global clock, but rather represents how much time has passed in the media stream. Several RTP packets can have the same timestamp if, for example, they are all part of the same video frame.

Synchronization Source (SSRC)

An SSRC is a unique identifier for this stream. This allows multiple streams of media to be run over a single RTP stream.

Contributing Source (CSRC)

This is a list that communicates which SSRCs contributed to this packet.

This is commonly used for talking indicators. For example, if multiple audio feeds are combined into a single RTP stream on the server side, this field could be used to indicate which input streams were active at a given moment.

Payload

This field contains the actual payload data, which may end with the count of how many padding bytes were added if the padding flag is set.

RTCP packet structure

Every RTCP packet has the following structure:

0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|V=2|P|    RC   |       PT      |             length            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                            Payload                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Version (V)

The Version is always 2.

Padding (P)

Padding is a boolean that controls the inclusion of padding in the payload.

The last byte of the payload includes a count of the added padding bytes.

Reception Report Count (RC)

This indicates the number of reports included in this packet. A single RTCP packet may contain multiple events.

Packet Type (PT)

This is a unique identifier for the type of RTCP packet. While a WebRTC agent does not necessarily need to support all these types, there may be differences in support between agents. Some of the commonly seen packet types include:

192 - Full INTRA-frame Request (FIR)
193 - Negative Acknowledgements (NACK)
200 - Sender Report
201 - Receiver Report
205 - Generic RTP Feedback
206 - Payload Specific Feedback (including PLI)

RTCP packet types in detail

RTCP is a flexible protocol and supports several types of packets. The following details some of the most commonly used packet types.

PLI (Picture Loss Indication)/FIR (Full Intra Request)

Both FIR and PLI messages serve a similar purpose, requesting a full key frame from the sender. However, PLI is specifically used when the decoder failed to decode partial frames, which could be due to packet loss or decoder crashing.

As per RFC 5104, FIR should not be used when packets or frames are lost; that is the job of PLI. FIR is used to request a key frame for other reasons, such as when a new member enters a video conference and needs a full key frame to start decoding the video stream. The decoder will discard frames until a key frame arrives.

However, in practice, software that handles both PLI and FIR packets will send a signal to the encoder to produce a new full key frame in both cases.

Typically, a receiver will request a full key frame immediately after connecting to minimize the time to the first frame appearing on the user's screen.

PLI packets are a part of Payload Specific Feedback messages.

NACK (Negative Acknowledgement)

When a receiver issues a NACK, it requests that the sender re-transmit a single RTP packet. This is typically done when a packet is lost or delayed. The NACK is preferable to requesting that the entire frame be re-sent because RTP divides packets into small pieces, and the receiver is usually only missing one piece. To request the missing piece, the receiver creates an RTCP message with the SSRC and Sequence Number. If the sender does not have the requested RTP packet to re-send, it will simply ignore the message.

Sender and Receiver Reports

These reports are crucial for transmitting statistics between agents. They effectively communicate the exact amount of packets that are received as well as the levels of jitter.

This feature provides valuable diagnostic information and enables effective congestion control. We’ll see more about how these reports are used to overcome unreliable network conditions below.

Overcoming unreliable networks

Real-time communication is heavily dependent on networks. In an ideal scenario, bandwidth would be unlimited and packets would arrive instantaneously. Unfortunately, networks are limited and conditions can change unexpectedly, making it difficult to measure and observe network performance. Additionally, different hardware, software, and configurations can cause unpredictable behavior.

RTP/RTCP runs across many different types of networks, so it is common for some communication to be lost between the sender and the receiver. Because it is built on top of UDP, there is no built-in way to retransmit packets or handle congestion control.

Measuring and communicating network status

RTP/RTCP operates on various network types and topologies, and communication drops may occur from sender to receiver due to this. As they are built on UDP, there is no inherent mechanism for packet retransmission or congestion control.

For optimal user experience, we must assess network path qualities and adapt to their changes over time. The key traits to monitor are available bandwidth (in each direction, which may not be symmetrical), round trip time, and jitter (variations in round trip time). Our systems must take into account packet loss and convey changes in these properties as network conditions change.

The protocols have two main objectives:

Estimate the available bandwidth (in each direction) supported by the network.
Communicate network characteristics between sender and receiver.

Receiver reports / Sender reports

Receiver Reports and Sender Reports are sent over RTCP and are defined in RFC 3550. They communicate network status between endpoints. Receiver Reports communicate network quality, including packet loss, round-trip time, and jitter. These reports pair with other algorithms that estimate available bandwidth based on the network quality.

Sender and Receiver Reports (SR and RR) together give a picture of network quality. They are sent on a schedule for each SSRC and are used to estimate available bandwidth. The sender estimates available bandwidth after receiving the RR data, which contains the following fields:

Fraction Lost - Percentage of packets lost since the last Receiver Report.
Cumulative Number of Packets Lost - Number of packets lost during the entire call.
Extended Highest Sequence Number Received - Last Sequence Number received and how many times it has rolled over.
Interarrival Jitter - Rolling Jitter for the entire call.
Last Sender Report Timestamp - Last known time on sender, used for round-trip time calculation.

These statistics are further fed into bandwidth estimation algorithms such as GCC (Google Congestion Control), which estimate available bandwidth which in turn drives the encoding bitrate and frame resolution.

Conclusion

In conclusion, RTP and RTCP are essential protocols for enabling real-time communication in WebRTC. RTP is responsible for transmitting audio and video data over the network, while RTCP is responsible for monitoring network conditions and providing feedback to the sender. Together, these protocols enable high-quality real-time communication over the internet. Understanding how RTP and RTCP work together is essential for anyone interested in developing real-time communication applications using WebRTC.

References

RFC3550(RTP: A Transport Protocol for Real-Time Applications)
RFC5104 (Codec Control Messages in the RTP Audio-Visual Profile with Feedback)
RFC8888 (RTP Control Protocol (RTCP) Feedback for Congestion Control)

Try Dyte if you don't want to deal with the hassle of managing your own peer-to-peer connections!

If you haven't heard of Dyte yet, go to https://dyte.io to learn how our SDKs and libraries revolutionize the live video and voice-calling experience. Don't just take our word for it; try it for yourself! Dyte offers free 10,000 minutes every month to get you started quickly.If you have any questions or simply want to chat with us, please contact us through support or visit our developer community forum. Looking forward to it!

Tools of the Trade: Topgrade

Spherical Flying Kat — Mon, 06 Jun 2022 19:44:41 +0000

In the third post of this series, let's talk about topgrade.

What is topgrade?

You've probably run into this scenario before: you have a package manager for your system; brew on Mac, pacman on Arch Linux etc, and you have a bunch of packages installed using these package managers.

Now comes the twist, you end up having several other package managers (eg: pip, npm); because nobody can agree on a standard and it's probably too far gone at this point to care anyway.

To keep the entirety of your system up to date, you end up having to run several commands, which eats up precious time that you could be spending watching cat videos.

topgrade remedies this situation by detecting the tools and package managers installed on your system and runs the appropriate commands to update them.

How do I get it?

Pretty straightforward. topgrade provides packages for Arch Linux, NixOS, and Mac (via homebrew or macports).

If you're on a different platform or OS, install topgrade via the cargo package manager by running cargo install topgrade.

If you're a normie who doesn't use Rust, go ahead and download one of the binaries available in the releases and place it on your $PATH.

Usage

Dead simple. Just run topgrade.

And that's it from me. Enjoy those cat videos!

Tools of the Trade: Declarative DNS with dnscontrol

Spherical Flying Kat — Fri, 27 May 2022 02:13:04 +0000

💡 A bit of context

I recently moved my site's hosting from DigitalOcean to fly.io. That's when I realized that I've been changing DNS records manually for years; go to the Cloudflare dashboard, change records, rinse-repeat...

As a programmer and self-professed automation fanatic, this rubbed me the wrong way, so I set out to do something about it. Enter DNSControl.

🤔 Hold up, what's that?

DNSControl is an opinionated platform for seamlessly managing your DNS configuration across any number of DNS hosts, both in the cloud or in your own infrastructure.

It was created by StackOverflow to manage their own domains, and was subsequently open-sourced for the rest of us (Thanks StackOverflow!).

It lets you declare your DNS records using a JavaScript-ish DSL (domain specific language); for example:

Okay, why should I switch?

Switching to a new tool can be scary, but here's some reasons that make DNScontrol unrivaled (in my opinion):

Supports 10+ DNS Providers including BIND, AWS Route 53, Google DNS, Cloudflare, and name.com
Enable/disable Cloudflare proxying (the "orange cloud" button) directly from your DNSControl files.
Apply CI/CD principles to DNS: Unit-tests, system-tests, automated deployment.
Super extensible! Plug-in architecture makes adding new DNS providers and Registrars easy!
Eliminate vendor lock-in. Switch DNS providers easily, any time, with full fidelity.
All the benefits of Git (or any VCS) for your DNS zone data. View history. Accept PRs.

And much more!

🤨 Wait! I heard opinionated?

That's right. DNSControl has opinions about how to manage DNS. However, that isn't a bad thing. Let's go over their opinions and see which ones really affect us.

DNS should be treated like code: Code is written in a high-level language, version controlled, commented, tested, and reviewed by a third party… and all of that happens before it goes into production.
Non-experts should be able to safely make DNS changes.
All DNS is lowercase for languages that have such a concept.
Users should state what they want, and DNSControl should do the rest.
If it is ambiguous in DNS, it is forbidden in DNSControl.
Hostnames don’t have underscores.

These make total sense. To read more about the opinions and the reasons behind them, check out the documentation.

✅ Okay, I'm convinced. How do I begin?

Glad to hear it! To get started, first install DNSControl.

go install github.com/StackExchange/dnscontrol/v3@latest

Now, create a file called dnsconfig.js. This will house all the records for your site(s). Populate it with the starter boilerplate:

// Providers
var REG_NONE = NewRegistrar('none');    // No registrar.
var DNS_BIND = NewDnsProvider('bind');  // ISC BIND.

// Domains
D('example.com', REG_NONE, DnsProvider(DNS_BIND),
    A('@', '1.2.3.4')
);

Next, create a file called creds.json for storing provider configurations (API tokens and other account information). For example, to use both name.com and Cloudflare, you would have:

{
  "cloudflare": {
    "TYPE": "CLOUDFLAREAPI",
    "accountid": "your-cloudflare-account-id",
    "apitoken": "your-cloudflare-api-token"
  },
  "namecom": {
    "TYPE": "NAMEDOTCOM",
    "apikey": "key",
    "apiuser": "username"
  },
  "none": { "TYPE": "NONE" } // the no-op provider
}

For a complete list of providers and how to use them, check out the provider docs.

Warning: Remember to add creds.json to your .gitignore before committing and pushing your configuration.

Now, run dnscontrol preview and make sure that it completes with no errors. The preview command is the “dry run” mode that shows what changes need to be made and never makes any actual changes. It will use APIs if needed to find out what DNS entries currently exist.

The results should look something like this:

dnscontrol preview
Initialized 1 registrars and 1 dns service providers.
******************** Domain: example.com
----- Getting nameservers from: bind
----- DNS Provider: bind... 1 correction
#1: GENERATE_ZONEFILE: example.com
 (2 records)

----- Registrar: none
Done. 1 corrections.

Next, run dnscontrol push to actually make the changes. In this case, the change will be to create a zone file where one didn’t previously exist.

dnscontrol push
Initialized 1 registrars and 1 dns service providers.
******************** Domain: example.com
----- Getting nameservers from: bind
----- DNS Provider: bind... 1 correction
#1: GENERATE_ZONEFILE: example.com
 (2 records)

CREATING ZONEFILE: zones/example.com.zone
SUCCESS!
----- Registrar: none
Done. 1 corrections.

⚙️ I already have many DNS records set up. How do I import them?

This was a problem I faced myself, and this is how you import your existing DNS configurations.

Most DNS Service Providers have an 'export to zonefile' feature. (If yours doesn't, consider moving to Cloudflare; It's free, they'll import all your records during setup, and will let you export a zonefile)

Place this zonefile inside a directory called zones in the directory where your dnsconfig.js resides. Rename the zonefile to <domain>.zone. Your directory structure should now look something like this:

.
├── creds.json
├── dnsconfig.js
└── zones
    └── example.com.zone

Next, run the following command:

dnscontrol get-zones --format=js --out=draft.js bind BIND example.com

Now you can copy the contents of draft.js to dnsconfig.js and modify as needed.

🚀 Lastly, some advice for running in production

Store the configuration files in Git.
Encrypt or completely omit the creds.json file before storing it in Git. Do NOT store API keys or other credentials without encrypting them.
Use a CI/CD tool like Github Actions/Jenkins/CircleCI/etc. to automatically push DNS changes.
Join the DNSControl community. File issues and PRs.

I hope this tool is just as useful to you as it is to me. Stay tuned for further posts about useful tooling.

Tools of the Trade: Dyte CLI

Spherical Flying Kat — Thu, 05 May 2022 13:14:19 +0000

Dyte CLI brings Dyte to your terminal. It reduces context switching, helps you focus, and empowers you to more easily script and create your own workflows on top of Dyte.

With our CLI, you can:

Run your entire Dyte workflow from the terminal, from meetings through recordings
Play around with webhooks
Release your own custom plugins

🎬 From meetings to recordings

The inherent scriptability brought to you by a CLI lets you chain many logical actions together. For example, you could have a workflow like:

You’re an ed-tech startup and have a quick extra session scheduled for your students. Let’s create a meeting in a jiffy:

Now that you have a meeting to work with, you can share the link with your students and have your session 🤝.
You’ve noticed some people are tardy or completely skip extra sessions. Not good 😠. What can we do? We can keep a track of who joins using webhooks!

Oh, the session is very important, you should record this!

And that’s just an example of how you can make Dyte CLI your own!

🎨 Design considerations

While a CLI might not have a lot of snazzy eye-candy like a website or an app, the design that goes into it is just as important.

We kept a lot of things in mind while designing the CLI so that our end-users have the best possible experience while using it. Here’s a few of the things that went on in our heads while we were working on it:

#️⃣ POSIX compliance

Unix-like operating systems popularised the use of the command line and tools such as awkand sed. Such tools have effectively standardised the behaviour of command line options (aka flags), options-arguments, and other operands.

Why is it important? Users might get confused if our CLI’s syntax for arguments, options, or command parameters deviate from the de facto Unix standards they are used to.

Some examples of expected behaviour:

Option-arguments or options can be notated in help or examples as square brackets ([]) to indicate they are optional, or with angle brackets (<>) to indicate they are required.
Allowing short-form single letter arguments as aliases for long-form arguments (see reference from the GNU Coding Standards).
specifying multiple options with no values may be grouped, such as dyte -abc being equivalent to dyte -a -b -c.

💕 Empathy for the user

We’ve put workflows in place that assist the user in interacting with the CLI successfully. We understand the frustration that comes with being unable to use an unfamiliar tool effectively, and we’re here to help you through it!

As we’ve seen while starting a recording, the start subcommand takes in a meeting ID. Failing to pass in one and being presented with an error would get pretty frustrating!

And so, we’ve taken care to prompt the user gently for any parameters they miss out on.

✨ Rich interactions and a colourful experience

We’ve leveraged rich command-line interactions beyond that of plain text input to provide a smoother experience for our users.

The CLI makes heavy use of prompts, drop-down selections, hidden password inputs, auto-complete and searchable lists.

Moving on to colour, most terminals used today to interact with command line applications support coloured text. We’ve made use of this to highlight important bits of output, and to draw your attention to things that might otherwise get lost in an otherwise text-heavy experience.

🧠 Stateful data

We’ve strived to provide a stateful experience, by remembering values and data where we can to improve future interactions.

This is important to ensure we don’t bug the user to enter the same information again and again.

All data stored by the CLI can be found at ~/.config/dyte

🧹 Cleaning up

We’re sad to see you go, but look forward to seeing you again! We don’t leave any configuration files behind when you uninstall the CLI by leveraging the convenient postuninstall script that NPM provides.

📦 Zero dependencies

Okay, that was a bit misleading. We certainly use a few dependencies to make the CLI great, we’re not that good 😛.

What we do however, is bundle all these dependencies, and our own code into a single JS file using the nifty ncc tool (thanks Vercel!)

This lets us ship the CLI without having to worry about any breaking changes in dependencies or nasty supply-chain attacks (remember left-pad and colors.js?). This way, you get exactly what our CI systems build and publish, Untouched by Human Hands™️

🔥 Where can I get it?

The Dyte CLI is currently available on the NPM registry. Please also take a moment to check out how to use the CLI in our comprehensive docs.

We’re also planning on open-sourcing the CLI soon, so keep a watch out for any announcements in that space and feel free to contribute!

Wrapping up

We’ve tried to make the usage of Dyte’s CLI to be as pleasant and painless as possible. Using it, you can automate an end-to-end workflow without having to interact with dashboards or our API directly.

We’ve already been using the CLI internally for months now, so bugs should be few and far between. Even so, if you find any, please report them to us!

Take your Dyte experience to new heights with the CLI!

Originally published at https://blog.dyte.io on May 5, 2022.