This question arose from an annoyance while watching twitch streams. I often participate in streams where the streamer is playing a jackbox game. It's quite important to see what's going on on the streamer's screen. My internet connection is quite slow, and so Twitch often buffers. When it resumes, it resumes from the point where it let off.
When TCP Vs. UDP is discussed, the canonical use case for UDP is often teleconferencing, where you don't care if you miss a frame of video, you just want it to be quick. Live video streaming online seems not much different rom that, yet Twitch's video player seems to behave as if it's going over TCP, waiting until it has received enough data to show the next few seconds of video.
So is this what's going on?
Top comments (7)
I've never used twitch, but I love the engineering challenges with streaming video and have worked with it quite a bit in the past. I briefly glanced through the twitch app to decipher what is going on.
This part is standard practice. The backend has a video streaming server where the content gets pushed via a client-side streaming software. This can use UDP or TCP. It depends on how it is configured. If the content will be made available later, it is most likely using TCP.
Now, the server side software can be configured to serve content directly (most likely twitch isn't doing that). I think there's a proxy server that the client connects to. It is some internal/external CDN.
On the client, a web-socket connection is initiated and upgraded to TCP or downgraded to HTTP depending on what the network and browser support. I'm not sure if they intended to use it for streaming or other notifications. On my setup, it seems to use HTTP. I noticed that there's a pub-sub module setup which means they could also push the updates from the server as it would be efficient that way with thousands of clients.
Twitch uses a service worker in the background. This service-worker thread fetches aggregate video frames sequentially over HTTP and caches them locally. If you pause a live feed and resume it, the client will drop expired packets and continue requesting data from the current time.
When I tried this through Insomnia, I noticed that they still serve older packets which means the fee is being stored. Although, the player didn't allow me to rewind a live feed. I didn't get a chance to save these packets, but it is possible to decipher the content and play it through another video player.
This is a common practice in most web-applications as it is not only easy to implement, but it also helps you get through firewalls and works via SSL.
Most of what we use is HTTP (which is on TCP). It downloads the video in chunks (maybe 5-10 seconds? not sure) over TCP, and the client-side (javascript) player tries to keep a couple of chunks ahead of where the user is. If the player can't download fast enough to keep ahead of the user, it switches to a lower resolution (smaller) chunk on the next request.
Using HTTP, each chunk has a different url (usually thru query parameters), so the chunks can be cached. I can host the videos in a central location and then have a local caching proxy that will serve the video (chunks) it has seen recently without using internet bandwidth, for example. Or do the same in the cloud to scale internet video load.
UDP is seemingly more appropriate for live / multicast videos.
This SO question on the same topic has a pretty good accepted answer.
Edit: I don't specifically know what strategy Twitch is using. But I could see a use case for HTTP (TCP) live streaming. It is much easier to support with existing tools and through browsers. With UDP you would need to develop your own protocol on top of it (and maybe client) if one of the existing ones (mostly for teleconf?) did not quite fit your use case.
Edit2: Currently you cannot make UDP connections through the browser, period. So using UDP, the user would have to download and install a custom app to watch streams. Twitch probably didn't want to require that of users. The closest available browser-based capability is WebRTC, but it is not widely supported as yet.
I believe UDP is ideal for teleconferencing for the reasons you described, but for live video like Twitch and other live video streams, the content is first sent to a CDN and then distributed from there where the HTTP transfer is more native.
Not exactly my area of expertise at all, but I think that's the main difference here. Twitch etc. will run through a service like Akamai.
Short answer: Neither directly. Mostly HTTP (and thus TCP) due to the limitations of browsers etc, but some could use WebRTC and other technologies to improve performance now.
Longer answer:
First, video. Video works by sending a "key frame", following by a bunch of delta frames. If you know the basic parameters of the video (size, colour depth, etc) you can start playing at any key frame - but if you lose the key frame you have to wait until you get the next one.
TCP operates by creating a "virtual circuit", a continuous data stream containing a sequence of bytes, where each byte won't arrive until the previous ones have done so.
So if I send a sequence of 20 packets, and you miss the first one, the other 19 will be buffered at your end while your computer asks for the missing one.
Great for file transfer, less good for video - but on a lightly loaded network, you're unlikely to lose any packets at all these days, so the downside isn't too bad.
But the longer the stream of video, and the larger your latency (or smaller the bandwidth, or both), the more a lost packet will mess things up for you.
HTTP packetizes data anyway, though, so you can frame your video segments into HTTP responses, and then if one is lost or slow you can discard it if you're falling behind. Conveniently, web browsers speak HTTP, so it's highly interoperable too. There's a minor downside that the increased latency of HTTP means you need to get a few segments in the pipe before you can smoothly play through them all. Typically an application makes segments pretty big, and then allows the client-side to ask for fairly arbitrary segments - and the server-side can "fix" the client requests to the nearest key frame.
But if you really did want live streaming to mean "live" - that is, a low broadcast latency - then HTTP will introduce an unacceptable delay.
Examples might be teleconferencing, video calls, perhaps a very interactive TV show, etc.
For these, you want to use a protocol that will handle lost segments at a smaller, and lower, level.
RTP is built on UDP, and UDP is essentially a fire-and-forget packet - there's very little in the way of error handling in UDP. RTP does add some, and also adds ordering and timing information. Its companion protocol, RTCP, carries control information (like QoS data).
Routers can also spot key frames in RTP - RTP packets can have a Marker bit set which a router can then use to affect queue discards.
Finally, you can send the same RTP packets to everyone watching - in fact, you can use special routing tricks like multicast to save a lot of bandwidth if it's supported.
You can, of course, still lose packets with RTP - but the loss of a few delta packets really won't hurt you at all, and even losing a key frame will cause a minor jolt in the video instead of 30 seconds of "Buffering..."
As WebRTC becomes more popular - and if you're on Chrome or Firefox it's pretty stable already - I'd imagine we'll start to see some "broadcast" applications like Twitch start to experiment with it.
I don't know, but if I wanted to find out, the first thing that would come to my mind is to use a network sniffer like Wireshark to observe what is actually happening.
We've got to remember that UDP and TCP were created at a different time. With today's networks the difference is really not that big
Youtube is using QUIC protocol for transport layer over UDP. It is fast, secure and reliable.