Foreword
There is a small side project I've been working on – relaying raw RTSP streams into HLS. Essentially, the goal is simple: allow any ordinary user to view the feed from their cameras without using proprietary cloud software from the camera manufacturer. First, that costs a lot of money, as many clients want to store camera recordings for quite a long time. Second, almost all camera manufacturers have different software, and it's simply inconvenient to have 10 different apps and/or portals just to view it all. And third, very few offer AI integrations (which is critically important for my clients). Even if this integration exists, it is either highly specialized, proprietary again, or costs a lot of money—and sometimes all of these combined. The solution is to use raw RTSP streams; 90% of all cameras on the market support and provide them.
So, the setup was quite simple: a few cameras, a simple backend, and we serve the video via HLS on a single resource for clients, using the hls.js player. Everyone was happy with it, everyone liked it. Fast, simple, convenient, and cheap. Tests worked perfectly, clients were satisfied. There was a chat with each client where the viewing link was shared, as well as a general chat with all clients where connection issues, financials, and other things were discussed. And then the moment of truth arrived: someone mistakenly dropped a link into the general chat... and that was the end. Apparently, the entire chat clicked on the link. 3 minutes in, alerts started flying in the bot: alarm, achtung, panic. We check the hardware: the server is down, OOM, all streams dropped. 20 minutes later, the chat was exploding with angry messages about nothing working for anyone. Curtain drop.
Welcome to the Thundering Herd problem.
If any of you have tried serving live video, you definitely know this problem. When hundreds or even thousands of viewers try to watch the same stream, a standard server's logic is simple – open a new connection or spawn a separate process for each viewer. Generally, the result of this action is always the same – OOM, and the processor is dead.
The simplest way to solve this would have been renting a more powerful cloud instance or buying new hardware. There's a saying that "if a problem can be solved with money, it's not a problem," but there was no money. Plus, we are engineers; consider it a challenge thrown at your professionalism. The idea emerged to write our own engine in GO. It’s cheap, and you can squeeze out maximum performance. And honestly, your own thing in GO – it's cool, trendy, and hip. The spoiler won't be long: in the end, we squeezed out 8.8 Gbps on a single CPU core, while the garbage collector was basically taking a vacation during all this.
Here’s how Ruseon Core was put together.
Why not the great and mighty FFmpeg?
Let’s be honest, FFmpeg is an "axiom" in the world of any video processing. It's awesome, truly. Essentially, it's the standard for ages. But my god, does it love to "eat." Launching even a hundred FFmpeg processes for restreaming is equivalent to suicide. We specifically want to get away from the same problems it has. Even if you use it as an ingest proxy, the overhead will zero out all the benefits.
While searching for a solution, MediaMTX came up. It’s a gorgeous project and essentially solves our "main" problem. But (I think "but" is becoming my favorite word, hehe), we need seamless integration with the AI pipeline, as well as archive recording. Out of the box, it doesn't have this, and writing plugins or wrappers takes a long time. Plus, what's the point if we already decided to solve the problem conceptually? Besides, "showing off" to clients that we only use our own proprietary development is a sweet deal. It also adds weight in the eyes of other engineers and companies.
So we wrote an engine from scratch, while trying to keep it "modular." That means the ability to embed it or use it as an SDK. The main rule during development – absolutely no transcoding (we just move bytes around), minimum overhead, maximum performance.
The main feature: Zero-Copy RingBuffer
When a new frame arrives from the camera (usually it's H.264, though we implemented the 265 codec too), it’s just another chunk of bytes.
Imagine this "brilliant" idea: we distribute this chunk to thousands of viewers by copying the bytes into separate response buffers. Pictured it, right? In the world of GO – that's a path to success (sarcasm). The garbage collector will be absolutely thrilled, trying somehow to deal with a mountain of trash. The CPU, instead of streaming, will be entirely busy cleaning up.
We decided to go a different route. We made a Zero-Copy RingBuffer and attached a sync.Pool to it.
How it works in theory:
A frame arrives from the camera.
We take an empty byte buffer from a pre-allocated pool.
We write the frame into it (just once!).
We hand out a pointer to this buffer to a thousand viewers.
When everyone has read it — we return the buffer back to the pool.
No allocations. No garbage collection pauses. 250 MB of RAM, 1-2% CPU load at 100 streams.
In benchmarks, it looks like this:
BenchmarkWriteFrame-12 13.9 ns/op 0 B/op 0 allocs/op
Here you can experience a true engineering "orgasm." When streaming video, seeing zero data volume per operation. When the processor is grinding out tens of thousands of frames, and the heap remains as minimal as it was.
Tests and load testing, where would we be without them?
I'm telling you all about how this was implemented, but we all love numbers (especially the numbers in a bank account). Writing fast code is cool, but you need to understand how it works in reality and where its limit is. For testing, we chose k6 by Grafana; it allows us to emulate exactly the problem that started this whole project.
Test scenario: 1000 users hammering our muxer, constantly downloading the playlist (index.m3u8), and snatching megabyte-sized .ts chunks as fast as the server allows.
There were thoughts that the "bottleneck" would start at 300 users... but thankfully I was wrong.
70 seconds of testing on a single core of a workstation Ryzen 5600x:
data_received..................: 81 GB 1.1 GB/s
http_req_failed................: 0.00% ✓ 0 ✗ 60822
http_req_duration..............: avg=3.13ms p(95)=6.13ms
That is 8.8 Gbps of throughput (Carl!). 60 thousand successful HTTP responses. Not a single dropped connection.
Big numbers, looks nice, but what’s the catch? It’s simple here, the HLS segments are just sitting in RAM. When a massive request for a segment occurs, the server simply serves the exact same bytes from the cache. That’s it. The disk is resting, there's no repackaging, the processor is ordering a whiskey and cola.
Nuances
The question of infinite buffer accumulation arises, but there won't be infinite buffer accumulation (and subsequent OOM), as some protection against this is implemented:
The base Go network stack (net/http), which essentially operates at the delivery level. The server just hands a static chunk of memory to the socket. That's it. Whether it downloads fast or slow - it doesn't matter.
But at the core level (Ring.go), the protection implementation is more interesting - inside the core, subscribers (HLS Muxer or AI workers) read frames through channels in Go. Writing to the channel is implemented as a non-blocking send (via select + default). The channel has a strictly defined depth. If a subscriber lags, its channel gets clogged, the core doesn't wait for it and doesn't allocate new memory. Here the default branch kicks in - the frame is dropped for that specific subscriber. Here, unfortunately, you have to "sacrifice" a bad client for the sake of, say, 1000 "good" ones.
There's also the common problem of resynchronization and lag, which is solved as follows. If a subscriber inside the core drops a frame due to lagging, they cannot be given the next P-frame, because the picture will fall apart (by the way, this is even mentioned in gortsplib, the RTSP library used in the project). In this case, the core sets the NeedsIFrame flag to true for them. The subscriber stays silent until the next I-frame (keyframe) arrives; from there, reading resumes cleanly and with minimal losses. In real-world operation, this takes milliseconds and is essentially unnoticeable to the end viewer. Especially since the muxer keeps a "sliding window" of the last 5 segments in memory. If the end viewer has a really bad connection, when trying to download an outdated segment, they will get a 404. The player on the client (in 99% of cases, the player users have is hls.js or its implementations) catches the 404, realizes it has fallen behind the live feed, and jumps to the current segment, synchronizing with the rest.
And an important point: the HLS protocol is a PULL implementation. This means that an infinite accumulation of lag will not happen. That happens when the server tries to shove data into a socket, the socket gets blocked due to the client's poor network, packets pile up in the queue, and when the network "clears its throat," the client starts watching a video from 10 minutes ago. The mechanics here are different:
The muxer keeps a sliding window (Live Playlist) in memory — for example, only the last 5 segments (let's say, 10 seconds of video). Older segments are deleted forever.
A client with a terrible connection pulls segment 100 for more than 3 minutes.
Downloaded it, the player asks for the next segment, #101.
But the freshest on the server is 200. Segment 101 is physically no longer in memory.
The server serves an honest 404.
The player catches the 404, downloads a fresh index.m3u8, sees that the current segment is already 200, and jumps to the Live edge.
A client with a bad network will simply see constant "jumps" forward and buffering (which is logical with a dead network), but they will not force our server to store a personal 10-minute cache for them.
Why is this not a perfect mechanism?
In this implementation, your CPU is essentially chilling, load is at 1-2%. But at the same time, the load on the network will be colossal. Based on tests, 9 Gbps is basically the limit of a 10 Gigabit interface. The code will hold up, but the network card will start dropping packets. Physics has its limits. Also, don't forget that when we write about 0 B/op and 250 MB of RAM consumption for 100 streams, this refers only to userspace memory (the heap), the one the garbage collector is responsible for. Socket buffers aren't going anywhere. Hundreds or thousands of connections will eat up their rightful megabytes of memory for tcp_mem. This rather describes that the problem doesn't multiply at the software level, i.e., we are not allocating gigabytes of structures inside GO. You have to understand that OS memory is an unavoidable tax on networking.
The trickiest part is not messing up the sync.Pool implementation. Otherwise, your picture will just fall apart, and some frames will simply be green. All because another thread is already writing new bytes in there.
In short, the point of this article is that if you don't need transcoding, or you need minimal hardware load – then you shouldn't use heavy artillery like FFmpeg and the like. Control your memory. Move bytes around without copying.
The source code and load testing scripts are located in the Ruseon Core repository in the benchmarks/ folder. Go ahead and try to crash your computer (I've killed mine more than once).
Source code: https://github.com/RUSEGAL/ruseon-core
Top comments (0)