DEV Community: RUSEGAL

Writing terabytes to disk in Go: Stopping the OS Page Cache from eating all your RAM (FADV_DONTNEED)

RUSEGAL — Sun, 09 Aug 2026 06:04:49 +0000

Hello everyone! This is the second article about the development of RUSEON-core, a Zero-Copy video streaming server for AI platforms and Edge video infrastructure. In the first article, I talked about the fundamental reason why we decided to create our own server in the first place. I also covered the main problem with most similar solutions — the "thundering herd" — and how we managed to squeeze out 8 Gbps on a single CPU core. By the way, I forgot to mention in that article that besides simple streaming, we also record the streams in fMP4 format. It’s stored locally for N amount of time, and it can fly off to an S3 bucket (depending on how long the clients want to keep the recordings).

This article is precisely about a non-obvious (well, at least to me, maybe for someone else it's an everyday thing) problem related to data storage and its specifics across all Operating Systems. So, let's dive in.

We rolled out our first release to production (100 cameras), made the clients happy, and started working. About an hour passed, and the alerts started flying. I SSH into the server, open htop, and see there's only 100 MB of free RAM. Uh-oh. I should clarify that the production server had 32 gigs of RAM. The expected behavior was that the CPU is chilling, the network card is chewing through the traffic, RAM usage is around 250-300 MB, and the disks are not heavily loaded. So, when you see numbers like that in htop, you start blaming yourself and your crooked hands that wrote this piece of "garbage". But still, we decided to go to Google, ChatGPT, and the like. Fortunately, the answer was found quickly, and we stopped beating ourselves up.

The code was absolutely not the culprit; Linux itself ate the memory. If you've ever written tons of data to a disk, I think you already know what’s going on. There is an "invisible enemy" known as the Page Cache. That was exactly the root of this problem.

How does the Page Cache work and what to do with it?
When your function that is supposed to write data to the disk actually writes data, it doesn't write it to the disk. It writes it to RAM. The logic of the Linux kernel is simple and trivial, and it is aimed at accelerating the "responsiveness" of the system. The whole essence can be explained like this: "Oh, they just wrote a hundred gigabytes of data, they will probably need to read this data soon. Let me keep it in the cache, the user will be happy they could read it so fast." And so it goes, gigabyte after gigabyte, until the server runs out of physical memory.

The typical solution to the problem is writing a bash script that runs echo 3 > /proc/sys/vm/drop_caches once an hour. And some people just ignore it and let the system kill random processes via the OOM Killer. But we are building a fault-tolerant thing. That doesn't work for us. The task is to explain to the OS kernel that our fMP4 video archive segments are write-only trash for a short amount of time (because if the client doesn't want to keep records for long, the archive gets cleaned up, and if they do, we send the archive to S3 after N time, and still clean up the local copy). So everything should work on the principle of "write and forget".

How to tame the Linux kernel via Go (the right way)
In C/C++, there is a system call for this: posix_fadvise. You can explicitly tell the OS exactly how you will be working with the file. In Go, this doesn't exist out of the box. But there is a very good package, golang.org/x/sys/unix, which allows you to easily replace the system call. The flag is called FADV_DONTNEED. We literally say to the kernel: "We wrote it, flush it to disk and get the hell out of the cache."

But there is one cruel nuance here, which I stumbled upon myself and racked my brain over for a long time (and all it took was reading the docs, but here, just like with assembling IKEA furniture: "Why do I need a manual, I know how to do it myself"). The Linux kernel does not remove pages from the cache if they are so-called "dirty" — meaning they haven't been physically written to the disk platter yet. If you just call Fadvise, nothing will happen.

First, you need to do a hard Sync().

// File: pkg/storage/localfs/file_linux.go
package localfs
import (
    "os"
    "golang.org/x/sys/unix"
)
type FileWrapper struct {
    *os.File
}
func (fw *FileWrapper) DropCache() error {
    // First, flush dirty pages to disk!
    if err := fw.File.Sync(); err != nil {
        return err
    }
    // And only then order the kernel to forget them
    return unix.Fadvise(int(fw.File.Fd()), 0, 0, unix.FADV_DONTNEED)
}

Architecture: How not to break the build on Windows
System calls are almost always a huge cross-platform headache. On Windows, the FADV_DONTNEED flag simply does not exist (hello there, Microsoft! Are you guys doing okay?). If you don't split the code for different OSs, the compiler will just tell you to get lost.

Therefore, a rather elegant (I invite you to argue this statement in the comments) interface was implemented. In the core of the recorder, there is now a check to see if the file descriptor knows how to drop the cache:

// OPTIMIZATION: Saving RAM from Page Cache
if dropper, ok := file.(registry.CacheDropper); ok {
    _ = dropper.DropCache()
}

And then comes the magic of Go build tags (thank you for those). In the file_linux.go file (with the //go:build linux tag), we call unix.Fadvise. And right next to it lies the file_others.go file (with the //go:build !linux tag), where the DropCache() method just does a regular Sync() and returns. The code is crystal clear, the linters are happy, and the build works everywhere.

So, what was the result?
We roll out the fixes, launch the exact same 100 cameras. I open the dashboard. The memory consumption graph looks almost like a perfect straight line. The server grabbed its rightful 250 megs for the Go process heap — and that's it. Hooray, victory! There is no more massive Page Cache growth. No processes are being evicted to swap. The server honestly writes tens of gigabytes per hour, and RAM is at peace.

By the way, when you are backing up databases, parsing giant logs, or simply writing heavy files — this feature will save you a mountain of headaches. I also don't understand why this flag is barely talked about or written about anywhere. Usually, people only write about how to properly allocate slices, but there is complete silence regarding the fact that at the level of file operations, your OS can reduce all your efforts to zero.

In short, in the open-source part of ruseon-core, this logic is now wired deep into the recording engine. The conclusion and advice I want to give is — don't blindly trust the kernel with memory, hoping that the OS is hypothetically "perfect and maximally thought out." Sometimes you have to slap the kernel on the wrist. Otherwise, you will run into similar problems as I did.

The source code, as always, is available on GitHub: https://github.com/RUSEGAL/ruseon-core

How We Served 8 Gbps of Video on a Single Go CPU Core (And Survived the Thundering Herd)

RUSEGAL — Fri, 07 Aug 2026 09:24:02 +0000

Foreword

There is a small side project I've been working on – relaying raw RTSP streams into HLS. Essentially, the goal is simple: allow any ordinary user to view the feed from their cameras without using proprietary cloud software from the camera manufacturer. First, that costs a lot of money, as many clients want to store camera recordings for quite a long time. Second, almost all camera manufacturers have different software, and it's simply inconvenient to have 10 different apps and/or portals just to view it all. And third, very few offer AI integrations (which is critically important for my clients). Even if this integration exists, it is either highly specialized, proprietary again, or costs a lot of money—and sometimes all of these combined. The solution is to use raw RTSP streams; 90% of all cameras on the market support and provide them.

So, the setup was quite simple: a few cameras, a simple backend, and we serve the video via HLS on a single resource for clients, using the hls.js player. Everyone was happy with it, everyone liked it. Fast, simple, convenient, and cheap. Tests worked perfectly, clients were satisfied. There was a chat with each client where the viewing link was shared, as well as a general chat with all clients where connection issues, financials, and other things were discussed. And then the moment of truth arrived: someone mistakenly dropped a link into the general chat... and that was the end. Apparently, the entire chat clicked on the link. 3 minutes in, alerts started flying in the bot: alarm, achtung, panic. We check the hardware: the server is down, OOM, all streams dropped. 20 minutes later, the chat was exploding with angry messages about nothing working for anyone. Curtain drop.

Welcome to the Thundering Herd problem.

If any of you have tried serving live video, you definitely know this problem. When hundreds or even thousands of viewers try to watch the same stream, a standard server's logic is simple – open a new connection or spawn a separate process for each viewer. Generally, the result of this action is always the same – OOM, and the processor is dead.
The simplest way to solve this would have been renting a more powerful cloud instance or buying new hardware. There's a saying that "if a problem can be solved with money, it's not a problem," but there was no money. Plus, we are engineers; consider it a challenge thrown at your professionalism. The idea emerged to write our own engine in GO. It’s cheap, and you can squeeze out maximum performance. And honestly, your own thing in GO – it's cool, trendy, and hip. The spoiler won't be long: in the end, we squeezed out 8.8 Gbps on a single CPU core, while the garbage collector was basically taking a vacation during all this.

Here’s how Ruseon Core was put together.

Why not the great and mighty FFmpeg?

Let’s be honest, FFmpeg is an "axiom" in the world of any video processing. It's awesome, truly. Essentially, it's the standard for ages. But my god, does it love to "eat." Launching even a hundred FFmpeg processes for restreaming is equivalent to suicide. We specifically want to get away from the same problems it has. Even if you use it as an ingest proxy, the overhead will zero out all the benefits.
While searching for a solution, MediaMTX came up. It’s a gorgeous project and essentially solves our "main" problem. But (I think "but" is becoming my favorite word, hehe), we need seamless integration with the AI pipeline, as well as archive recording. Out of the box, it doesn't have this, and writing plugins or wrappers takes a long time. Plus, what's the point if we already decided to solve the problem conceptually? Besides, "showing off" to clients that we only use our own proprietary development is a sweet deal. It also adds weight in the eyes of other engineers and companies.
So we wrote an engine from scratch, while trying to keep it "modular." That means the ability to embed it or use it as an SDK. The main rule during development – absolutely no transcoding (we just move bytes around), minimum overhead, maximum performance.

The main feature: Zero-Copy RingBuffer

When a new frame arrives from the camera (usually it's H.264, though we implemented the 265 codec too), it’s just another chunk of bytes.
Imagine this "brilliant" idea: we distribute this chunk to thousands of viewers by copying the bytes into separate response buffers. Pictured it, right? In the world of GO – that's a path to success (sarcasm). The garbage collector will be absolutely thrilled, trying somehow to deal with a mountain of trash. The CPU, instead of streaming, will be entirely busy cleaning up.
We decided to go a different route. We made a Zero-Copy RingBuffer and attached a sync.Pool to it.

How it works in theory:
A frame arrives from the camera.
We take an empty byte buffer from a pre-allocated pool.
We write the frame into it (just once!).
We hand out a pointer to this buffer to a thousand viewers.
When everyone has read it — we return the buffer back to the pool.

No allocations. No garbage collection pauses. 250 MB of RAM, 1-2% CPU load at 100 streams.

In benchmarks, it looks like this:

BenchmarkWriteFrame-12 13.9 ns/op 0 B/op 0 allocs/op

Here you can experience a true engineering "orgasm." When streaming video, seeing zero data volume per operation. When the processor is grinding out tens of thousands of frames, and the heap remains as minimal as it was.

Tests and load testing, where would we be without them?

I'm telling you all about how this was implemented, but we all love numbers (especially the numbers in a bank account). Writing fast code is cool, but you need to understand how it works in reality and where its limit is. For testing, we chose k6 by Grafana; it allows us to emulate exactly the problem that started this whole project.
Test scenario: 1000 users hammering our muxer, constantly downloading the playlist (index.m3u8), and snatching megabyte-sized .ts chunks as fast as the server allows.
There were thoughts that the "bottleneck" would start at 300 users... but thankfully I was wrong.

70 seconds of testing on a single core of a workstation Ryzen 5600x:

data_received..................: 81 GB  1.1 GB/s
http_req_failed................: 0.00%  ✓ 0 ✗ 60822
http_req_duration..............: avg=3.13ms p(95)=6.13ms

That is 8.8 Gbps of throughput (Carl!). 60 thousand successful HTTP responses. Not a single dropped connection.
Big numbers, looks nice, but what’s the catch? It’s simple here, the HLS segments are just sitting in RAM. When a massive request for a segment occurs, the server simply serves the exact same bytes from the cache. That’s it. The disk is resting, there's no repackaging, the processor is ordering a whiskey and cola.

Nuances

The question of infinite buffer accumulation arises, but there won't be infinite buffer accumulation (and subsequent OOM), as some protection against this is implemented:
The base Go network stack (net/http), which essentially operates at the delivery level. The server just hands a static chunk of memory to the socket. That's it. Whether it downloads fast or slow - it doesn't matter.

But at the core level (Ring.go), the protection implementation is more interesting - inside the core, subscribers (HLS Muxer or AI workers) read frames through channels in Go. Writing to the channel is implemented as a non-blocking send (via select + default). The channel has a strictly defined depth. If a subscriber lags, its channel gets clogged, the core doesn't wait for it and doesn't allocate new memory. Here the default branch kicks in - the frame is dropped for that specific subscriber. Here, unfortunately, you have to "sacrifice" a bad client for the sake of, say, 1000 "good" ones.
There's also the common problem of resynchronization and lag, which is solved as follows. If a subscriber inside the core drops a frame due to lagging, they cannot be given the next P-frame, because the picture will fall apart (by the way, this is even mentioned in gortsplib, the RTSP library used in the project). In this case, the core sets the NeedsIFrame flag to true for them. The subscriber stays silent until the next I-frame (keyframe) arrives; from there, reading resumes cleanly and with minimal losses. In real-world operation, this takes milliseconds and is essentially unnoticeable to the end viewer. Especially since the muxer keeps a "sliding window" of the last 5 segments in memory. If the end viewer has a really bad connection, when trying to download an outdated segment, they will get a 404. The player on the client (in 99% of cases, the player users have is hls.js or its implementations) catches the 404, realizes it has fallen behind the live feed, and jumps to the current segment, synchronizing with the rest.

And an important point: the HLS protocol is a PULL implementation. This means that an infinite accumulation of lag will not happen. That happens when the server tries to shove data into a socket, the socket gets blocked due to the client's poor network, packets pile up in the queue, and when the network "clears its throat," the client starts watching a video from 10 minutes ago. The mechanics here are different:
The muxer keeps a sliding window (Live Playlist) in memory — for example, only the last 5 segments (let's say, 10 seconds of video). Older segments are deleted forever.
A client with a terrible connection pulls segment 100 for more than 3 minutes.
Downloaded it, the player asks for the next segment, #101.
But the freshest on the server is 200. Segment 101 is physically no longer in memory.
The server serves an honest 404.
The player catches the 404, downloads a fresh index.m3u8, sees that the current segment is already 200, and jumps to the Live edge.

A client with a bad network will simply see constant "jumps" forward and buffering (which is logical with a dead network), but they will not force our server to store a personal 10-minute cache for them.

Why is this not a perfect mechanism?

In this implementation, your CPU is essentially chilling, load is at 1-2%. But at the same time, the load on the network will be colossal. Based on tests, 9 Gbps is basically the limit of a 10 Gigabit interface. The code will hold up, but the network card will start dropping packets. Physics has its limits. Also, don't forget that when we write about 0 B/op and 250 MB of RAM consumption for 100 streams, this refers only to userspace memory (the heap), the one the garbage collector is responsible for. Socket buffers aren't going anywhere. Hundreds or thousands of connections will eat up their rightful megabytes of memory for tcp_mem. This rather describes that the problem doesn't multiply at the software level, i.e., we are not allocating gigabytes of structures inside GO. You have to understand that OS memory is an unavoidable tax on networking.
The trickiest part is not messing up the sync.Pool implementation. Otherwise, your picture will just fall apart, and some frames will simply be green. All because another thread is already writing new bytes in there.
In short, the point of this article is that if you don't need transcoding, or you need minimal hardware load – then you shouldn't use heavy artillery like FFmpeg and the like. Control your memory. Move bytes around without copying.
The source code and load testing scripts are located in the Ruseon Core repository in the benchmarks/ folder. Go ahead and try to crash your computer (I've killed mine more than once).

Source code: https://github.com/RUSEGAL/ruseon-core