DEV Community

Khoa.Vo
Khoa.Vo

Posted on

Zero-Disk YouTube Downloader: How I Used Rust & Linux Kernel FIFOs for High-Throughput Streaming

#rust #selfhosted #webdev #docker

Zero-Disk YouTube Downloader: How I Used Rust & Linux Kernel FIFOs for High-Throughput Streaming

Most self-hosted YouTube and media downloaders (like MeTube, YoutubeDL-Material, or standard wrapper scripts) follow the same sequential workflow:

  1. Client submits a URL.
  2. Server spawns yt-dlp to download the entire video track and audio track onto the server's hard drive or SSD.
  3. Server executes ffmpeg to mux both files into an .mp4 container on disk.
  4. Server serves the resulting file to the client browser over HTTP.
  5. A background cleanup job eventually deletes the temporary file from disk.

While straightforward, this approach has fatal drawbacks for homelabs and small VPS instances:

  • Severe Disk Wear: Downloading 4K or 1080p60 videos creates gigabytes of redundant read/write cycles on your host SSD.
  • Latency Delay: The user must wait until the entire media file finishes downloading and transcoding on the server before the browser download even begins.
  • YouTube Throttling: Unoptimized single-stream downloads frequently get throttled down to sluggish speeds.

I built KV-DL to rethink this pipeline from the ground up.

⚡ The Solution: The Zero-Disk Kernel FIFO Pipeline

KV-DL does not write a single byte of video or audio to your server's storage. Everything is streamed in-memory and multiplexed in real-time through Linux kernel named pipes (FIFOs):

yt-dlp (Video Stream) ──▶ FIFO #1 ─┐
                                   ├──▶ ffmpeg (RAM Muxer) ──▶ Fragmented MP4 ──▶ Browser Client
yt-dlp (Audio Stream) ──▶ FIFO #2 ─┘
Enter fullscreen mode Exit fullscreen mode

Here is how the end-to-end data flow works:

  1. Format Resolution: When the user enters a video or playlist URL, the Rust backend queries yt-dlp in JSON mode to extract title, thumbnail, and stream bitrate options.
  2. Named Pipe Creation: Upon download request, the backend creates two temporary POSIX FIFOs (mkfifo) inside a volatile in-memory filesystem (/dev/shm or RAM).
  3. Concurrent Stream Feeding: yt-dlp fetches high-speed HTTP chunk ranges from YouTube and pipes raw H.264/AV1 video into FIFO #1 and AAC/Opus audio into FIFO #2.
  4. On-the-Fly Multiplexing: ffmpeg reads directly from both FIFOs simultaneously. It merges the video and audio containers without re-encoding (-c copy) and emits fragmented MP4 (-movflags frag_keyframe+empty_moov) directly to stdout.
  5. Streaming HTTP Response: The Rust Axum web framework streams stdout directly into the HTTP response body as chunked transfer encoding (Transfer-Encoding: chunked).

Why Fragmented MP4?

Standard MP4 files require an index atom (moov atom) positioned at the beginning of the file, which usually requires knowing the exact file size after completion. By using fragmented MP4 (frag_keyframe), chunks can be transmitted to the browser incrementally the moment they are muxed, allowing the client's browser download to start in under 2 seconds!

🍪 In-Memory HMAC Cookie Vault

With YouTube increasingly restricting unauthenticated requests or triggering bot verification, maintaining valid session cookies is essential.

KV-DL includes a frictionless Cookie Vault:

  • Users can paste standard Netscape cookies.txt, JSON exports, or raw HTTP headers directly into the web UI.
  • Cookies are stored purely in volatile RAM, authenticated with an HMAC session key, and injected into yt-dlp requests on the fly.
  • Sensitive credentials are never persisted to disk or plain-text logs.

🔄 Frictionless Domain Swapping

To make downloading effortless, KV-DL supports universal domain swap routing. If you're browsing YouTube and see a video you want to grab, simply replace youtube.com in your address bar with your self-hosted KV-DL domain:

https://www.youtube.com/watch?v=dQw4w9WgXcQ
             ↓↓↓
https://dl.yourdomain.com/watch?v=dQw4w9WgXcQ
Enter fullscreen mode Exit fullscreen mode

The video loads automatically in KV-DL, pre-populated with quality presets and instant download triggers.

🚀 Running KV-DL with Docker

KV-DL is packaged as a single lightweight multi-arch Docker container including the compiled Rust binary, embedded Next.js static UI, yt-dlp, and ffmpeg:

docker run -d \
  --name kv-dl \
  -p 3000:3000 \
  vndangkhoa/kv-dl:latest
Enter fullscreen mode Exit fullscreen mode

Open http://localhost:3000 to start downloading media at full network speed with zero server disk wear.

🔗 Check it Out on GitHub

KV-DL is 100% open-source under the MIT license:

⭐ GitHub Repository: https://github.com/vndangkhoa/kv-dl

🐳 Docker Hub: vndangkhoa/kv-dl

Try it out on your server or homelab!

Top comments (0)