Migrating a Live TV & Radio Streaming Platform from VM to Kubernetes

Introduction

When our team was tasked with migrating a production live streaming platform to Kubernetes, we knew it wouldn't be straightforward. This platform records 50+ live TV and radio channels 24/7, processes them with AI models, and serves them to thousands of users in real time.

This article covers our complete journey — the architecture decisions, the unexpected challenges, and the solutions we found along the way. If you're working with live streaming, Kubernetes, or both, this should save you significant time.

What We Were Migrating

Our platform, built for a government media organization, does the following:

• Records 50+ live TV channels (UDP multicast) and radio stations (HTTP streams) continuously

• Processes recordings with AI models — Speech-to-Text, Voice Sentiment, Speaker Recognition, OCR

• Serves live HLS streams to a React frontend

• Stores everything on TrueNAS NFS shared storage

• Runs background tasks via Celery for channel management and playback date updates

Old setup:Single Ubuntu VM running everything via Docker Compose
New setup:RKE2 Kubernetes cluster with Calico CNI, 5 master nodes, 8 worker nodes

Old Architecture (VM-based)

UDP Multicast (239.x.x.x)
        ↓
VM receives multicast directly
        ↓
Docker Compose services:
  - gstream (new_gstream.py) → records TV channels
  - radio-stream (radio_streams.py) → records radio
  - clipping-backend (Django/Gunicorn)
  - celery-worker + celery-beat
  - nginx-media (serves HLS files)
  - redis
        ↓
NFS mount at /app/LIVEFTP
        ↓
Frontend plays live.m3u8

Simple, but had limitations:

• Single point of failure

• No auto-scaling

• Manual restarts when services crashed

• No CI/CD pipeline

New Architecture (Kubernetes)

UDP Multicast (239.x.x.x)
        ↓
MediaMTX DaemonSet (hostNetwork: true)
converts UDP → RTSP on every node
        ↓
gstream Deployment
pulls RTSP via cluster DNS
rtsp://mediamtx-service.svc.cluster.local:8554/channel
        ↓
radio-stream Deployment
pulls HTTP streams directly
        ↓
TrueNAS NFS (PV/PVC)
/mnt/truenas/clipping-server2
        ↓
nginx-media Deployment (2 replicas)
serves HLS files
        ↓
Ingress Controller
clipping-nginx.k8s.local
        ↓
Frontend plays stream

Challenge 1: UDP Multicast in Kubernetes
This was our biggest challenge. Our TV channels use UDP multicast (udp://239.x.x.x:port). Multicast traffic works at the network level — a single stream is sent to a multicast group address, and any device that "joins" the group receives it.

The problem:Kubernetes pods have their own network namespace. They cannot receive multicast traffic by default because:

• Pods get their own virtual network interface

• Multicast join requests don't reach the physical network interface

• Standard CNI plugins don't bridge multicast to pod networks

What we tried first:Running gstream with hostNetwork: true
This worked but meant the pod used the node's network directly — defeating the purpose of Kubernetes networking and creating port conflicts.

Our solution: MediaMTX DaemonSet
MediaMTX is a production-ready media server that can receive multicast UDP and re-stream it as RTSP, HLS, WebRTC, or SRT.

We deployed it as a DaemonSet with hostNetwork: true — but only MediaMTX needed host networking, not our application pods:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: mediamtx
  namespace: staging-nimar-clipping
spec:
  selector:
    matchLabels:
      app: mediamtx
  template:
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - name: mediamtx
        image: bluenviron/mediamtx:latest
        volumeMounts:
        - name: mediamtx-config
          mountPath: /mediamtx.yml
          subPath: mediamtx.yml

protocols: [tcp]
rtspAddress: :8554
api: true
apiAddress: :9997

paths:
  ary-news:
    source: udp://239.1.1.9:1111
  ptv-news:
    source: udp://239.1.1.10:1111
  # ... all 50+ channels

Now gstream could pull streams normally via cluster DNS:

rtsp://mediamtx-service.staging-nimar-clipping.svc.cluster.local:8554/ary-news

No hostNetwork needed for application pods. Clean k8s networking.

Challenge 2: inotify Limit on Nodes

After deploying MediaMTX, some pods on certain nodes kept crashing with:
ERR: couldn't initialize inotify: too many open files

Root cause:Linux has a default limit of 128 inotify instances per user (fs.inotify.max_user_instances). Our nodes were already running many pods, each consuming inotify instances. When MediaMTX tried to start on those nodes, the limit was already hit.

Diagnosis:
sysctl fs.inotify.max_user_instances
fs.inotify.max_user_instances = 128

Fix:
#Immediate fix
sudo sysctl -w fs.inotify.max_user_instances=1280

Permanent fix

echo "fs.inotify.max_user_instances=1280" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p

After applying this to all affected nodes, all MediaMTX pods came up healthy. This is a common issue in production Kubernetes clusters — always increase inotify limits on your nodes.

Challenge 3: Stream URLs Stored in Database

Our application (new_gstream.py) reads stream URLs from the database:

channel = Channels.objects.get(id=channel_id)
input_url = channel.rtpUrl # reads from DB

On VM, the DB had:
rtpUrl = udp://239.1.1.9:1111

In Kubernetes, gstream pods can't receive multicast. We needed to update all TV channel URLs in the database to point to MediaMTX:

from live.models import Channels

channels = Channels.objects.filter(source='tv')
for ch in channels:
    channel_slug = ch.display_name.lower().replace(' ', '-')
    ch.rtpUrl = f'rtsp://mediamtx-service.staging-nimar-clipping.svc.cluster.local:8554/{channel_slug}'
    ch.save()
    print(f'Updated: {ch.display_name}')

Key lesson:Before migrating, always audit where your app reads its configuration from — environment variables, files, or database. Database-stored config requires data migration alongside code migration.

Challenge 4: Celery Beat Duplicate Schedules

Celery Beat kept crashing with:

MultipleObjectsReturned: get() returned more than one IntervalSchedule -- it returned 6!

Root cause:The application code was using IntervalSchedule.objects.create() instead of get_or_create(). Every time the service restarted (deployments, rollouts), it created a new duplicate schedule entry in the database

Quick fix:

from django_celery_beat.models import IntervalSchedule

seen = {}
for obj in IntervalSchedule.objects.all():
    key = (obj.every, obj.period)
    if key in seen:
        obj.delete()
    else:
        seen[key] = obj.id

Permanent fix (for developers):

# Wrong
IntervalSchedule.objects.create(every=5, period='minutes')

# Correct
schedule, created = IntervalSchedule.objects.get_or_create(
    every=5,
    period='minutes'
)

Final Kubernetes Manifest Structure

staging-nimar-clipping/
├── namespace.yaml
├── secret.yaml          # all env vars
├── pv-pvc.yaml          # NFS storage
├── nginx-config.yaml    # ConfigMap
├── mediamtx-daemonset.yaml
├── gstream-deployment.yaml
├── radio-stream-deployment.yaml
├── clipping-backend-deployment.yaml
├── celery-worker-deployment.yaml
├── celery-beat-deployment.yaml
└── nginx-media-deployment.yaml

Results

Service      Replicas   Status
MediaMTX      8 ✅ Running
gstream           1 ✅ Running
radio-stream      1 ✅ Running
clipping-backend  1 ✅ Running
celery-worker     1 ✅ Running
celery-beat   1 ✅ Running
nginx-media   2 ✅ Running