Building a Video Transcoding Job Queue with Python RQ and Redis

When one 4K upload freezes your whole request thread

The first time video transcoding took down a page for us, the cause was embarrassingly simple. A contributor dropped a 12-minute 4K clip into our ingest endpoint, our PHP application shelled out to ffmpeg synchronously, and LiteSpeed killed the HTTP request at its 180-second limit while ffmpeg was still pinning every core on the box. From the browser the upload "failed" — but the process kept running, and two more uploads landed on top of it. Within a minute the server was swapping and the public site was serving 503s to real visitors.

That is the moment you learn that transcoding is not a request-response task. It is a background job, and it needs a real queue. At ViralVidVault we ingest trending clips from creators across a dozen European regions, normalise each one into an adaptive-bitrate ladder, strip personal metadata for GDPR, and only then publish. This post is the queue we actually run in production: Python RQ on top of Redis, fed by our PHP 8.4 application, with a small Go exporter watching the depth. I'll show the real code and the sharp edges most tutorials skip — idempotency, warm shutdown, retry backoff, and the PHP-to-Python bridge that trips everyone up.

Why RQ instead of Celery

We evaluated Celery first, because everyone reaches for it. For a transcoding queue specifically, RQ won, and the reasons are worth stating because they are the reasons your choice might differ:

• The unit of work is coarse and long. A single transcode job runs for 30 seconds to 20 minutes. We do not need microsecond dispatch latency or fan-out primitives. We need durable, observable, retryable jobs. RQ gives exactly that with almost no configuration surface.
• Redis is already in the stack. We use it for rate limiting and for caching hot trend queries. Adding RQ meant one more db index, not one more piece of infrastructure to operate, back up, and monitor.
• The failure model is legible. RQ keeps failed jobs in a registry you can inspect with plain Redis commands. When ffmpeg dies on a corrupt upload at 3am, I can see the exact stderr tail without attaching a debugger to a worker.
• Introspection is trivial. Queue depth is a LLEN. The failed set is a ZCARD. That let us write a monitoring exporter in an afternoon instead of leaning on Flower.

Celery is the better tool the day you need routing keys, chords, or a non-Redis broker. We do not, so we did not pay for the complexity.

The shape of the pipeline

The end-to-end flow has four hops, and keeping them decoupled is what makes the whole thing survivable:

1. The PHP app receives an upload, writes the source file to a local ingest path, and pushes a small JSON message onto a plain Redis list. It returns 202 Accepted immediately.
2. A Python dispatcher does a blocking pop off that list and translates the message into a real RQ job. This bridge exists for one specific reason I'll explain below.
3. One or more RQ workers pull jobs and run the actual ffmpeg renditions, reporting progress into the job's metadata.
4. A Go exporter polls Redis and publishes queue depth, backlog, and failure counts for alerting.

The PHP request never blocks on ffmpeg, the worker never touches HTTP, and either side can be restarted independently. That last property is the whole point.

The transcode job itself

The job function is ordinary Python. RQ serialises a reference to it plus its arguments, so the function has to be importable by the worker — no closures, no lambdas. Here is the real ladder we run, trimmed to three renditions for readability:

# transcode.py
import os
import subprocess
from pathlib import Path
from rq import get_current_job

RENDITIONS = [
    {"name": "1080p", "height": 1080, "v": "5000k", "a": "128k"},
    {"name": "720p",  "height": 720,  "v": "2800k", "a": "128k"},
    {"name": "480p",  "height": 480,  "v": "1400k", "a": "96k"},
]

def transcode_video(source_path: str, out_dir: str, clip_id: str) -> dict:
    job = get_current_job()
    Path(out_dir).mkdir(parents=True, exist_ok=True)
    results = []

    for i, r in enumerate(RENDITIONS):
        target = os.path.join(out_dir, f"{clip_id}_{r['name']}.mp4")
        cmd = [
            "ffmpeg", "-y", "-i", source_path,
            "-vf", f"scale=-2:{r['height']}",
            "-c:v", "libx264", "-preset", "veryfast", "-b:v", r["v"],
            "-c:a", "aac", "-b:a", r["a"],
            "-map_metadata", "-1",     # GDPR: drop EXIF / GPS / author tags
            "-movflags", "+faststart", # move moov atom to front for streaming
            target,
        ]
        proc = subprocess.run(cmd, capture_output=True, text=True)
        if proc.returncode != 0:
            # Surface only the tail; ffmpeg stderr is enormous.
            raise RuntimeError(f"ffmpeg {r['name']} failed: {proc.stderr[-500:]}")

        results.append({
            "rendition": r["name"],
            "path": target,
            "bytes": os.path.getsize(target),
        })

        if job is not None:
            job.meta["progress"] = round((i + 1) / len(RENDITIONS) * 100)
            job.save_meta()

    return {"clip_id": clip_id, "renditions": results}

Three details matter here. -preset veryfast is a deliberate throughput-over-size tradeoff — on a viral discovery site the clip may never get 10,000 views, so spending CPU minutes chasing a smaller file is a bad bet. -movflags +faststart rewrites the file so playback can begin before the full download completes, which is non-negotiable for streaming. And save_meta() is how we surface progress; the worker writes it back to Redis after each rendition so the frontend can poll a real percentage instead of a spinner.

Enqueuing from a PHP application

Here is the wall everyone hits: RQ serialises jobs with Python's pickle. You cannot construct a valid RQ job payload from PHP, because PHP cannot produce a Python pickle of a Python callable. Every "just push to rq:queue:... from PHP" snippet you find online is subtly wrong and will hand the worker a payload it silently discards.

The clean fix is a thin bridge. PHP speaks a language-neutral protocol — a JSON message on an ordinary Redis list — and a tiny Python process owns the RQ-specific enqueue. The producer side is boring, which is exactly what you want in the code path that runs inside a web request:

<?php
// TranscodeProducer.php — PHP 8.4
declare(strict_types=1);

final class TranscodeProducer
{
    public function __construct(private readonly Redis $redis) {}

    public function submit(string $clipId, string $sourcePath): void
    {
        $payload = json_encode([
            'clip_id'      => $clipId,
            'source'       => $sourcePath,
            'region'       => 'eu-central',
            'submitted_at' => gmdate('c'),
        ], JSON_THROW_ON_ERROR);

        // Plain list push. The Python dispatcher owns everything RQ-specific.
        $this->redis->lPush('transcode:incoming', $payload);
    }
}

$redis = new Redis();
$redis->connect('127.0.0.1', 6379);
(new TranscodeProducer($redis))->submit('clip_8842', '/var/ingest/clip_8842.mov');

The dispatcher does a blocking pop so it costs nothing while idle, then translates each message into a real RQ job. This is also the natural place to enforce idempotency and set every per-job policy in one spot:

# dispatcher.py
import json
import redis
from rq import Queue, Retry
from transcode import transcode_video

conn = redis.Redis(host="127.0.0.1", port=6379, db=0)
q = Queue("transcode", connection=conn)

def run():
    while True:
        # 5s timeout so SIGTERM can break the loop cleanly on deploy.
        item = conn.brpop("transcode:incoming", timeout=5)
        if item is None:
            continue

        _, raw = item
        msg = json.loads(raw)
        q.enqueue(
            transcode_video,
            args=(msg["source"], f"/var/renditions/{msg['clip_id']}", msg["clip_id"]),
            job_id=f"transcode:{msg['clip_id']}",  # idempotency: dupes collapse
            job_timeout=1800,                       # 30 min hard ceiling
            result_ttl=86400,                       # keep results 24h
            failure_ttl=604800,                     # keep failures 7d for triage
            retry=Retry(max=3, interval=[30, 120, 300]),
        )

if __name__ == "__main__":
    run()

Setting an explicit job_id derived from the clip is what gives us idempotency for free: if a flaky upload retry pushes the same clip twice, RQ collapses both into a single job instead of transcoding the file twice and racing on the output directory. That one line has saved us more wasted CPU than any encoder flag.

Workers, warm shutdown, and cleanup

A worker is one line to start — rq worker transcode — but running it correctly under deploys is where the real work is. Two failure modes bite hard: a deploy that kills a worker mid-ffmpeg, and a job that dies leaving half-written renditions on disk that poison the retry.

# worker.py
import os
import shutil
import redis
from rq import Worker, Queue
from rq.job import Job

conn = redis.Redis(host="127.0.0.1", port=6379, db=0)

def cleanup_handler(job: Job, exc_type, exc_value, traceback) -> bool:
    """Remove partial renditions so the retry starts from a clean dir."""
    out_dir = job.args[1]
    if os.path.isdir(out_dir):
        shutil.rmtree(out_dir, ignore_errors=True)
    return True  # let RQ proceed to its retry logic

if __name__ == "__main__":
    worker = Worker(
        [Queue("transcode", connection=conn)],
        connection=conn,
        exception_handlers=[cleanup_handler],
    )
    # First SIGTERM = warm shutdown: finish the current job, then exit.
    # A second SIGTERM force-kills. Give the process manager a long grace.
    worker.work(with_scheduler=True)

The cleanup_handler runs whenever the job raises, before the retry is scheduled, so retry attempt two never inherits a directory of truncated .mp4 files. The warm-shutdown behaviour is built into RQ's default worker: the first SIGTERM lets the current job finish rather than orphaning a 15-minute encode, and only a second signal force-kills. The practical consequence is that your process supervisor's stop-grace-period must be longer than your longest job, or systemd/Kubernetes will SIGKILL the worker out from under a nearly-complete transcode. We set ours to match job_timeout. with_scheduler=True is what actually executes the delayed retries — forget it and your Retry(interval=...) jobs quietly never run again.

Retries, timeouts, and the failure queue

Transcoding fails for boring, recurrent reasons, and the retry policy has to distinguish the ones worth retrying from the ones that never will be:

• Transient: a worker OOM-killed under load, a full disk that a cleanup cron then frees, a network blip fetching a remote source. These deserve retries with backoff. Our interval=[30, 120, 300] spaces the three attempts across ~7 minutes so a temporary resource crunch resolves before the last try.
• Permanent: a genuinely corrupt or truncated upload, an unsupported codec, a zero-byte file. Retrying these three times just burns CPU and delays the alert. The right move is to catch those specific ffmpeg exit signatures in the job and raise a non-retryable error so RQ sends them straight to the failed registry.
• The job_timeout is a hard ceiling enforced by the worker, independent of ffmpeg. Without it, a pathological input that makes ffmpeg spin forever would hold a worker slot indefinitely. Thirty minutes is generous for us; tune it to your longest legitimate clip plus headroom.

Everything that exhausts its retries lands in rq:failed:transcode with the full traceback and a 7-day failure_ttl. That window is deliberate: it is long enough to catch a Monday-morning pattern of failures but short enough that the registry doesn't grow unbounded. Requeuing a fixed batch is a one-liner with rq requeue --queue transcode --all once the root cause is patched.

Reporting progress back to the frontend

Because the job writes job.meta["progress"] after each rendition, the frontend gets a real number instead of an indeterminate spinner. The PHP side exposes a tiny status endpoint that reads the job hash straight from Redis — it never touches the worker. We also mirror the terminal state into our SQLite database (running in WAL mode, so the read that renders the clip's public page never blocks the write that marks it published). Redis holds the live progress; SQLite holds the durable record. Keeping those two roles separate means a Redis flush during maintenance loses in-flight progress bars but never loses the knowledge that a clip was successfully published. That division — ephemeral coordination state in Redis, durable truth in SQLite — is the single design decision I'd most strongly defend.

Watching queue depth in production

A queue you cannot see is a queue that will surprise you. Everything RQ needs is already legible in Redis, so the exporter is trivial — and I wrote ours in Go precisely because it can run as a single static binary with no Python runtime to keep in sync with the workers:

// queuedepth.go — export RQ queue metrics for Prometheus/alerting
package main

import (
    "context"
    "fmt"
    "net/http"

    "github.com/redis/go-redis/v9"
)

func main() {
    rdb := redis.NewClient(&redis.Options{Addr: "127.0.0.1:6379"})
    ctx := context.Background()

    http.HandleFunc("/metrics", func(w http.ResponseWriter, r *http.Request) {
        queued, _ := rdb.LLen(ctx, "rq:queue:transcode").Result()
        incoming, _ := rdb.LLen(ctx, "transcode:incoming").Result()
        failed, _ := rdb.ZCard(ctx, "rq:failed:transcode").Result()

        fmt.Fprintf(w, "transcode_queue_depth %d\n", queued)
        fmt.Fprintf(w, "transcode_incoming_backlog %d\n", incoming)
        fmt.Fprintf(w, "transcode_failed_total %d\n", failed)
    })

    _ = http.ListenAndServe(":9308", nil)
}

Three numbers tell you almost everything. transcode_incoming_backlog growing means the dispatcher is stuck or dead — messages are piling up on the PHP-facing list and never becoming RQ jobs. transcode_queue_depth growing while the backlog is flat means the workers can't keep up and you need more of them. And any movement in transcode_failed_total pages someone. We alert on the rate of change of depth, not the absolute value: a queue of 40 during a traffic spike is healthy; a queue of 40 that has only grown for 20 minutes is an incident. Behind Cloudflare Workers we also shed obviously-bad uploads at the edge before they ever reach ingest, which keeps the backlog honest.

GDPR: the transcode is also a scrub

One benefit of forcing every clip through a transcode is that the encode is a natural, unavoidable choke point where you can enforce privacy. The -map_metadata -1 flag in the job strips EXIF, GPS coordinates, device identifiers, and author tags that creators routinely — and unknowingly — embed in source files. For a European audience that is not a nice-to-have; publishing a creator's home GPS coordinates because you passed metadata straight through is a concrete data-protection failure. Because every published rendition is a fresh ffmpeg output rather than the original file, we can guarantee the metadata is gone, and we retain the raw source only as long as the job's retry window requires before a cleanup cron deletes it. The queue turned a legal requirement into a single flag in a command array.

Conclusion

The architecture that has held up for us is deliberately unglamorous: a plain Redis list as the language boundary between PHP and Python, a dispatcher that owns every RQ-specific decision in one auditable place, workers with a warm-shutdown grace longer than their longest job, idempotent job_ids, backoff retries that distinguish transient from permanent failure, and three Redis counters that tell you the health of the whole thing at a glance. None of it is exotic, and that is the feature. If you are about to shell out to ffmpeg inside an HTTP handler, stop — put a queue between the request and the encoder, keep ephemeral coordination in Redis and durable truth in your database, and make the queue depth something a machine watches so a human doesn't have to.