Building a Video Transcoding Job Queue with Python RQ and Redis

The Job That Blocked the Whole Request

The first version of our upload pipeline did something embarrassing: it transcoded video inside the web request. A user handed us a 400 MB MP4, PHP passed it to ffmpeg through shell_exec, and the LiteSpeed worker sat there for ninety seconds re-encoding to H.264 while the browser spinner turned. When two people uploaded at the same time, both LiteSpeed workers were pinned, and the rest of DailyWatch — a free video discovery platform — started serving slow responses to everyone else browsing the catalog.

Transcoding is CPU-bound, unpredictable in duration, and prone to failure: corrupt input, unsupported codecs, audio streams that don't decode, out-of-memory kills on a 4K source. None of that belongs in a synchronous HTTP request. It belongs in a queue with explicit timeouts, retries, and a pool of workers you can scale independently of your web tier. This post walks through the queue we actually run in production: Python RQ on top of Redis, wrapping ffmpeg, with the PHP front end (PHP 8.4, LiteSpeed, SQLite with FTS5, Cloudflare in front) responsible only for enqueueing work and reporting status back to the user.

The goal is a system where the upload request returns in under 200 ms, a background worker does the heavy lifting, failures retry with backoff, and a permanently broken job lands somewhere you can inspect it instead of vanishing.

Why RQ Instead of Celery

Celery is the default answer for Python task queues, and it is a fine tool, but for a transcoding pipeline it brings more machinery than the problem needs. RQ (Redis Queue) is deliberately small. A job is just a Python function plus arguments; Redis is the only broker and result backend; a worker is a single process that forks a child per job. That fork-per-job model matters here: ffmpeg can leak memory or get OOM-killed, and because RQ runs each job in a forked child, a catastrophic job death doesn't take down the parent worker. The worker reaps the child, marks the job failed, and moves on.

What we get from RQ that we care about:

• Per-job timeouts — a job that runs longer than its budget is killed, not left to spin forever.
• A dead-letter queue out of the box — failed jobs land on a FailedJobRegistry you can requeue or inspect.
• Retry with backoff — declarative, no custom loop.
• Trivial horizontal scaling — start more rq worker processes, even on other machines, all reading the same Redis.
• No pickle surprises across languages — which forces a clean boundary between PHP and Python, discussed below.

The tradeoff is that RQ is Redis-only and Python-only. Both are fine for us. Redis was already in the stack for rate limiting and cache, and the transcoding logic is Python because ffmpeg orchestration is nicer there than in PHP.

The Shape of a Transcoding Job

An RQ job is a plain function. The important discipline is that the function must be idempotent-ish and self-contained: it receives primitive arguments (a source path, an output directory, a video id), does its work, and returns a small JSON-serializable result. It should never receive an open file handle, a database connection, or anything that can't survive being pickled and executed in a forked child minutes later.

Here is the core job. It probes the source with ffprobe, transcodes to a 720p H.264/AAC MP4 with a faststart flag so the file streams before it fully downloads, and returns metadata the PHP side will store.

# tasks.py
import json
import os
import subprocess
import tempfile
from pathlib import Path

class TranscodeError(Exception):
    """Raised when ffmpeg fails in a way that is worth retrying."""

def _probe(src: str) -> dict:
    proc = subprocess.run(
        ["ffprobe", "-v", "error", "-print_format", "json",
         "-show_format", "-show_streams", src],
        capture_output=True, text=True, timeout=60,
    )
    if proc.returncode != 0:
        raise TranscodeError(f"ffprobe failed: {proc.stderr.strip()}")
    return json.loads(proc.stdout)

def transcode_video(video_id: int, src: str, out_dir: str) -> dict:
    if not os.path.isfile(src):
        # Missing input is not retryable. Fail fast, no exception that RQ retries.
        return {"video_id": video_id, "status": "error", "reason": "source_missing"}

    info = _probe(src)
    duration = float(info["format"].get("duration", 0.0))

    Path(out_dir).mkdir(parents=True, exist_ok=True)
    final_path = os.path.join(out_dir, f"{video_id}_720p.mp4")

    # Write to a temp file first, then atomically rename. A worker that dies
    # mid-encode must never leave a half-written file where the CDN can serve it.
    fd, tmp_path = tempfile.mkstemp(suffix=".mp4", dir=out_dir)
    os.close(fd)

    cmd = [
        "ffmpeg", "-y", "-i", src,
        "-c:v", "libx264", "-preset", "veryfast", "-crf", "23",
        "-vf", "scale=-2:720",
        "-c:a", "aac", "-b:a", "128k",
        "-movflags", "+faststart",
        "-f", "mp4", tmp_path,
    ]
    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=1800)
    if proc.returncode != 0:
        os.path.exists(tmp_path) and os.remove(tmp_path)
        # Raise so RQ's retry policy gets a chance. Transient FS / OOM issues recover.
        raise TranscodeError(proc.stderr.strip()[-500:])

    os.replace(tmp_path, final_path)
    return {
        "video_id": video_id,
        "status": "done",
        "duration": round(duration, 2),
        "output": final_path,
        "bytes": os.path.getsize(final_path),
    }

Two decisions in that code earn their keep. First, write-to-temp-then-rename: os.replace is atomic on the same filesystem, so a worker killed mid-encode never leaves a truncated MP4 that Cloudflare might cache and serve. Second, the difference between returning an error and raising one. A missing source file is permanent — retrying will never conjure the file — so we return a terminal error dict. An ffmpeg non-zero exit might be a transient disk-full or OOM condition, so we raise and let RQ's retry policy decide.

Enqueueing Across the PHP / Python Boundary

Here is the trap most people hit. RQ stores jobs in Redis by pickling the Python callable reference and its arguments. PHP cannot produce a valid RQ job payload — you would be reverse-engineering Python's pickle protocol, and it breaks the moment you upgrade RQ. So do not try to make PHP enqueue directly into the RQ queue.

The clean boundary is a plain Redis list as a request inbox. PHP pushes a small JSON message; a tiny Python bridge pops messages and enqueues real RQ jobs. PHP speaks only JSON and Redis lists, both of which are stable and language-neutral.

From PHP, using the phpredis extension:

<?php
// UploadController.php (PHP 8.4)
declare(strict_types=1);

final class TranscodeQueue
{
    public function __construct(private readonly \Redis $redis) {}

    public function enqueue(int $videoId, string $sourcePath): string
    {
        $jobId = bin2hex(random_bytes(8));

        $message = json_encode([
            'job_id'   => $jobId,
            'video_id' => $videoId,
            'src'      => $sourcePath,
            'out_dir'  => '/var/media/transcoded',
        ], JSON_THROW_ON_ERROR);

        // RPUSH onto the inbox the Python bridge is blocking on.
        $this->redis->rPush('transcode:requests', $message);

        // Track state in SQLite so the status endpoint has something to read
        // before the worker has even started.
        $stmt = $this->db->prepare(
            'INSERT INTO transcode_jobs (job_id, video_id, status, created_at)
             VALUES (:id, :vid, :status, :ts)'
        );
        $stmt->execute([
            ':id' => $jobId, ':vid' => $videoId,
            ':status' => 'queued', ':ts' => time(),
        ]);

        return $jobId;
    }
}

The upload request does three cheap things — push to a Redis list, insert a queued row into SQLite, and return the job_id — then responds. No ffmpeg, no blocking, no pinned LiteSpeed worker. Cloudflare sees a fast dynamic response it will not cache (the status is per-user), and the browser can poll a status endpoint that simply reads the SQLite row.

The Python bridge that turns those messages into real RQ jobs is a short loop using a blocking pop so it consumes no CPU while idle:

# bridge.py
import json
import redis
from rq import Queue, Retry
from tasks import transcode_video

r = redis.Redis(host="127.0.0.1", port=6379, db=0)
q = Queue("transcode", connection=r, default_timeout=1800)

def run():
    while True:
        # BLPOP blocks up to 5s, then loops — lets us handle SIGTERM cleanly.
        item = r.blpop("transcode:requests", timeout=5)
        if item is None:
            continue
        _, raw = item
        msg = json.loads(raw)
        q.enqueue(
            transcode_video,
            msg["video_id"], msg["src"], msg["out_dir"],
            job_id=msg["job_id"],            # reuse PHP's id so both sides agree
            retry=Retry(max=3, interval=[30, 120, 300]),
            result_ttl=86400,
            failure_ttl=604800,              # keep failures a week for inspection
        )

if __name__ == "__main__":
    run()

Reusing PHP's job_id as RQ's job_id is what ties the two systems together. PHP already wrote a queued row under that id; when the worker finishes, it looks up the same id and updates the row.

Timeouts, Retries, and the Dead-Letter Queue

The Retry(max=3, interval=[30, 120, 300]) above is the whole retry policy: on failure, RQ requeues the job after 30 seconds, then 120, then 300, before giving up. Explicit backoff intervals matter for transcoding because most transient failures are resource contention — disk filling, memory pressure from another job — and those clear on a timescale of minutes, not milliseconds. Hammering an immediate retry just fails again against the same full disk.

default_timeout=1800 caps each job at thirty minutes. If a pathological input makes ffmpeg hang, RQ's work-horse fork is killed and the job is marked failed rather than occupying a worker forever. That timeout is the single most important setting in the whole system; without it, one bad file can silently starve your worker pool.

When a job exhausts its retries it moves to the FailedJobRegistry — RQ's built-in dead-letter queue. Nothing is lost. You can list, inspect the traceback, and requeue:

# inspect_failures.py
from redis import Redis
from rq import Queue
from rq.registry import FailedJobRegistry

r = Redis()
q = Queue("transcode", connection=r)
registry = FailedJobRegistry(queue=q)

for job_id in registry.get_job_ids():
    job = q.fetch_job(job_id)
    print(job_id, job.args, "\n", (job.exc_info or "")[-400:], "\n---")
    # registry.requeue(job_id)  # uncomment to retry a fixed batch

In practice we scan this registry from a cron job and alert if it grows. A single failure is a corrupt upload; twenty failures in an hour means the encoder host lost its scratch disk.

Writing Results Back to SQLite

The worker needs to update the PHP-owned SQLite row when a job finishes. RQ supports success and failure callbacks, but a cleaner approach for cross-language state is to have the job itself write the result to a second Redis key that PHP already knows how to read, or update SQLite directly from Python. We update SQLite directly, because the status endpoint reads from there and we want one source of truth.

SQLite with WAL mode handles a single Python writer plus many PHP readers comfortably, which is exactly this access pattern. A success callback keeps the write out of the transcode function so the job stays pure:

# callbacks.py
import sqlite3

DB = "/var/media/app.db"

def on_success(job, connection, result, *args, **kwargs):
    db = sqlite3.connect(DB, timeout=10)
    db.execute("PRAGMA journal_mode=WAL")
    db.execute(
        "UPDATE transcode_jobs SET status=?, output=?, updated_at=strftime('%s','now') "
        "WHERE job_id=?",
        (result["status"], result.get("output", ""), job.id),
    )
    db.commit()
    db.close()

def on_failure(job, connection, typ, value, tb):
    db = sqlite3.connect(DB, timeout=10)
    db.execute("PRAGMA journal_mode=WAL")
    db.execute(
        "UPDATE transcode_jobs SET status='failed', updated_at=strftime('%s','now') "
        "WHERE job_id=?",
        (job.id,),
    )
    db.commit()
    db.close()

Wire them in at enqueue time with on_success=on_success, on_failure=on_failure. Now the flow is complete: PHP inserts queued, the worker flips it to done or failed, and the FTS5 index that powers on-site search only ingests a video once its row reads done — so search never surfaces a video that has no playable file yet.

Scaling Workers and Watching Queue Depth

Scaling is the payoff for doing this correctly. A worker is one command:

rq worker transcode --url redis://127.0.0.1:6379/0 --with-scheduler

Transcoding is CPU-bound, so we run roughly one worker per physical core, each pinned with taskset so two encodes don't fight over the same core. Under a burst we start more worker processes — on the same box or a separate encoding host that only needs network access to Redis and the shared media volume. No web-tier change, no code change. The queue absorbs the burst and workers drain it at whatever rate the hardware allows.

The number you must watch is queue depth: how many jobs are waiting. If depth grows faster than workers drain it, uploads still succeed instantly but users wait longer for playable video, and you need more workers. We export depth as a metric with a tiny Go sidecar, because Go gives us a static binary with no runtime to install on the encoding hosts:

// queuedepth.go
package main

import (
    "fmt"
    "net/http"
    "github.com/redis/go-redis/v9"
    "context"
)

func main() {
    rdb := redis.NewClient(&redis.Options{Addr: "127.0.0.1:6379"})
    ctx := context.Background()

    http.HandleFunc("/metrics", func(w http.ResponseWriter, r *http.Request) {
        // RQ stores queued jobs in a Redis list named rq:queue:<name>
        depth, _ := rdb.LLen(ctx, "rq:queue:transcode").Result()
        inbox, _ := rdb.LLen(ctx, "transcode:requests").Result()
        fmt.Fprintf(w, "transcode_queue_depth %d\n", depth)
        fmt.Fprintf(w, "transcode_inbox_depth %d\n", inbox)
    })
    http.ListenAndServe(":9101", nil)
}

Scraping /metrics gives us two numbers: the RQ queue depth and the bridge inbox depth. If the inbox grows but the RQ queue does not, the bridge is stuck; if the RQ queue grows, we need workers. Two integers tell you exactly where the backpressure is.

What This Bought Us

Moving transcoding out of the request path changed the failure mode of the whole site. Before, a single large upload degraded browsing for everyone because it held a LiteSpeed worker hostage. Now the upload path is a Redis push and a SQLite insert that returns in milliseconds, and the expensive work happens on a worker pool we scale on its own schedule.

The pieces worth copying, regardless of your stack:

• Never transcode in the request. Enqueue and return; poll for status.
• Keep the PHP/Python boundary language-neutral — a JSON message on a Redis list, not a pickled RQ payload.
• Set a per-job timeout. It is the setting that stops one bad file from starving the pool.
• Distinguish retryable from terminal failures — raise for the former, return an error for the latter.
• Write output atomically so a dead worker never leaves a half-file the CDN can cache.
• Watch queue depth, not worker CPU. Depth tells you when to scale before users feel it.

RQ will not win a feature checklist against Celery, but for wrapping ffmpeg behind a reliable queue it is exactly enough tool and no more — which, on a small team running a real site, is precisely the point.