A user lands on a discovery page and sees forty video cards. If thirty-nine of them show a crisp preview frame and one shows a gray box, the gray box is the one they don't click. On DailyWatch we index videos from a lot of sources, and not all of them ship a usable thumbnail. Some give us a 120x90 JPEG that looks like it was faxed. Some give us nothing at all. So we built a small service whose only job is to take a video URL or a keyframe and produce clean, consistently-sized thumbnails on demand.
The old approach was a PHP cron that shelled out to FFmpeg one video at a time. It worked until it didn't: a single 4K source could pin a CPU core for eight seconds, the cron overran its window, and LiteSpeed workers piled up behind it. FFmpeg is the right tool for the extraction, but PHP is the wrong tool for orchestrating dozens of concurrent, CPU-bound, long-running child processes. Go is very good at exactly that. This post walks through the service we ended up with: a Go HTTP daemon that wraps FFmpeg, enforces concurrency limits, deduplicates work, and hands finished thumbnails back to the PHP 8.4 application that renders the pages.
Why FFmpeg extraction is trickier than it looks
People assume thumbnail generation is one command. It usually is one command, but the defaults will hurt you. A few things we learned the hard way:
- The first frame is often black. Fades-in, slates, and letterbox bars mean frame zero is frequently useless. You want a frame a few seconds in, or better, a frame FFmpeg thinks is interesting.
-
Seeking before decoding is dramatically faster. Putting
-ssbefore-iuses input seeking (keyframe-accurate jump) instead of decoding every frame up to the timestamp. On a 40-minute source that is the difference between 200ms and 20 seconds. - Aspect ratios are a minefield. Sources are 16:9, 9:16, 4:3, and occasionally something absurd. If you hard-scale to a fixed size you get squashed faces. You want scale-then-crop to a target box.
- FFmpeg will happily run forever. A malformed stream or a slow network source can hang the process. Every invocation needs a hard timeout and a killed process group.
The thumbnail filter is the underrated piece here. Instead of grabbing a fixed timestamp, it scores a window of frames and picks the most representative one, which avoids black frames and transition mush. Combined with input seeking to skip the intro, it gives good results without any per-video tuning.
Here is the core extraction command we settled on, expressed as the arguments we pass:
ffmpeg -ss 3 -i input.mp4 \
-vf "thumbnail=n=100,scale=640:360:force_original_aspect_ratio=increase,crop=640:360" \
-frames:v 1 -q:v 3 -f image2 -y output.jpg
That seeks 3 seconds in, scores the next 100 frames, scales so the shorter side fills 640x360, then center-crops the overflow. -frames:v 1 stops after one image, and -q:v 3 keeps the JPEG small without visible artifacts. This single command handles the vast majority of our sources.
The Go service structure
The service is a plain HTTP daemon. A request comes in with a source and a target size, the service extracts a frame, writes it to a cache directory, and returns the path (or the bytes). The important parts are the concurrency bound and the timeout, not the HTTP plumbing.
We use a buffered channel as a semaphore. FFmpeg is CPU-bound, so there is no point letting more jobs run than we have cores minus a couple for the OS and the web server. On our boxes that is a fixed number set from an env var.
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"syscall"
"time"
)
type Generator struct {
cacheDir string
sem chan struct{} // concurrency limiter
timeout time.Duration
}
func NewGenerator(cacheDir string, maxConcurrent int, timeout time.Duration) *Generator {
return &Generator{
cacheDir: cacheDir,
sem: make(chan struct{}, maxConcurrent),
timeout: timeout,
}
}
// cacheKey is deterministic: same source + size => same file on disk.
func cacheKey(source string, w, h int) string {
sum := sha256.Sum256([]byte(fmt.Sprintf("%s|%dx%d", source, w, h)))
return hex.EncodeToString(sum[:])[:32] + ".jpg"
}
func (g *Generator) Generate(ctx context.Context, source string, w, h int) (string, error) {
out := filepath.Join(g.cacheDir, cacheKey(source, w, h))
// Cache hit: nothing to do.
if _, err := os.Stat(out); err == nil {
return out, nil
}
// Acquire a slot or bail if the caller gave up.
select {
case g.sem <- struct{}{}:
defer func() { <-g.sem }()
case <-ctx.Done():
return "", ctx.Err()
}
return out, g.runFFmpeg(ctx, source, out, w, h)
}
The cacheKey matters more than it looks. Because it is a pure function of the source plus dimensions, two simultaneous requests for the same card resolve to the same output path. That is the foundation for deduplication, which I will come back to.
Running FFmpeg without leaking processes
The single most important piece of a shell-out service is cleanup. If you use exec.CommandContext and the context expires, Go sends SIGKILL to the direct child. But FFmpeg often is the direct child, so that works — until you wrap it in a shell, at which point killing the shell orphans FFmpeg and it keeps burning a core. We avoid shells entirely and put the child in its own process group so we can kill the whole group.
func (g *Generator) runFFmpeg(ctx context.Context, source, out string, w, h int) error {
ctx, cancel := context.WithTimeout(ctx, g.timeout)
defer cancel()
vf := fmt.Sprintf(
"thumbnail=n=100,scale=%d:%d:force_original_aspect_ratio=increase,crop=%d:%d",
w, h, w, h,
)
cmd := exec.CommandContext(ctx, "ffmpeg",
"-nostdin", "-loglevel", "error",
"-ss", "3", "-i", source,
"-vf", vf,
"-frames:v", "1", "-q:v", "3",
"-f", "image2", "-y",
out+".tmp",
)
// Own process group so a timeout kills ffmpeg and any children.
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error {
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
}
if err := cmd.Run(); err != nil {
_ = os.Remove(out + ".tmp")
if errors.Is(ctx.Err(), context.DeadlineExceeded) {
return fmt.Errorf("ffmpeg timed out after %s: %w", g.timeout, err)
}
return fmt.Errorf("ffmpeg failed: %w", err)
}
// Atomic publish: write to .tmp, then rename into place.
return os.Rename(out+".tmp", out)
}
Three details earn their keep here:
-
-nostdinstops FFmpeg from ever blocking on a read from a terminal it doesn't have. Without it, a stray keypress or a closed pipe can wedge the process. -
Writing to
out+".tmp"then renaming makes publishing atomic. A reader — the PHP app, or asendfilefrom LiteSpeed — never sees a half-written JPEG. On the same filesystemrename(2)is atomic, so the file either exists complete or does not exist. -
The process-group kill in
cmd.Cancelis what actually stops a runaway.Setpgidputs FFmpeg in its own group; killing the negated PID signals the entire group.
Deduplicating in-flight work
Cache-on-disk handles repeat requests over time, but it does nothing for a thundering herd. When our cron warms a category page, forty PHP workers can ask for the same forty thumbnails within a few milliseconds of each other. Without deduplication that is forty FFmpeg processes doing identical work, forty writes racing on the same .tmp file.
Go's singleflight package solves this in about six lines. It collapses concurrent calls with the same key into a single execution and fans the result back out to every caller.
import "golang.org/x/sync/singleflight"
type Generator struct {
cacheDir string
sem chan struct{}
timeout time.Duration
group singleflight.Group
}
func (g *Generator) GenerateDeduped(ctx context.Context, source string, w, h int) (string, error) {
key := cacheKey(source, w, h)
res, err, _ := g.group.Do(key, func() (interface{}, error) {
return g.Generate(ctx, source, w, h)
})
if err != nil {
return "", err
}
return res.(string), nil
}
Now forty simultaneous requests for the same card trigger exactly one FFmpeg run. The other thirty-nine block on the shared call and receive the same path when it finishes. Combined with the semaphore, the worst case for a cold category page is bounded: at most maxConcurrent FFmpeg processes, each producing a unique thumbnail, everything else deduplicated or queued.
One caveat worth stating plainly: singleflight shares the first caller's context. If that caller cancels, the shared call is canceled for everyone waiting on it. For our workload every caller wants the same result and the timeout is identical, so this is fine. If your callers have wildly different deadlines, detach the work into a background context and let each caller time out independently.
The HTTP surface and the PHP side
The HTTP handler is deliberately boring. It validates the requested size against an allowlist — you do not want an attacker asking for 20000x20000 and OOM-ing the box — and returns a small JSON payload the PHP app can act on.
func (g *Generator) handleThumb(w http.ResponseWriter, r *http.Request) {
source := r.URL.Query().Get("src")
size := r.URL.Query().Get("size") // e.g. "640x360"
dims, ok := allowedSizes[size] // map[string][2]int, fixed set
if source == "" || !ok {
http.Error(w, "bad request", http.StatusBadRequest)
return
}
path, err := g.GenerateDeduped(r.Context(), source, dims[0], dims[1])
if err != nil {
http.Error(w, "generation failed", http.StatusBadGateway)
return
}
w.Header().Set("Content-Type", "application/json")
fmt.Fprintf(w, `{"path":%q}`, filepath.Base(path))
}
The Go service writes into a directory that the web server can serve directly. It does not stream image bytes back through PHP — that would put a CPU-bound Go process in the request path of every page load, which is the mistake we were trying to escape. Instead PHP asks the service to ensure a thumbnail exists, then emits a URL that LiteSpeed serves as a static file, cached at the edge by Cloudflare.
Here is the PHP 8.4 client. Note the short connect timeout: if the thumbnail service is slow or down, we degrade to a placeholder rather than hanging the page render.
<?php
declare(strict_types=1);
final class ThumbnailClient
{
public function __construct(
private readonly string $serviceBase = 'http://127.0.0.1:8088',
private readonly string $placeholder = '/assets/thumb-placeholder.jpg',
) {}
public function urlFor(string $source, string $size = '640x360'): string
{
$query = http_build_query(['src' => $source, 'size' => $size]);
$ctx = stream_context_create([
'http' => [
'method' => 'GET',
'timeout' => 2.0, // fail fast, never block a page
'ignore_errors' => true,
],
]);
$raw = @file_get_contents("{$this->serviceBase}/thumb?{$query}", false, $ctx);
if ($raw === false) {
return $this->placeholder;
}
$data = json_decode($raw, true);
if (!is_array($data) || empty($data['path'])) {
return $this->placeholder;
}
// Served statically by LiteSpeed, cached at the edge by Cloudflare.
return '/thumbs/' . basename((string) $data['path']);
}
}
The two-second timeout and the placeholder fallback are non-negotiable. A page that renders with a gray box is a minor annoyance; a page that hangs because a downstream service is wedged is an outage. The PHP app should never be more reliable than its slowest synchronous dependency, so we make the dependency non-fatal.
Warming the cache instead of generating on request
Generating on the first request works, but it means the first visitor to any new video pays the FFmpeg latency. For a discovery platform where new videos arrive in batches from a cron, it is better to warm thumbnails at ingest time. We keep a lightweight Python worker that reads freshly-ingested video rows out of SQLite and fires them at the Go service ahead of any user, using a bounded thread pool so we never overwhelm the semaphore.
import sqlite3
import requests
from concurrent.futures import ThreadPoolExecutor
SERVICE = "http://127.0.0.1:8088/thumb"
DB = "/var/www/dailywatch/data/videos.db"
def warm(source: str) -> tuple[str, bool]:
try:
r = requests.get(SERVICE, params={"src": source, "size": "640x360"}, timeout=30)
return source, r.status_code == 200
except requests.RequestException:
return source, False
def main() -> None:
con = sqlite3.connect(DB)
con.row_factory = sqlite3.Row
rows = con.execute(
"SELECT source_url FROM videos WHERE thumb_ready = 0 LIMIT 500"
).fetchall()
con.close()
sources = [row["source_url"] for row in rows]
ok = 0
# Keep pool small; the Go semaphore is the real bound, this just feeds it.
with ThreadPoolExecutor(max_workers=8) as pool:
for source, success in pool.map(warm, sources):
if success:
ok += 1
print(f"warmed {ok}/{len(sources)} thumbnails")
if __name__ == "__main__":
main()
A couple of notes on this warmer:
- The thread pool is intentionally small. The real concurrency limit lives in the Go service's semaphore. The Python side just needs enough threads to keep the pipe full without opening 500 sockets at once.
-
thumb_readygets flipped separately. In production the Go service posts back completion, or a follow-up query checks the cache directory, and only then does the row get marked ready. That keeps the warmer idempotent — rerunning it never regenerates what already exists, because the Go cache check short-circuits anyway. -
Batching with
LIMIT 500keeps each cron run bounded. SQLite is single-writer, and we do not want a warm run holding a long transaction while the fetch cron is trying to insert new videos.
Operational lessons
A few things that only became obvious once this was live:
- Bound everything twice. The semaphore bounds concurrency inside the service. The Python pool bounds it from the producer side. The HTTP client bounds its wait. Redundant bounds are cheap and they save you when one layer misbehaves.
- Log the FFmpeg exit reason, not just failure. We tag timeouts distinctly from decode errors. Timeouts usually mean a bad source URL; decode errors usually mean a genuinely corrupt file. Conflating them cost us a week of chasing the wrong problem.
- Cache invalidation is a filename problem. Because the cache key includes the source and dimensions, we never invalidate — we just change the key. If a source URL changes, it is a new key and a new file. Stale files get swept by a nightly job that deletes anything not referenced by a live video row.
-
Keep FFmpeg pinned. Filter behavior —
thumbnail,crop,scale— is stable across versions but not identical. Pin the binary version in your deploy so a distro upgrade doesn't silently change how every thumbnail looks. - Static serving is the whole point. The Go service touches a file exactly once per unique thumbnail. Every subsequent hit is LiteSpeed reading a static JPEG and Cloudflare caching it. The expensive path runs once; the cheap path runs a million times.
Conclusion
The shape of this solution is more important than the specific code. FFmpeg does the extraction because nothing does it better. Go orchestrates the processes because it handles concurrent, CPU-bound, killable child processes cleanly — semaphores, context timeouts, process-group kills, and singleflight deduplication are all small, boring, and reliable. PHP stays out of the hot path entirely: it asks for a thumbnail, degrades gracefully if the answer is slow, and emits a static URL that the web server and CDN serve without ever touching application code again.
If you take one thing from this, make it the separation: keep the CPU-bound work in a dedicated service with hard bounds, and keep your request-serving tier able to fall back to a placeholder the moment that service misbehaves. A missing thumbnail is a shrug. A hung page render is a bad day. Build the boundary so the first can never become the second.
Top comments (0)