Every video card on our category grids was hotlinking a 1280x720 JPEG from a third-party CDN and then letting CSS scale it down to about 320 device-independent pixels. That is roughly 90 KB of wasted transfer per card, 24 cards per page, across eight regional page variants that each carry their own cache key. Mobile LCP on the busiest category pages sat at 4.1s, and the largest single contributor was an image we did not host, could not resize, and could not re-encode to WebP.
The fix was not clever CSS. It was owning the frame. We built a small Go service that takes a source video (a partner preview MP4, or a poster frame that arrives at the wrong dimensions), pulls a representative frame with FFmpeg, encodes it at three widths in WebP, and writes the result to a content-addressed path the front end links directly. That service now feeds the same multi-region cron that runs TrendVidStream, and the generated files ride the same FTP mirror as the rest of the deploy.
What follows is the part that actually mattered: the FFmpeg invocations, the Go concurrency model that keeps a 2-core build box from melting, and how a stateless Go daemon hands work to a PHP 8.4 + SQLite front end that cannot run a daemon at all.
Why this is not a PHP job
Our front end is PHP 8.4 on LiteSpeed shared hosting with SQLite (FTS5 for search) as the only datastore. It is a genuinely good fit for a read-heavy discovery site: no database server to babysit, page cache on disk, cron jobs pulling regional feeds every 2-7 hours depending on the site.
It is a terrible fit for thumbnail extraction:
- Shared hosting caps
max_execution_timeat 180s. A cold FFmpeg decode of a 4-minute 1080p preview can burn 20-40s. Do 200 of them in one cron tick and you are wearing a hard timeout. -
shell_execis frequently disabled, and when it is not, you get one process per request with no way to bound total concurrency. - There is no shared memory between PHP requests, so two cron ticks racing on the same video ID will happily run FFmpeg twice and write the same file twice.
Go solves the three things PHP cannot here: a long-lived process that owns a bounded worker pool, context deadlines that reliably kill a wedged FFmpeg child, and in-process deduplication so N concurrent requests for one video produce exactly one decode. FFmpeg does the pixels; Go does the traffic control. There is no image processing in the Go code at all, which is deliberate — every attempt I have seen to do frame extraction with a pure-Go decoder ends in a codec matrix nobody wants to own.
The FFmpeg contract
Before any of the Go matters, the FFmpeg command has to be right, because the defaults are wrong for a service.
Four flags do the heavy lifting:
-
-ssbefore-i, not after. Placed before the input it becomes input seeking: FFmpeg jumps to the nearest keyframe and starts decoding there. Placed after, it decodes every frame from zero and throws them away. On a 4-minute file seeking to 00:00:30 that is the difference between ~300ms and ~8s. -
-nostdin. A daemon-spawned FFmpeg that inherits stdin will silently eat bytes from it and can block forever. This flag has saved me more debugging hours than any other. -
-frames:v 1. Stop after one frame. Without it a filter graph that produces frames keeps producing them. -
thumbnail=N. This is the filter that makes output not look like garbage. Frame zero on a streaming preview is almost always a black fade-in or a distributor logo.thumbnail=300buffers 300 frames, computes a histogram for each, and picks the one least like the batch average — i.e. the most visually distinctive frame. It costs one decode pass over those 300 frames and it is worth it every time.
Here is the extraction call, wrapped so the caller gets a deadline and a real error instead of a hang:
package thumb
import (
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"syscall"
"time"
)
// Variant is one output width. We ship 320/640/1280 and let srcset pick.
type Variant struct {
Width int
Quality int // libwebp -quality, 0-100
}
var Variants = []Variant{
{Width: 320, Quality: 78},
{Width: 640, Quality: 80},
{Width: 1280, Quality: 82},
}
// Extract writes one WebP per variant into dstDir, named <base>-<width>.webp.
// seek is where to start looking; the thumbnail filter picks the actual frame.
func Extract(ctx context.Context, src, dstDir, base string, seek time.Duration) ([]string, error) {
if err := os.MkdirAll(dstDir, 0o755); err != nil {
return nil, err
}
out := make([]string, 0, len(Variants))
for _, v := range Variants {
final := filepath.Join(dstDir, fmt.Sprintf("%s-%d.webp", base, v.Width))
tmp := final + ".tmp"
args := []string{
"-nostdin", "-y", "-loglevel", "error",
"-ss", strconv.FormatFloat(seek.Seconds(), 'f', 3, 64),
"-i", src,
"-an", "-sn", "-dn",
"-map_metadata", "-1",
"-vf", fmt.Sprintf(
"thumbnail=300,scale=%d:-2:flags=lanczos,format=yuv420p", v.Width),
"-frames:v", "1",
"-c:v", "libwebp",
"-quality", strconv.Itoa(v.Quality),
"-compression_level", "6",
"-preset", "picture",
"-threads", "1", // pool size controls parallelism, not FFmpeg
tmp + ".webp",
}
cmd := exec.CommandContext(ctx, "ffmpeg", args...)
// Ask nicely first, then hard-kill. Requires Go 1.20+.
cmd.Cancel = func() error { return cmd.Process.Signal(syscall.SIGTERM) }
cmd.WaitDelay = 5 * time.Second
var stderr strings.Builder
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
os.Remove(tmp + ".webp")
return nil, fmt.Errorf("ffmpeg %dpx: %w: %s", v.Width, err, strings.TrimSpace(stderr.String()))
}
// Rename last so a reader never sees a half-written file.
if err := os.Rename(tmp+".webp", final); err != nil {
return nil, err
}
out = append(out, final)
}
return out, nil
}
Three details in there are load-bearing and easy to miss:
-
scale=W:-2, not-1. The-2rounds the computed height to an even number. Odd dimensions break several encoders outright, and you will only find out on the one 1440x1079 source in your corpus. -
-threads 1. FFmpeg will happily spawn a thread per core. If you also run 8 workers you get 64 threads fighting over 2 cores and your p99 goes vertical. One thread per process, N processes, N = pool size. -
Write to
.tmp, thenos.Rename. Rename within a filesystem is atomic on Linux, so the web server either serves the complete old file or the complete new one. Without this you will eventually serve a truncated WebP and Chrome will render a grey box.
A bounded pool with request coalescing
The service takes HTTP requests, but it is really a queue with an HTTP face. Two constraints shape it: never run more FFmpeg processes than cores, and never run the same job twice.
The second one matters more than it sounds. Our regional cron jobs overlap — eight regions, staggered by minutes, and the same trending video shows up in five of them. Without coalescing, five simultaneous requests for dQw4w9WgXcQ mean five decodes of the same file, five writes to the same path, and a real chance of two renames interleaving. golang.org/x/sync/singleflight collapses them into one execution whose result is shared by all callers:
package main
import (
"context"
"encoding/json"
"log"
"net/http"
"runtime"
"time"
"golang.org/x/sync/singleflight"
"example.com/thumbd/thumb"
)
type Server struct {
sem chan struct{} // bounded FFmpeg concurrency
group singleflight.Group // one decode per key, no matter how many callers
dstDir string
}
func NewServer(dstDir string) *Server {
n := runtime.NumCPU()
if n > 4 {
n = 4 // leave headroom; the box also runs the article pipeline
}
return &Server{sem: make(chan struct{}, n), dstDir: dstDir}
}
type request struct {
ID string `json:"id"` // our internal video id, used as the file base
Src string `json:"src"` // path or URL FFmpeg can open
Seek int `json:"seek"` // seconds; 0 means let the filter decide
}
func (s *Server) handle(w http.ResponseWriter, r *http.Request) {
var req request
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 8<<10)).Decode(&req); err != nil {
http.Error(w, "bad json", http.StatusBadRequest)
return
}
if req.ID == "" || req.Src == "" {
http.Error(w, "id and src required", http.StatusBadRequest)
return
}
// Client gets 90s max; the pool wait is inside that budget.
ctx, cancel := context.WithTimeout(r.Context(), 90*time.Second)
defer cancel()
res, err, shared := s.group.Do(req.ID, func() (any, error) {
select {
case s.sem <- struct{}{}:
defer func() { <-s.sem }()
case <-ctx.Done():
return nil, ctx.Err()
}
return thumb.Extract(ctx, req.Src, s.dstDir, req.ID,
time.Duration(req.Seek)*time.Second)
})
if err != nil {
log.Printf("extract %s: %v", req.ID, err)
http.Error(w, "extract failed", http.StatusBadGateway)
return
}
if shared {
w.Header().Set("X-Coalesced", "1")
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(map[string]any{"id": req.ID, "files": res})
}
func main() {
s := NewServer("/srv/thumbs")
mux := http.NewServeMux()
mux.HandleFunc("POST /thumb", s.handle)
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) {
w.Write([]byte("ok"))
})
srv := &http.Server{
Addr: "127.0.0.1:8431",
Handler: mux,
ReadHeaderTimeout: 5 * time.Second,
WriteTimeout: 120 * time.Second, // must exceed the job budget
}
log.Fatal(srv.ListenAndServe())
}
A few things I got wrong the first time and would flag for anyone building this:
-
WriteTimeoutmust be longer than your job deadline. Mine was 30s against a 90s job. The handler completed successfully and the client got a truncated response, which looks exactly like an FFmpeg failure in the logs. Cost me an afternoon. -
Acquire the semaphore inside
singleflight.Do, not outside. Outside, every duplicate caller burns a pool slot waiting on a job it is not running. -
singleflightis per-process only. If you ever run two instances, you need the filesystem check (below) as the real guard. Coalescing is an optimization, not a correctness mechanism.
Wiring it to PHP and SQLite
The front end does not call the Go service during a page render — that would put an FFmpeg decode on the critical path of an HTTP request, which is the whole problem we were solving. Instead the multi-region cron enqueues, and the renderer reads whatever exists.
The schema is deliberately dumb. One row per video, a status column, and a nullable path. It sits alongside the FTS5 search table but is not part of it — you do not want thumbnail churn triggering FTS index writes.
<?php
declare(strict_types=1);
final class ThumbQueue
{
private const ENDPOINT = 'http://127.0.0.1:8431/thumb';
private const TIMEOUT = 95;
public function __construct(private readonly PDO $db) {}
public function migrate(): void
{
$this->db->exec(<<<SQL
CREATE TABLE IF NOT EXISTS thumbs (
video_id TEXT PRIMARY KEY,
src TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending',
width_max INTEGER,
updated_at INTEGER NOT NULL DEFAULT 0,
attempts INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX IF NOT EXISTS thumbs_status
ON thumbs(status, attempts, updated_at);
SQL);
}
/** Called by cron. Returns number of videos successfully rendered. */
public function drain(int $limit = 40): int
{
$rows = $this->db->prepare(
"SELECT video_id, src FROM thumbs
WHERE status IN ('pending','failed') AND attempts < 3
ORDER BY attempts ASC, updated_at ASC
LIMIT :lim"
);
$rows->bindValue(':lim', $limit, PDO::PARAM_INT);
$rows->execute();
$done = 0;
foreach ($rows->fetchAll(PDO::FETCH_ASSOC) as $row) {
$ok = $this->render($row['video_id'], $row['src']);
$this->db->prepare(
'UPDATE thumbs
SET status = :s, attempts = attempts + 1,
width_max = :w, updated_at = :t
WHERE video_id = :id'
)->execute([
':s' => $ok ? 'ready' : 'failed',
':w' => $ok ? 1280 : null,
':t' => time(),
':id' => $row['video_id'],
]);
$done += (int) $ok;
}
return $done;
}
private function render(string $id, string $src): bool
{
$ch = curl_init(self::ENDPOINT);
curl_setopt_array($ch, [
CURLOPT_POST => true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => self::TIMEOUT,
CURLOPT_HTTPHEADER => ['Content-Type: application/json'],
CURLOPT_POSTFIELDS => json_encode(
['id' => $id, 'src' => $src, 'seek' => 5],
JSON_THROW_ON_ERROR
),
]);
$body = curl_exec($ch);
$code = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);
return $body !== false && $code === 200;
}
}
On the render side the template asks one question: is status = 'ready'? If yes it emits a srcset over our own three widths; if no it falls back to the old remote URL. That fallback is the reason we could ship this incrementally instead of as a big-bang migration — the site worked identically at 0% coverage and at 100%, and we watched the ratio climb across four sites over about a week.
One SQLite note that bit us: with several cron jobs writing at once you want PRAGMA journal_mode=WAL and a busy_timeout of a few seconds, or the queue update will throw SQLITE_BUSY right after a successful render and the video gets re-decoded next tick.
Backfilling 40,000 rows without a thundering herd
The steady-state cron handles a few dozen new videos per tick. The initial backfill was a different animal, and PHP's 180s wall meant it had to run from the build box. A short Python script was the pragmatic answer — stdlib only, so it runs anywhere:
#!/usr/bin/env python3
"""Backfill thumbnails by walking the SQLite queue and posting to thumbd."""
import json
import sqlite3
import sys
import time
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor, as_completed
ENDPOINT = "http://127.0.0.1:8431/thumb"
WORKERS = 6 # > server pool: keeps the pool saturated, no more
DB = sys.argv[1] if len(sys.argv) > 1 else "data/site.db"
def post(video_id: str, src: str) -> tuple[str, bool]:
payload = json.dumps({"id": video_id, "src": src, "seek": 5}).encode()
req = urllib.request.Request(
ENDPOINT, data=payload, headers={"Content-Type": "application/json"}
)
for attempt in range(3):
try:
with urllib.request.urlopen(req, timeout=120) as resp:
return video_id, resp.status == 200
except (urllib.error.URLError, TimeoutError):
time.sleep(2 ** attempt) # 1s, 2s, 4s
return video_id, False
def main() -> int:
conn = sqlite3.connect(DB)
conn.execute("PRAGMA busy_timeout = 5000")
rows = conn.execute(
"SELECT video_id, src FROM thumbs WHERE status != 'ready' AND attempts < 3"
).fetchall()
print(f"{len(rows)} pending", flush=True)
ok = bad = 0
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
futures = {pool.submit(post, vid, src): vid for vid, src in rows}
for fut in as_completed(futures):
vid, success = fut.result()
conn.execute(
"UPDATE thumbs SET status = ?, attempts = attempts + 1,"
" updated_at = ? WHERE video_id = ?",
("ready" if success else "failed", int(time.time()), vid),
)
ok, bad = ok + success, bad + (not success)
if (ok + bad) % 250 == 0:
conn.commit()
print(f"ok={ok} failed={bad}", flush=True)
conn.commit()
print(f"done ok={ok} failed={bad}")
return 0 if bad == 0 else 1
if __name__ == "__main__":
raise SystemExit(main())
Six client threads against a four-slot server pool is intentional. You want the pool queue shallow but never empty; pushing client concurrency to 30 just moves the wait from the semaphore to the HTTP layer, where the timeout is less forgiving.
Shipping the output over FTP
Here is the constraint that shapes the deploy: our four sites run on shared LiteSpeed hosting with FTP access and no ability to run a persistent process. The Go service therefore never runs on the sites. It runs on the build box, and the WebP files it produces are just static assets that ride the same lftp mirror as PHP templates and CSS.
That turns out to be an advantage:
- The static binary is built once with
CGO_ENABLED=0 GOOS=linux go build -trimpath. No runtime, no extensions, no PHP version coupling. - Thumbnails are content-addressed by video ID and width, so
lftp mirror --only-newer --parallel=4uploads only what changed. A typical tick pushes a few hundred KB. - Regional sites share the generated files. Eight regions do not mean eight renders — the region only affects which videos get requested, never how they are encoded.
- If the generator is down, deploys still work. Nothing on the hosting side depends on it being alive.
The one gotcha worth repeating: make sure the host list file used by your deploy script has Unix line endings. A trailing \r gets appended to the remote path and lftp will cheerfully create a directory with a carriage return in its name, then report success. cat -A on the config file is a five-second check that has saved me twice.
What actually changed
Measured on the same set of category pages, before and after, mobile throttled:
- Thumbnail payload per grid page: ~2.1 MB → ~310 KB. Three widths in
srcsetmeans phones fetch the 320px variant, which averages 11 KB at quality 78. - Mobile LCP: 4.1s → 1.9s. Most of that is not the byte count — it is that the image now comes from the same origin with a long
Cache-Control, instead of a third-party host requiring a fresh DNS lookup and TLS handshake. - Generation throughput: about 26 videos/minute on 2 cores, three variants each. FFmpeg spends roughly 70% of that in the
thumbnail=300decode pass, which is the price of not shipping black frames. - Failure rate settled near 1.4%, almost entirely sources that 404 or have no video stream. Three attempts then permanent
failed, and the template falls back to the remote URL.
If you build one of these, the ordering that saved me time was: get the FFmpeg command right in a shell first, then wrap it, then add concurrency, then add the queue. Every bug I chased in the Go layer for the first two days turned out to be an FFmpeg flag I had not verified by hand.
Conclusion
The service is under 400 lines of Go and it is boring by design: FFmpeg does the decoding, a buffered channel does the throttling, singleflight does the deduplication, and os.Rename does the atomicity. The interesting engineering was not in the code — it was in accepting that the PHP front end should stay a pure reader, and that anything with an unbounded runtime belongs in a separate process with its own deadline.
The pattern generalizes past thumbnails. Any time you have a fast, cacheable read path and a slow, resource-hungry transform, splitting them at a queue boundary lets each side use the right tool. Ours happens to be PHP on one side and Go on the other, connected by a SQLite table and an FTP mirror, which is not an architecture anyone would draw on a whiteboard — but it has been running unattended for months across four sites and eight regions, and the grid pages load in under two seconds.
Top comments (0)