File presence is all you need: a 7-stage video pipeline with no database and no queue service

Phillip — Fri, 17 Jul 2026 01:59:41 +0000

I run a video automation pipeline on a home Windows PC: idea file in, QA'd
1080p YouTube upload out. Seven stages — script generation (Ollama/Claude),
TTS narration (Piper), image sourcing, Remotion render, thumbnail, human QA,
YouTube Data API upload. It has produced 156 uploads and once ran 8+ days
unattended.

Here's the part worth writing about: it has no database, no message queue,
and no daemon. The entire orchestration layer is the filesystem.

One folder per job, file presence = state

jobs/
  connect4-tactics/
    idea.md            <- input
    script.json        <- stage 1 done
    manifest.json      <- stage 2 done (+ audio/*.wav)
    images/            <- stage 3 done
    connect4-tactics.mp4   <- stages 4-5 done (+ .jpg thumbnail)
    qa-approved.txt    <- the human said yes
    youtube-id.txt     <- stage 7 done; folder moves to archive/uploaded/

Every stage's contract is the same:

for (const job of listJobs()) {
  if (fs.existsSync(outputFileFor(job))) continue; // already done — skip
  await runStage(job);                              // do the work
  // the output file IS the state transition
}

That's the whole scheduler. There is no status column to update, so there
is no status column to be wrong.

What this buys you

Crash safety for free. Power cut mid-render? The mp4 was never written,
so the next scheduled run re-renders that job and skips everything else.
Recovery isn't a code path — it's the only code path.

Redo = delete. Bad narration on one video? Delete manifest.json and the
WAVs; only stage 2 re-runs. Every partial redo you'd normally build tooling
for is a rm.

The dashboard is ls. Any file manager shows you exactly where every
video is in the pipeline. Debugging is opening a folder.

The QA gate is incorruptible by automation. Upload requires
qa-approved.txt, and only a human watching the video creates that file. The
gate isn't a boolean a bug can flip — it's a file only I write.

What it costs you

Honesty section: this design assumes one machine, one writer. Two
concurrent runs of the same stage would race (I serialize via Task Scheduler,
which never overlaps runs). File-presence also can't express rich state like
"attempted 3 times, failing" — my answer in the one flaky stage (network
image downloads) is to not track attempts at all: a failed download degrades
to a text-card visual and the pipeline keeps moving. If you need multi-node
workers or per-job retry metadata, use a real queue; below that threshold,
the filesystem is shockingly hard to beat.

The health check

The other half of unattended reliability: a check-health.ts that verifies
every credential, binary, model and token before a scheduled run touches
anything — expired OAuth tokens are found by the health check at 6am, not by
the upload stage at 2am.

I've packaged the whole pipeline (TypeScript source, setup + production
docs) as a one-time self-hosted kit: https://beforeyoustart.gumroad.com/l/channel-kit. No revenue promises —
it's infrastructure, not a business model. But if you've ever wanted to see
what "boring tech" looks like applied to video automation, the source is the
best argument I have.

DEV Community: Phillip

File presence is all you need: a 7-stage video pipeline with no database and no queue service

One folder per job, file presence = state

What this buys you

What it costs you

The health check