DEV Community

Cover image for File presence is all you need: a 7-stage video pipeline with no database and no queue service
Phillip
Phillip

Posted on

File presence is all you need: a 7-stage video pipeline with no database and no queue service

#typescript #architecture #automation #showdev

I run a video automation pipeline on a home Windows PC: idea file in, QA'd
1080p YouTube upload out. Seven stages — script generation (Ollama/Claude),
TTS narration (Piper), image sourcing, Remotion render, thumbnail, human QA,
YouTube Data API upload. It has produced 156 uploads and once ran 8+ days
unattended.

Here's the part worth writing about: it has no database, no message queue,
and no daemon. The entire orchestration layer is the filesystem.

One folder per job, file presence = state 

jobs/
  connect4-tactics/
    idea.md            <- input
    script.json        <- stage 1 done
    manifest.json      <- stage 2 done (+ audio/*.wav)
    images/            <- stage 3 done
    connect4-tactics.mp4   <- stages 4-5 done (+ .jpg thumbnail)
    qa-approved.txt    <- the human said yes
    youtube-id.txt     <- stage 7 done; folder moves to archive/uploaded/
Enter fullscreen mode Exit fullscreen mode

Every stage's contract is the same:

for (const job of listJobs()) {
  if (fs.existsSync(outputFileFor(job))) continue; // already done — skip
  await runStage(job);                              // do the work
  // the output file IS the state transition
}
Enter fullscreen mode Exit fullscreen mode

That's the whole scheduler. There is no status column to update, so there
is no status column to be wrong.

What this buys you

Crash safety for free. Power cut mid-render? The mp4 was never written,
so the next scheduled run re-renders that job and skips everything else.
Recovery isn't a code path — it's the only code path.

Redo = delete. Bad narration on one video? Delete manifest.json and the
WAVs; only stage 2 re-runs. Every partial redo you'd normally build tooling
for is a rm.

The dashboard is ls. Any file manager shows you exactly where every
video is in the pipeline. Debugging is opening a folder.

The QA gate is incorruptible by automation. Upload requires
qa-approved.txt, and only a human watching the video creates that file. The
gate isn't a boolean a bug can flip — it's a file only I write.

What it costs you

Honesty section: this design assumes one machine, one writer. Two
concurrent runs of the same stage would race (I serialize via Task Scheduler,
which never overlaps runs). File-presence also can't express rich state like
"attempted 3 times, failing" — my answer in the one flaky stage (network
image downloads) is to not track attempts at all: a failed download degrades
to a text-card visual and the pipeline keeps moving. If you need multi-node
workers or per-job retry metadata, use a real queue; below that threshold,
the filesystem is shockingly hard to beat.

The health check

The other half of unattended reliability: a check-health.ts that verifies
every credential, binary, model and token before a scheduled run touches
anything — expired OAuth tokens are found by the health check at 6am, not by
the upload stage at 2am.

I've packaged the whole pipeline (TypeScript source, setup + production
docs) as a one-time self-hosted kit: https://beforeyoustart.gumroad.com/l/channel-kit. No revenue promises —
it's infrastructure, not a business model. But if you've ever wanted to see
what "boring tech" looks like applied to video automation, the source is the
best argument I have.

Top comments (0)