Somewhere between "a handful of cron jobs" and "we need Airflow" is a large middle ground most small-to-mid automation setups actually live in: a few dozen jobs, some with dependencies on each other, that need to run reliably without the operational overhead of standing up and maintaining a full orchestration platform. This is a walkthrough of building that middle ground yourself, in an afternoon, with tools you probably already have.
The instinct to reach straight for a full-featured scheduler is understandable, these tools are mature and well documented, but they also come with real operational weight: a scheduler service to run and monitor, a web UI to secure, and a learning curve for anyone new to the team who needs to understand DAG definitions before they can safely touch a job. For a dozen jobs with straightforward dependencies, that weight often isn't worth carrying yet.
Step 1: Decide what you actually need orchestration for
Before writing any code, get specific about what's failing today with plain cron. Usually it's one of three things: jobs that need to run in a specific order (job B shouldn't start until job A finishes successfully), jobs that need retry logic beyond what cron gives you for free, or visibility into which jobs ran, when, and whether they succeeded. Not all three problems require the same fix, and conflating them is how teams end up reaching for a full scheduler to solve a visibility problem that a much smaller tool would have handled.
Step 2: Model dependencies as an explicit list, not implicit timing
The most common cron anti-pattern is encoding a dependency as a time offset: job A runs at 2:00am, job B runs at 2:15am because it "should" be done by then. This works until job A runs long one night and job B starts against incomplete data, silently.
Replace implicit timing with an explicit dependency list in a small config file or database table: each job declares which jobs must complete successfully before it starts. A runner script checks this list before triggering each job, rather than relying on wall-clock offsets to approximate an ordering guarantee that was never actually enforced.
JOBS = {
"extract_orders": {"depends_on": []},
"transform_orders": {"depends_on": ["extract_orders"]},
"load_warehouse": {"depends_on": ["transform_orders"]},
}
Step 3: Track run state in a table, not in memory
A minimal job_runs table (job name, started_at, finished_at, status, error) is the backbone of the whole system. Every run of every job writes a row here on start and updates it on completion or failure. This single table answers "did job A finish successfully today" for the dependency check in step 2, and it's also your monitoring surface, since a dashboard query against this table shows run history without any additional tooling.
PostgreSQL or any relational database you already run is sufficient for this. There's no need for a dedicated state store when the job volume is in the dozens rather than the thousands.
Step 4: Write a runner that checks dependencies before triggering
A single script, triggered on a short interval (every minute is common), checks each job's dependency list against the job_runs table. If all dependencies have a successful run for the current cycle and the job itself hasn't already run, trigger it. This replaces the manual "if it's 2:15, assume 2:00 finished" logic with an actual check against real completion state.
This runner is the entire orchestration layer. It's a few hundred lines of code, not a platform. For teams that eventually outgrow this approach, tools like Apache Airflow formalize exactly this pattern (dependency graphs, run state, retries) at much greater scale and complexity, which is worth knowing about even if it's not where you start.
Step 5: Add retries at the runner level, not inside each job
Rather than duplicating retry logic inside every individual job script, handle retries in the runner: if a job's status comes back as failed, the runner decides whether to retry based on a per-job retry count and backoff policy, then updates the job_runs row accordingly. Centralizing this in one place means changing the retry policy doesn't require touching every job script individually.
For jobs that use a message-passing pattern rather than direct invocation, Redis is a common, lightweight choice for the queue itself, avoiding the need for a heavier message broker when job volume doesn't justify one.
Photo by Brett Sayles on Pexels
Step 6: Build the smallest possible dashboard
A single page that queries the job_runs table and shows the last run status of every job, sorted by most recently failed, is usually enough. It doesn't need to be fancy. The goal is that anyone on the team can glance at it and answer "is everything running" without needing to know which log file to grep.
Handling the job that fails partway through a dependency chain
One case worth planning for explicitly: what happens when a job in the middle of a dependency chain fails. If transform_orders fails, the runner shouldn't trigger load_warehouse, since its dependency check will correctly see that transform_orders doesn't have a successful run for the current cycle. That part works automatically once the dependency check in step 4 is in place.
What needs explicit handling is what happens to extract_orders, the job upstream of the failure. It already ran successfully. Does it need to run again once transform_orders is fixed and retried, or can the orchestrator reuse the existing extracted data? This is a decision specific to each job, and it's worth documenting per job rather than assuming one answer covers the whole pipeline. Jobs that extract from a source that changes quickly (near-real-time data) usually need to re-run from the top. Jobs extracting from a stable source (a daily export that doesn't change after it lands) can often safely resume from the point of failure without re-running everything upstream.
A note on idempotency across the whole chain
None of this holds up if the individual jobs aren't safe to re-run. A transform_orders job that appends rows every time it runs, rather than upserting based on a stable key, will produce duplicate data the moment the orchestrator retries it after a failure. Every job in the chain needs to be safe to run twice with the same input, which usually means writes go through an upsert or ON CONFLICT DO NOTHING pattern rather than a plain append. This is worth verifying for every job before relying on the orchestrator's retry behavior, since a retry that silently duplicates data is often worse than the original failure it was trying to recover from.
When this stops being enough
This pattern holds up well for teams running somewhere under a hundred distinct jobs with straightforward dependency chains. Once you need dynamic DAGs generated at runtime, complex branching logic, or you're running enough jobs that a single runner script becomes a bottleneck, that's the point to seriously evaluate a dedicated orchestrator instead of continuing to extend the homegrown version. Building it yourself first, though, means you'll know exactly which of Airflow's features you actually need, instead of adopting the whole platform and discovering later that you use ten percent of it.
The failure handling side of this deserves its own attention beyond what fits in a job orchestrator. 137Foundry has a detailed guide on building a dead letter queue for jobs in a system like this one, covering what to capture when a job in the dependency chain fails permanently and how to review and safely reprocess it without breaking the jobs downstream of it.
Top comments (0)