Makefiles & Taskfiles for Reproducible Data Workflows

makefiles for data workflows are the difference between a pipeline anyone on the team can run with a single command and a pile of shell steps that only work on the laptop of the person who wrote them. Every data pipeline — extract a CSV, clean it, build features, train a model, publish a table — is really a small dependency graph, but most teams encode that graph as an implicit sequence in a stale README, a Slack thread, or someone's shell history. When the person leaves, the sequence leaves with them, and the next engineer spends a day reverse-engineering "which script runs first" and "why does step 3 fail unless I delete the temp folder." A task runner turns that implicit sequence into an explicit, declarative graph that is checked into git and invoked the same way everywhere.

This guide is the practical walkthrough of how a task runner makes a data workflow reproducible — one command, one dependency graph, and a build that skips the work it already did. It covers the two task runners a data engineer reaches for every day: GNU Make, whose target-and-prerequisite model has driven builds since 1976 and is still installed on every Unix box, and the modern YAML-based Taskfile, a single cross-platform Go binary that trades Make's tab-and-quoting minefield for a friendlier syntax and checksum-based fingerprints. Along the way it shows how file timestamps drive incremental rebuilds across a data DAG, why .PHONY targets exist, how automatic variables keep rules DRY, and the patterns that let the exact same file run on your laptop and in CI and produce a byte-identical result. Each section pairs a teaching block with a Solution-Tail interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the pandas practice library →, rehearse pipeline glue on the ETL practice library →, and harden your scripts on the defensive-coding practice library →.

---

On this page

• Why a task runner — one command, reproducible
• Make basics — targets, prerequisites, .PHONY, variables
• Incremental builds and dependency DAGs for data
• Taskfile (YAML) as a modern alternative
• Patterns for reproducible pipelines and CI
• Cheat sheet — Makefile and Taskfile recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why a task runner — one command, reproducible

One command, one graph — why a task runner is the reproducibility spine of every data pipeline

The one-sentence invariant: a task runner replaces an implicit, order-sensitive sequence of shell steps living in a README or someone's head with a declarative dependency graph, checked into git, that any engineer or CI job can invoke with a single command and get the same result — and the value it delivers is not "running commands" but making the order, the inputs, and the skip-what-is-done logic explicit and portable. The moment your pipeline has more than two steps and more than one person runs it, the cost of an undocumented sequence starts compounding: onboarding time, failed reruns, "works on my machine" incidents, and the slow erosion of trust in the numbers the pipeline produces. A task runner is the cheapest possible fix — a single text file that is simultaneously the runbook, the automation, and the dependency documentation.

The axes that matter — what a task runner actually buys you.

• One command. The whole pipeline collapses to make pipeline (or task pipeline). No memorising the order of six scripts; the graph knows the order. New engineers are productive in minutes, not days, and the on-call runbook is one line.
• Reproducibility. Given the same inputs, the same command produces the same outputs. The task runner does not guarantee reproducibility on its own, but it is the place you enforce it — pinned tools, deterministic seeds, hermetic inputs, and a checksum check all hang off the same file.
• Incrementality. Make and Taskfile both track what changed and rebuild only the stale parts. Re-running a pipeline after touching one CSV rebuilds one branch of the graph, not the whole thing — the difference between a 3-second no-op and a 40-minute full rerun.
• Dependency graph (DAG). Rules declare "this output depends on these inputs." The runner topologically sorts them, so you never manually sequence steps again and you get free parallelism (make -j) for independent branches.
• Portability. The same file runs on a laptop, a teammate's machine, and the CI runner. There is exactly one source of truth for "how the pipeline runs," and it is version-controlled next to the code.

The 2026 reality — task runners sit below orchestrators, not against them.

• GNU Make is still everywhere. It ships with every Linux distribution and macOS developer toolchain, needs zero install, and its file-target model maps perfectly onto data artifacts (a .parquet file is a build target). For local development, repo glue, and CI entrypoints, Make remains the default.
• Taskfile is the modern ergonomic alternative. A single Go binary, YAML syntax, no tab-versus-space traps, checksum-based sources/generates fingerprinting, and first-class cross-platform support (Windows included). Teams that find Make's syntax hostile reach for task.
• Orchestrators live one layer up. Airflow, Dagster, Prefect, and dbt handle scheduling, retries, backfills, distributed execution, and observability across many machines over time. A task runner handles local, single-machine, on-demand builds. The senior answer is "Make/Task for the developer inner loop and CI; an orchestrator for scheduled production runs" — they compose, they do not compete.
• The overlap is real but bounded. A small analytics pipeline that runs nightly on one box can be a Makefile plus a cron entry. The moment you need cross-machine execution, a scheduler UI, SLA alerting, or backfills over date partitions, you graduate to an orchestrator — and often still call make from inside the orchestrated task.

What interviewers listen for.

• Do you frame a pipeline as a dependency graph rather than a script? — senior signal.
• Do you say "given the same inputs, the same command gives the same output" when asked to define reproducibility? — required answer.
• Do you name incrementality ("rebuild only what changed") as the reason to use file targets instead of a shell script? — senior signal.
• Do you correctly place a task runner below an orchestrator ("Make for the inner loop, Airflow for the schedule") rather than treating them as substitutes? — senior signal.
• Do you call out idempotency and hermetic inputs as the things that make a rerun safe? — required answer.

Worked example — the README-rot pipeline, before and after

Detailed explanation. The most common data-pipeline anti-pattern is the "numbered scripts + README" setup: 01_extract.py, 02_clean.py, 03_features.py, 04_train.py, plus a README that says "run them in order, and delete tmp/ if step 3 complains." It works until it does not — someone runs step 4 without step 3, or forgets the tmp/ cleanup, or runs step 2 twice and doubles the rows. Converting it to a task runner makes the order and the inputs explicit and removes the human from the sequencing loop.

• The symptom. New hires cannot run the pipeline without pairing with a veteran; reruns are unreliable; "it worked yesterday" incidents recur.
• The root cause. The dependency graph is documented in prose, not in a machine-executable form. Prose drifts; code does not.
• The fix. One Makefile that declares each artifact as a target with its inputs as prerequisites, plus a single make pipeline entrypoint.

Question. Convert a four-step numbered-script pipeline into a single-command Makefile and explain what reproducibility guarantee you gained.

Input.

Before (README-driven)	After (task runner)
python 01_extract.py (manual)	make data/raw.csv
python 02_clean.py (manual)	make data/clean.parquet
python 03_features.py (delete tmp first)	make data/features.parquet
python 04_train.py (manual)	make model.pkl
"run in order" (prose)	make pipeline (graph)

Code.

# Makefile — the pipeline as a dependency graph
.PHONY: pipeline clean

pipeline: model.pkl            ## build everything up to the model

data/raw.csv:
    python 01_extract.py --out $@

data/clean.parquet: data/raw.csv
    python 02_clean.py --in $< --out $@

data/features.parquet: data/clean.parquet
    python 03_features.py --in $< --out $@

model.pkl: data/features.parquet
    python 04_train.py --in $< --out $@

clean:
    rm -f data/raw.csv data/clean.parquet data/features.parquet model.pkl

Step-by-step explanation.

1. Each artifact — data/raw.csv, data/clean.parquet, data/features.parquet, model.pkl — is a file target. The text before the colon is what the rule produces; the text after the colon is what it depends on. The graph is now machine-readable, not prose.
2. make pipeline asks for model.pkl. Make walks the prerequisite chain: model.pkl needs features, which needs clean, which needs raw. It builds them bottom-up in exactly the right order, every time, with no human sequencing.
3. The automatic variables $@ (the target being built) and $< (the first prerequisite) keep each recipe DRY and prevent the classic bug where a copy-pasted script writes to the wrong path.
4. The tmp/ cleanup problem disappears: because each rule declares its real inputs and outputs, there is no hidden state to remember. Re-running make pipeline after touching 02_clean.py's output rebuilds only the downstream half of the graph.
5. clean is a .PHONY target — it is not a file, it is a named action. We will unpack .PHONY in section 2; for now, note that the destructive "start fresh" action is now a documented, discoverable command instead of a README footnote.

Output.

Command	What happens
make pipeline (cold)	builds raw → clean → features → model in order
make pipeline (warm)	"Nothing to be done" — all targets up to date
touch data/clean.parquet; make pipeline	rebuilds features + model only
make clean	removes all generated artifacts

Rule of thumb. If your pipeline lives as "numbered scripts plus a README that says run them in order," you do not have a pipeline — you have a liability. Encode the order as a dependency graph in a Makefile or Taskfile; the file becomes the runbook, the automation, and the documentation at once.

Worked example — what "reproducible" actually means

Detailed explanation. Engineers throw the word "reproducible" around loosely. In a data-workflow interview, the precise definition matters: reproducibility is the property that the same command, on the same inputs, with the same pinned toolchain, produces the same outputs — regardless of who runs it or where. A task runner is where you encode the three levers (command, inputs, toolchain), but each lever has to be actually controlled or the guarantee is hollow.

• Same command. One entrypoint (make pipeline) that hides all flags and ordering. If two people run it two different ways, you have no reproducibility to reason about.
• Same inputs. Hermetic, versioned inputs — a pinned dataset snapshot or a checksum-verified download, not "whatever was in the bucket today."
• Same toolchain. Pinned Python/library versions (a lockfile), pinned CLI tools, and deterministic randomness (fixed seeds). Otherwise the code is reproducible but the environment is not.

Question. List the three levers a task runner must control to make a data pipeline reproducible, and give the failure mode when each is uncontrolled.

Input.

Lever	Controlled by	Failure mode if uncontrolled
Command	single make / task entrypoint	two people run it differently; results diverge
Inputs	pinned snapshot + checksum	"the data changed under us"; silent drift
Toolchain	lockfile + fixed seed	different library version reorders floats; non-determinism

Code.

# Reproducibility levers encoded in one Makefile
PYTHON := python3
SEED   := 42                       # deterministic randomness
DATA_SHA := data/raw.csv.sha256    # checksum of the pinned input

.PHONY: pipeline verify-input

verify-input:                      ## fail fast if the input drifted
    sha256sum -c $(DATA_SHA)

pipeline: verify-input model.pkl

model.pkl: data/features.parquet
    $(PYTHON) 04_train.py --in $< --out $@ --seed $(SEED)

Step-by-step explanation.

1. PYTHON, SEED, and DATA_SHA are variables at the top of the file — the reproducibility knobs are visible in one place instead of scattered across scripts.
2. verify-input runs sha256sum -c against a committed checksum file. If the pinned input has changed, the build fails immediately with a clear message rather than silently producing different numbers — this is the "same inputs" lever made enforceable.
3. pipeline lists verify-input as a prerequisite before model.pkl, so the checksum gate runs first. This is the "same command" lever: everyone gets the guard for free.
4. The --seed $(SEED) flag threads a fixed random seed into the training step. Without it, any model with stochastic initialisation produces a different model.pkl on every run, and "reproducible" is a lie — this is the "same toolchain / same randomness" lever.
5. Pinning the library versions is out of scope for the Makefile itself, but the Makefile is where you invoke the pinned environment (uv run, poetry run, a pinned Docker image). The runner is the single place all three levers are wired together.

Output.

Scenario	Result
pinned input intact, fixed seed	identical model.pkl every run
input file changed	sha256sum -c fails; build stops before training
seed removed	model.pkl differs run to run; not reproducible
different library version	subtle numeric drift; caught only by output checksum

Rule of thumb. "Reproducible" means same command × same inputs × same toolchain → same output. A task runner does not make a pipeline reproducible by itself; it is the single file where you wire in the checksum gate, the fixed seed, and the pinned environment so the guarantee is enforced rather than hoped for.

Worked example — task runner versus orchestrator, the decision

Detailed explanation. The most common senior-level confusion is treating Make/Taskfile and Airflow/Dagster as competitors. They operate at different layers. A task runner builds a dependency graph now, here, on one machine. An orchestrator schedules and monitors graphs over time, across machines, with retries, backfills, and a UI. The right architecture usually uses both — the orchestrator schedules a task that shells out to make.

• Task runner scope. Local dev inner loop, CI entrypoint, single-machine on-demand builds, incremental rebuilds by file state.
• Orchestrator scope. Cron-style scheduling, retries with backoff, backfills over date partitions, distributed execution, lineage/observability UI, SLA alerting.
• The hybrid. Airflow BashOperator → make transform; or a Dagster op that invokes task load. The orchestrator owns when and observability; the runner owns how and incrementality.

Question. Given three pipelines, decide whether each needs a task runner, an orchestrator, or both.

Input.

Pipeline	Runs	Machines	Needs schedule/backfill?
Analyst's local feature build	on demand	1 (laptop)	no
Nightly warehouse load, one box	1x/day	1	schedule yes, backfill rare
Multi-source hourly ingest + SLA	hourly	many	schedule + backfill + alerts

Code.

# Decision helper (illustrative)
def pick_automation(on_demand: bool,
                    scheduled: bool,
                    many_machines: bool,
                    needs_backfill_ui: bool) -> list[str]:
    """Return the automation layer(s) a pipeline needs."""
    layers = ["task-runner (make/taskfile)"]   # always the inner-loop build
    if scheduled or many_machines or needs_backfill_ui:
        layers.append("orchestrator (airflow/dagster) calling the runner")
    return layers


print(pick_automation(True,  False, False, False))
# -> ['task-runner (make/taskfile)']

print(pick_automation(False, True,  False, False))
# -> ['task-runner (make/taskfile)', 'orchestrator (airflow/dagster) calling the runner']

print(pick_automation(False, True,  True,  True))
# -> ['task-runner (make/taskfile)', 'orchestrator (airflow/dagster) calling the runner']

Step-by-step explanation.

1. Every pipeline gets a task runner. Even the multi-machine hourly job benefits from a Makefile that encapsulates "how to build one partition" — the orchestrator just calls it. The runner is the reusable, testable unit.
2. The analyst's laptop build needs only a task runner. Adding Airflow here is over-engineering — there is no schedule, no second machine, no backfill.
3. The nightly one-box load needs a schedule, so it graduates to an orchestrator (or a humble cron) that invokes make load. The orchestrator owns "run at 02:00 and alert me if it fails."
4. The multi-source hourly ingest needs scheduling and backfills and cross-machine execution and alerting — the full orchestrator feature set. But the per-partition transform logic still lives in a Makefile/Taskfile that the orchestrated task calls, keeping the build logic identical between local debugging and production.
5. The anti-patterns are symmetric: using Airflow for a laptop build (heavyweight, hard to run locally) and using a bare Makefile for a cross-machine SLA pipeline (no scheduling, no retries, no UI). Match the layer to the need.

Output.

Pipeline	Task runner	Orchestrator
Analyst local feature build	yes	no
Nightly one-box warehouse load	yes	yes (or cron)
Multi-source hourly + SLA	yes (per-partition build)	yes (schedule, backfill, alerts)

Rule of thumb. A task runner and an orchestrator are not competitors — they are different layers. Put the build logic (how to produce an artifact, incrementally) in a Makefile/Taskfile, and let the orchestrator own when it runs and what happens on failure. When in doubt, start with a task runner; add an orchestrator only when you need scheduling, backfills, or cross-machine execution.

Data engineering interview question on choosing the automation layer

A senior interviewer often opens with: "A team ships an analytics pipeline as five numbered Python scripts plus a README. New hires can't run it, reruns are flaky, and 'it worked yesterday' incidents are common. There's no schedule yet — it runs on demand on one box. Walk me through what you'd introduce, why a task runner before an orchestrator, and how you'd make a single make pipeline command reproducible."

Solution Using a single Makefile entrypoint with a checksum gate and pinned environment

# Makefile — one reproducible entrypoint for the whole pipeline
PYTHON   := uv run python          # pinned interpreter + locked deps
SEED     := 42
RAW_SHA  := data/raw.csv.sha256

.PHONY: pipeline verify clean help

help:                              ## list available targets
    @grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | \
        awk 'BEGIN{FS=":.*?## "}{printf "  %-16s %s\n", $$1, $$2}'

verify:                            ## fail fast if the pinned input drifted
    sha256sum -c $(RAW_SHA)

data/clean.parquet: data/raw.csv
    $(PYTHON) clean.py --in $< --out $@

data/features.parquet: data/clean.parquet
    $(PYTHON) features.py --in $< --out $@

model.pkl: data/features.parquet
    $(PYTHON) train.py --in $< --out $@ --seed $(SEED)

pipeline: verify model.pkl         ## verify inputs, then build end to end
    @echo "pipeline OK -> model.pkl"

clean:                             ## remove all generated artifacts
    rm -f data/clean.parquet data/features.parquet model.pkl

Step-by-step trace.

Step	Before (5 scripts + README)	After (make pipeline)
Onboarding	pair with a veteran to learn the order	run one command; graph knows the order
Rerun after editing clean.py	rerun everything manually	rebuilds clean + features + model only
Input drift	silent; wrong numbers ship	verify fails before any compute
Randomness	different model each run	fixed --seed 42; identical model
Discoverability	order buried in prose	make help lists every target

Running make pipeline first runs verify (the sha256sum -c gate), then walks the prerequisite chain model.pkl → features → clean → raw, building each stale artifact bottom-up with the pinned uv run python interpreter. A warm rerun prints "Nothing to be done" for up-to-date targets; a rerun after editing clean.py's output rebuilds only the downstream half. The five-script README is deleted; the Makefile is the runbook.

Output:

Metric	Before	After
Commands to run pipeline	5 (in the right order)	1
Onboarding time	hours (pairing)	minutes
Rerun after 1 change	full rerun	incremental (1 branch)
Input-drift detection	none	checksum gate, fail-fast
Reproducibility	hope	enforced (pin + seed + checksum)

Why this works — concept by concept:

• File targets as the graph — declaring each artifact as a target with its inputs as prerequisites turns the pipeline into a machine-executable DAG. Make topologically sorts it, so ordering is automatic and correct on every run.
• Single entrypoint — make pipeline hides all flags, order, and gates behind one command. Everyone runs it identically, which is the precondition for reasoning about reproducibility at all.
• Checksum verify gate — listing verify as a prerequisite makes the input-drift check run before any compute. Uncontrolled inputs are the silent killer of reproducibility; the gate makes drift a loud, fail-fast error.
• Pinned interpreter plus fixed seed — uv run python supplies a locked dependency set and --seed 42 removes stochastic nondeterminism, so the same command produces the same model.pkl regardless of who runs it.
• Cost — one text file, zero new infrastructure. Build time drops from O(whole pipeline) to O(changed branch) on warm reruns, and onboarding drops from hours to minutes. The only ongoing cost is keeping the checksum and lockfile current — cheap relative to the "works on my machine" incidents it removes.

SQL
Topic — pandas
Pandas pipeline and transformation problems

Practice →

ETL Topic — etl ETL problems on pipeline orchestration

Practice →

---

2. Make basics — targets, prerequisites, .PHONY, variables

Targets, prerequisites, recipes, and .PHONY — the four primitives that make GNU Make a data-workflow engine

The mental model in one line: a Makefile is a list of rules, and every rule is the same shape — a target (usually a file to produce), a colon, its prerequisites (the inputs it depends on), and a TAB-indented recipe (the shell commands that build it) — and Make's entire job is to look at file timestamps, decide which targets are out of date relative to their prerequisites, and run only those recipes in dependency order. Master those four primitives — target, prerequisite, recipe, and the .PHONY escape hatch for targets that are not files — and you can express any data pipeline as a Makefile.

The rule anatomy — target, prerequisites, recipe.

• Target. The thing the rule produces — almost always a file path (data/clean.parquet). Make checks whether this file exists and how old it is. If the target is newer than every prerequisite, Make considers it up to date and skips the recipe.
• Prerequisites. The inputs listed after the colon (data/clean.parquet: data/raw.csv clean.py). If any prerequisite is newer than the target — or the target does not exist — the recipe runs. Listing the script as a prerequisite means editing the script also triggers a rebuild.
• Recipe. The shell commands that build the target, each line indented with a literal TAB (not spaces). Make hands each recipe line to /bin/sh. The recipe should write the file named by the target.
• The default goal. Running bare make builds the first target in the file. Convention is to make the first target all (or pipeline) so make with no arguments does the sensible thing.

The TAB rule — the single most infamous Make gotcha.

• Recipes must start with a TAB, not spaces. This is a hard requirement inherited from 1976. A recipe indented with spaces produces the error *** missing separator. Stop.
• Why it bites. Editors that convert tabs to spaces silently break Makefiles. Configure your editor to keep literal tabs in Makefile, or use .RECIPEPREFIX to change the recipe character (rarely done).
• How to spot it. cat -A Makefile shows ^I for tabs; a recipe line that begins with spaces instead of ^I is the bug.

.PHONY targets — names that are not files.

• The problem they solve. Targets like clean, all, test, and lint are actions, not files. If a file named clean ever exists in the directory, Make sees the clean target as "up to date" and refuses to run it.
• The fix. Declare .PHONY: clean all test lint. This tells Make "these targets are always out of date; run their recipe every time regardless of any same-named file."
• The other benefit. .PHONY also documents intent and gives a small speedup (Make skips the file-existence check). Every action target should be phony.
• Common phony targets. all (build everything), clean (remove artifacts), test, lint, install, help, and pipeline entrypoints like pipeline or deploy.

Variables and automatic variables — keeping rules DRY.

• Recursive vs simple. VAR = value is recursively expanded (evaluated each use — can be surprising). VAR := value is simply expanded (evaluated once, at definition — usually what you want). Prefer := for predictability.
• Conditional assignment. VAR ?= value sets VAR only if it is not already set — perfect for defaults that CI or the environment can override (PYTHON ?= python3).
• Automatic variables. Inside a recipe, Make provides $@ (the target), $< (the first prerequisite), $^ (all prerequisites, deduplicated), and $* (the stem of a pattern rule). These let one rule serve many files without hard-coding paths.
• Escaping $. Because $ introduces a Make variable, a literal shell $ in a recipe must be doubled: $$HOME, $$(date). Forgetting this is a frequent bug when embedding shell logic.

Common beginner mistakes.

• Indenting a recipe with spaces instead of a TAB (missing separator).
• Forgetting .PHONY on clean/all, so a stray same-named file silently disables the target.
• Using = where := is meant, causing late/repeated expansion surprises.
• Writing a recipe that does not create the file named by the target, so Make reruns it every time (the target never appears "up to date").
• Forgetting to double $ for shell variables, so $HOME expands to an empty Make variable.

Worked example — anatomy of a single rule

Detailed explanation. Start with the smallest possible real rule: build a cleaned Parquet from a raw CSV using a Python script. This one rule exercises all three primitives — target, prerequisites, recipe — plus automatic variables and the "list the script as a prerequisite" habit that makes rebuilds correct.

• Target. data/clean.parquet — the file we produce.
• Prerequisites. data/raw.csv (the data) and clean.py (the code). Listing both means a code change triggers a rebuild, not just a data change.
• Recipe. One TAB-indented line invoking the script with $< (first prereq = the CSV) and $@ (the target = the Parquet).

Question. Write a single Make rule that builds data/clean.parquet from data/raw.csv and clean.py, and explain when the recipe runs.

Input.

Element	Value
Target	data/clean.parquet
Prerequisites	data/raw.csv, clean.py
Recipe	python clean.py --in <csv> --out <parquet>
Automatic vars	$< = raw.csv, $@ = clean.parquet

Code.

# One rule, all four primitives
data/clean.parquet: data/raw.csv clean.py
    python clean.py --in $< --out $@

Step-by-step explanation.

1. The text before the colon, data/clean.parquet, is the target — the file this rule knows how to build. Make will check its modification time to decide whether to run the recipe.
2. After the colon come two prerequisites: data/raw.csv (the input data) and clean.py (the transform code). Make rebuilds the target if either is newer than data/clean.parquet.
3. The second line — indented with a literal TAB — is the recipe. $< expands to the first prerequisite (data/raw.csv); $@ expands to the target (data/clean.parquet). Using automatic variables means renaming the target in one place updates the recipe automatically.
4. The recipe writes the file named by the target. This is essential: if clean.py wrote to a different path, Make would never see data/clean.parquet get created, would think the target failed, and would rerun the recipe on every invocation.
5. Listing clean.py as a prerequisite is the habit that separates correct Makefiles from subtly broken ones. Without it, editing the cleaning logic does not trigger a rebuild, and stale outputs ship silently.

Output.

Situation	Recipe runs?
data/clean.parquet missing	yes (build it)
data/raw.csv newer than target	yes (data changed)
clean.py newer than target	yes (code changed)
target newer than both prereqs	no (up to date)

Rule of thumb. Every rule is target: prerequisites then a TAB-indented recipe that writes the target. Always list the script as a prerequisite alongside the data, and always use $@/$< so the recipe stays in sync with the target — these two habits eliminate the most common "stale output" bugs.

Worked example — why clean needs .PHONY

Detailed explanation. The .PHONY declaration is the primitive beginners skip and then get burned by. Consider a clean target that removes generated artifacts. It works — until someone creates a file or directory literally named clean (a clean/ output folder, say). Now Make sees the target clean as "already exists and is up to date" and silently refuses to run the deletion. Declaring .PHONY: clean fixes it permanently.

• Without .PHONY. make clean may print "clean is up to date" and do nothing if a clean file exists.
• With .PHONY. make clean always runs its recipe, treating clean as an action, not a file.
• Scope. Every non-file target — all, clean, test, lint, help, pipeline — belongs in a .PHONY list.

Question. Show the failure mode of a non-phony clean target and the one-line fix.

Input.

Component	Before	After
Target clean	not phony	.PHONY: clean
A file named clean exists	clean "up to date"; no-op	always runs
Behaviour	fragile, silent no-op	correct, always deletes

Code.

# BROKEN: if a file/dir named "clean" exists, this silently no-ops
clean:
    rm -f data/*.parquet model.pkl

# FIXED: declare it phony so it always runs
.PHONY: all clean test

all: model.pkl

clean:
    rm -f data/*.parquet model.pkl

test:
    pytest -q

Step-by-step explanation.

1. In the broken version, clean is an ordinary target. Make's rule is "if a file named clean exists and is newer than its (nonexistent) prerequisites, it is up to date." So a stray clean file or directory turns the target into a no-op.
2. This failure is silent — make clean prints "make: 'clean' is up to date." and exits 0. The engineer believes artifacts were removed; they were not. The next build reuses stale outputs.
3. .PHONY: all clean test tells Make these three targets are not files. Their recipes run every time, unconditionally, regardless of any same-named file on disk.
4. .PHONY also skips the file-existence stat for those targets — a micro-optimisation, but the real value is correctness.
5. The rule of practice: the moment you write a target whose name is a verb (an action) rather than a noun (a file you produce), add it to the .PHONY list in the same edit.

Output.

Target	Non-phony behaviour	Phony behaviour
clean (no clean file)	runs	runs
clean (a clean file exists)	no-op (silent bug)	runs
all	may skip if all file exists	always evaluates
test	may skip if test file exists	always runs

Rule of thumb. Any target that is an action rather than a file must be declared .PHONY. Keep a single .PHONY: line near the top of the Makefile and add each action target to it as you write it — this removes an entire class of silent no-op bugs.

Worked example — variables and automatic variables

Detailed explanation. Variables turn a repetitive Makefile into a maintainable one. There are two axes: user variables (config like PYTHON, DATA_DIR, SEED) and automatic variables ($@, $<, $^ that Make fills in per rule). Getting the assignment operator right (:= vs = vs ?=) and using automatic variables consistently is what keeps a large Makefile DRY.

• := — simple assignment, evaluated once. Predictable; the default choice.
• ?= — set only if unset; lets the environment or CI override (PYTHON ?= python3).
• $@ $< $^ — target, first prerequisite, all prerequisites — used inside recipes to avoid hard-coding paths.

Question. Rewrite three near-identical rules using variables and automatic variables so paths and tools are declared once.

Input.

Symbol	Meaning	Example expansion
$@	the target	data/clean.parquet
$<	first prerequisite	data/raw.csv
$^	all prerequisites	data/raw.csv clean.py
:=	simple (once) assignment	PYTHON := python3
?=	default-if-unset	SEED ?= 42

Code.

# Config declared once at the top
PYTHON   := python3
DATA_DIR := data
SEED     ?= 42                     # CI can override: make SEED=7 pipeline

$(DATA_DIR)/clean.parquet: $(DATA_DIR)/raw.csv clean.py
    $(PYTHON) clean.py --in $< --out $@

$(DATA_DIR)/features.parquet: $(DATA_DIR)/clean.parquet features.py
    $(PYTHON) features.py --in $< --out $@

model.pkl: $(DATA_DIR)/features.parquet train.py
    $(PYTHON) train.py --in $< --out $@ --seed $(SEED)

Step-by-step explanation.

1. PYTHON := python3 and DATA_DIR := data are simple-assignment variables, evaluated once. Referencing them as $(PYTHON) and $(DATA_DIR) means switching interpreters or moving the data directory is a one-line change.
2. SEED ?= 42 uses conditional assignment: it defaults to 42 but yields to an override. Running make SEED=7 pipeline or exporting SEED in CI changes the seed without editing the file — useful for experiments while keeping the default reproducible.
3. In each recipe, $< is the first prerequisite (the input the script reads) and $@ is the target (the output the script writes). Because these are filled in per rule, the three recipes are structurally identical and none hard-code a path twice.
4. Listing the script (clean.py, features.py, train.py) as a second prerequisite means a code edit rebuilds that stage. $< deliberately grabs only the first prerequisite (the data), so the script path never leaks into the --in argument; if you needed all prerequisites you would use $^.
5. The payoff scales: a 40-rule Makefile with $(PYTHON), $(DATA_DIR), and automatic variables has one place to change the interpreter and zero duplicated paths, versus a shell script where every path is copy-pasted and drifts.

Output.

Change	Edits required
Switch to python3.12	1 line (PYTHON :=)
Move data to warehouse/	1 line (DATA_DIR :=)
Experiment with seed 7	0 edits (make SEED=7 pipeline)
Add a new stage	1 rule, reusing the same variables

Rule of thumb. Declare tools and directories once with :=, expose overridable knobs with ?=, and use $@/$</$^ in every recipe so no path is written twice. A Makefile that hard-codes paths in each rule is a shell script wearing a Makefile costume.

Data engineering interview question on Make fundamentals

A senior interviewer might ask: "Here's a Makefile a junior wrote. make builds nothing on a fresh checkout, make clean sometimes does nothing, and editing the transform script doesn't trigger a rebuild. Rewrite it correctly — cover the default goal, .PHONY, listing scripts as prerequisites, and automatic variables — and explain each fix."

Solution Using a corrected default goal, .PHONY, script prerequisites, and pattern rules

# Corrected Makefile
PYTHON   := python3
DATA_DIR := data

.PHONY: all clean            # <-- actions, not files

all: model.pkl               # <-- default goal: bare `make` builds the model

# Pattern rule: any data/%.parquet is built from data/%.csv + clean.py
$(DATA_DIR)/%.parquet: $(DATA_DIR)/%.csv clean.py
    $(PYTHON) clean.py --in $< --out $@

model.pkl: $(DATA_DIR)/features.parquet train.py
    $(PYTHON) train.py --in $< --out $@

clean:
    rm -f $(DATA_DIR)/*.parquet model.pkl

Step-by-step trace.

Symptom (before)	Cause	Fix (after)
make builds nothing	no default goal / first target was phony	all: model.pkl is the first real target
make clean sometimes no-ops	clean not phony	.PHONY: all clean
editing transform doesn't rebuild	script not a prerequisite	clean.py listed as prerequisite
paths duplicated per rule	no variables / automatic vars	$(DATA_DIR), $<, $@, % stem

Walking the fix on a fresh checkout: bare make targets all, which requires model.pkl, which requires data/features.parquet, which the pattern rule builds from data/features.csv + clean.py. Editing clean.py now bumps its mtime above every .parquet, so the pattern rule reruns for each affected file. make clean always fires because clean is phony. No path is written twice.

Output:

Command	Result after fix
make (fresh)	builds features.parquet then model.pkl
make (warm)	"Nothing to be done for 'all'"
edit clean.py; make	rebuilds affected .parquet + model
make clean	always removes artifacts

Why this works — concept by concept:

• Default goal all — placing all: model.pkl as the first non-phony target means bare make does the useful thing. The first target in the file is Make's default goal; making it explicit avoids "make builds nothing" surprises.
• .PHONY on actions — declaring all and clean phony guarantees their recipes run regardless of same-named files, killing the silent-no-op class of bugs.
• Script as a prerequisite — listing clean.py / train.py next to the data means a code change bumps the target out of date, so logic edits rebuild outputs instead of silently reusing stale ones.
• Pattern rule with % and $</$@ — one data/%.parquet: data/%.csv rule serves every partition; $* is the stem, $< the matched CSV, $@ the Parquet. This replaces N near-identical rules with one.
• Cost — the Makefile stays O(rules), not O(files): adding a new partition needs zero new rules because the pattern rule covers it. The only discipline required is keeping the .PHONY list current and always listing the script prerequisite.

SQL
Topic — pandas
Pandas and transformation-script problems

Practice →

Defensive Topic — defensive-coding Defensive-coding problems on scripts and edge cases

Practice →

---

3. Incremental builds and dependency DAGs for data

File timestamps plus the dependency DAG — how Make skips work that is already done

The mental model in one line: Make's incrementality is timestamp-driven — a target is rebuilt only when it is missing or older than at least one of its prerequisites, and because each target's prerequisites are themselves targets, the whole pipeline forms a dependency DAG that Make walks bottom-up, rebuilding exactly the stale sub-graph and skipping everything already up to date. This is the single most valuable property a task runner gives a data workflow: the difference between "re-run the whole 40-minute pipeline because I touched one file" and "rebuild the two downstream artifacts in 8 seconds."

How Make decides what to rebuild — the timestamp rule.

• The core comparison. For each target, Make compares its file modification time (mtime) against the mtime of every prerequisite. If the target is missing, or any prerequisite is newer, the recipe runs; otherwise it is skipped.
• Bottom-up evaluation. Make first resolves prerequisites (which may themselves need building), then evaluates the target. A change deep in the graph propagates upward: rebuild clean, and features and model become stale in turn.
• Timestamps, not content. Vanilla Make looks at mtime, not file contents. touch-ing a file marks it "changed" even if bytes are identical; conversely, restoring an old file with an old mtime can hide a change. Content-based fingerprinting is where Taskfile (section 4) improves on Make.
• make -n. The dry-run flag prints the recipes Make would run without running them — the fastest way to see which part of the DAG is stale.

Modelling a data pipeline as a DAG.

• Nodes are files. raw.csv → clean.parquet → features.parquet → model.pkl. Each arrow is a "prerequisite" relationship declared in a rule.
• Fan-out and fan-in. One raw file can feed several cleaners (fan-out); several feature files can feed one training step listed as multiple prerequisites (fan-in, consumed with $^).
• Independent branches build in parallel. Because the DAG encodes which targets are independent, make -j can build unrelated branches concurrently for free (covered in section 5).
• The graph is the documentation. make -Bnd or third-party tools can render the DAG; the Makefile is an executable lineage diagram.

Pattern rules — one rule for many files.

• The % wildcard. data/%.parquet: data/%.csv means "any Parquet under data/ is built from the same-named CSV." The stem % matches, and $* holds the matched text.
• Why it matters for data. Partitioned datasets (data/2026-01-01.csv, data/2026-01-02.csv, …) get one rule instead of one rule per day. Add a new partition file and it is built automatically.
• Static pattern rules. objs = a.parquet b.parquet; $(objs): %.parquet: %.csv scopes a pattern to an explicit list — useful when only some files follow the pattern.

Order-only prerequisites — directories and non-content deps.

• The problem. You need data/ to exist before writing data/clean.parquet, but the directory's mtime changes every time a file is added — which would spuriously mark every target stale.
• The fix. Order-only prerequisites, listed after a |: data/clean.parquet: data/raw.csv | data/. Make ensures data/ exists first but does not rebuild the target when the directory's mtime changes.
• Common use. Creating output directories, ensuring a virtualenv exists, or any dependency you need present but not tracked for freshness.

Common beginner mistakes.

• Relying on mtime when contents matter — a touch triggers a needless rebuild; a restored old file hides a real change.
• Forgetting to make the recipe write the target file, so the target never looks up to date and rebuilds forever.
• Putting a directory in the normal prerequisite list, so every new file in it marks the target stale (use an order-only | prerequisite).
• Not using pattern rules for partitioned data, so the Makefile grows one rule per file and rots.
• Assuming Make tracks removed prerequisites — deleting a source file does not automatically remove its downstream target.

Worked example — timestamp-based incremental rebuild

Detailed explanation. The heart of incrementality is the mtime comparison. Build a three-node chain, run it cold, then touch one intermediate file and observe that Make rebuilds only the downstream nodes. This is the behaviour that makes Make worth adopting over a shell script that always runs everything.

• Chain. raw.csv → clean.parquet → features.parquet.
• Cold run. Nothing exists; Make builds all three in order.
• Warm run. Everything up to date; Make does nothing.
• After touching clean.parquet. Only features.parquet is downstream and stale; Make rebuilds just that.

Question. Given a three-stage chain, list exactly which recipes run on a cold run, a warm run, and after touch data/clean.parquet.

Input.

File	mtime scenario A (cold)	scenario B (warm)	scenario C (touched clean)
data/raw.csv	exists	exists	exists
data/clean.parquet	missing	up to date	newer (touched)
data/features.parquet	missing	up to date	now stale

Code.

data/clean.parquet: data/raw.csv clean.py
    python clean.py --in $< --out $@

data/features.parquet: data/clean.parquet features.py
    python features.py --in $< --out $@

# Inspect without running: make -n data/features.parquet

Step-by-step explanation.

1. Cold run (make data/features.parquet, nothing built): Make needs features.parquet, whose prerequisite clean.parquet is missing, whose prerequisite raw.csv exists. It builds clean.parquet first, then features.parquet. Two recipes run, in dependency order.
2. Warm run (immediately again): both targets exist and are newer than their prerequisites. Make prints "Nothing to be done" and runs zero recipes. This is the incremental win — a no-op rerun is nearly instant.
3. After touch data/clean.parquet: the intermediate's mtime jumps to now. features.parquet is now older than its prerequisite clean.parquet, so it is stale. But clean.parquet itself is still newer than raw.csv, so it is not rebuilt. Exactly one recipe runs — features.
4. make -n data/features.parquet prints the recipes without executing them, so you can confirm the plan before committing to a long rebuild. This dry-run is the fastest debugging tool for "why did/didn't this rebuild?"
5. The mtime model has a sharp edge: touch-ing a file with identical bytes still triggers a rebuild, and copying in an older file can hide a genuine change. When byte-level correctness matters more than speed, fingerprinting (section 4) is the upgrade.

Output.

Run	Recipes executed
Cold	clean.parquet, then features.parquet
Warm	none ("Nothing to be done")
After touch clean.parquet	features.parquet only
After editing features.py	features.parquet only

Rule of thumb. Make rebuilds a target when it is missing or older than a prerequisite — nothing more. Use make -n to preview the stale sub-graph before a long run, and remember that mtime, not content, drives the decision (so a touch forces a rebuild and a restored old file can hide one).

Worked example — building a four-node data DAG

Detailed explanation. Scale the chain into a realistic DAG with fan-in: two raw sources are cleaned separately, joined into features, and trained into a model. This shows how prerequisites encode lineage and how $^ consumes multiple inputs.

• Sources. orders.csv and customers.csv, cleaned independently (fan-out from "raw" concept).
• Join. features.parquet depends on both cleaned files (fan-in), consumed with $^.
• Model. model.pkl depends on features.parquet.

Question. Write the DAG rules for a two-source, join, train pipeline, and show which nodes rebuild when only customers.csv changes.

Input.

Node	Depends on
clean/orders.parquet	raw/orders.csv
clean/customers.parquet	raw/customers.csv
features.parquet	both cleaned files
model.pkl	features.parquet

Code.

clean/orders.parquet: raw/orders.csv clean.py
    python clean.py --in $< --out $@

clean/customers.parquet: raw/customers.csv clean.py
    python clean.py --in $< --out $@

# Fan-in: features depends on BOTH cleaned files; $^ passes all prereqs
features.parquet: clean/orders.parquet clean/customers.parquet join_features.py
    python join_features.py --inputs clean/orders.parquet clean/customers.parquet --out $@

model.pkl: features.parquet train.py
    python train.py --in $< --out $@

Step-by-step explanation.

1. The two clean/*.parquet rules are independent branches — neither depends on the other. On a cold build with make -j2, Make can build them in parallel because the DAG proves they do not interfere.
2. features.parquet lists both cleaned files as prerequisites (fan-in). If you wanted to pass all prerequisites generically you would use $^; here the script takes an explicit --inputs list, and listing the script (join_features.py) keeps a code change triggering a rebuild.
3. When only raw/customers.csv changes: clean/customers.parquet becomes stale and rebuilds; clean/orders.parquet stays up to date and is skipped; features.parquet (depending on the now-newer customers file) rebuilds; model.pkl rebuilds in turn. Make touches exactly the affected sub-graph.
4. The un-touched branch (clean/orders.parquet) is not rebuilt — this is the lineage-aware skip that a naive run_all.sh cannot do. The DAG encodes precisely which outputs a given input can affect.
5. This is also why make -j parallelism is safe: independent DAG branches have no ordering constraint, so Make schedules them concurrently without you writing any threading code.

Output.

Changed input	Nodes rebuilt
raw/customers.csv	customers.parquet → features.parquet → model.pkl
raw/orders.csv	orders.parquet → features.parquet → model.pkl
join_features.py	features.parquet → model.pkl
nothing	none (warm no-op)

Rule of thumb. Model the pipeline as a DAG of file targets and let Make compute the blast radius of any change. Independent branches build in parallel for free; a change rebuilds only its downstream cone, never the whole graph.

Worked example — pattern rules and order-only prerequisites for partitions

Detailed explanation. Partitioned data — one file per day, per region, per shard — is where pattern rules earn their keep. One % rule builds every partition, and an order-only prerequisite creates the output directory without polluting the freshness check.

• Pattern rule. out/%.parquet: in/%.csv covers every date partition.
• Directory dependency. out/ must exist, but its mtime must not mark targets stale — an order-only | out/ prerequisite.
• Adding a partition. Drop a new CSV in in/; make builds its Parquet automatically.

Question. Write a partition-processing rule that (a) uses a pattern to cover all dates and (b) ensures the output directory exists without triggering spurious rebuilds.

Input.

Component	Value
Inputs	in/2026-01-01.csv, in/2026-01-02.csv, …
Outputs	out/2026-01-01.parquet, …
Directory	out/ must pre-exist
Freshness	directory mtime must NOT mark targets stale

Code.

# Discover partitions and compute their target paths
CSVS     := $(wildcard in/*.csv)
PARQUETS := $(patsubst in/%.csv,out/%.parquet,$(CSVS))

.PHONY: partitions
partitions: $(PARQUETS)            ## build every partition

# Order-only prereq (after |) ensures out/ exists but is not freshness-tracked
out/%.parquet: in/%.csv clean.py | out
    python clean.py --in $< --out $@

out:
    mkdir -p out

Step-by-step explanation.

1. $(wildcard in/*.csv) expands to the list of existing CSV partitions. $(patsubst in/%.csv,out/%.parquet,...) transforms each input path into its target output path — so PARQUETS is the full list of files to build, computed dynamically.
2. partitions: $(PARQUETS) is a phony aggregate target: make partitions builds every partition. Adding a new CSV to in/ automatically extends the list on the next run — no Makefile edit needed.
3. The pattern rule out/%.parquet: in/%.csv clean.py builds any partition from its same-named CSV. $< is the matched CSV, $@ the target Parquet, $* the date stem — one rule for N files.
4. | out is an order-only prerequisite: everything after the | must merely exist, and its mtime is ignored for freshness. This is critical for directories: out/'s mtime bumps every time a file is added, which — as a normal prerequisite — would mark every partition stale on every run. The | prevents that.
5. The out: rule runs mkdir -p out only when out/ is missing. Because it is referenced order-only, it fires once (to create the directory) and never spuriously afterward.

Output.

Action	Result
make partitions (cold)	builds every out/*.parquet
add in/2026-01-03.csv; make partitions	builds only the new partition
rerun (warm)	no-op; directory mtime ignored
rm -rf out; make partitions	recreates out/, rebuilds all

Rule of thumb. Use $(wildcard) + $(patsubst) + a % pattern rule to cover partitioned data with one rule, and always list output directories as order-only prerequisites (after |) so a directory's mtime never marks your data targets stale.

Data engineering interview question on incremental pipelines

A senior interviewer might ask: "You have a daily-partitioned ingest — one CSV per day under in/, each cleaned to Parquet, then all days rolled up into a single summary.parquet. Re-running the whole thing nightly wastes an hour reprocessing unchanged days. Design a Makefile that rebuilds only changed partitions and the rollup, creates the output directory safely, and lets you preview the plan before running."

Solution Using pattern rules, a wildcard partition list, order-only dirs, and a rollup fan-in

# Incremental daily-partition pipeline
CSVS     := $(wildcard in/*.csv)
PARQUETS := $(patsubst in/%.csv,out/%.parquet,$(CSVS))

.PHONY: all preview
all: summary.parquet               ## build changed partitions + rollup

# Per-partition clean (pattern rule); out/ is order-only
out/%.parquet: in/%.csv clean.py | out
    python clean.py --in $< --out $@

# Rollup fan-in: depends on every partition; $^ passes them all
summary.parquet: $(PARQUETS) rollup.py
    python rollup.py --inputs $(PARQUETS) --out $@

out:
    mkdir -p out

preview:                           ## dry-run: show what WOULD rebuild
    @$(MAKE) -n all

Step-by-step trace.

Input state	Stale targets	Recipes run
all partitions fresh	none	none (warm no-op)
edit in/2026-01-02.csv	that day's parquet + summary	2 recipes
add in/2026-01-03.csv	new parquet + summary	2 recipes
edit rollup.py	summary only	1 recipe
edit clean.py	every parquet + summary	N+1 recipes

Take "edit one day's CSV": out/2026-01-02.parquet is now older than its prerequisite and rebuilds; the other days' Parquet files are untouched and skipped; summary.parquet depends on the whole $(PARQUETS) list, so the newer partition marks it stale and it rebuilds once. Two recipes run instead of an hour of full reprocessing. make preview (make -n) prints exactly this plan without executing it, so an on-call engineer can confirm the blast radius before a production run.

Output:

Metric	Full rerun	Incremental (this Makefile)
Recipes on 1-day change	all N days + rollup	1 day + rollup
Wall time on 1-day change	~1 hour	seconds
New partition handling	manual	automatic ($(wildcard))
Preview before run	none	make preview (-n)
Directory safety	ad hoc mkdir	order-only `

Why this works — concept by concept:

• Timestamp-driven staleness — Make rebuilds a partition only when its CSV (or {% raw %}clean.py) is newer than its Parquet. Unchanged days are skipped entirely, turning an O(all-days) nightly job into O(changed-days).
• Wildcard + patsubst partition list — $(wildcard in/*.csv) discovers partitions and $(patsubst) maps them to targets, so new days are picked up automatically with no Makefile edit.
• Pattern rule — one out/%.parquet: in/%.csv rule serves every partition; the stem % and $</$@ keep it generic, so the file does not grow one rule per day.
• Fan-in rollup with $(PARQUETS) — summary.parquet listing every partition as a prerequisite means any changed day correctly marks the rollup stale, so the summary is never silently out of date.
• Order-only | out — the output directory is required to exist but excluded from freshness, so its mtime churn never triggers spurious full rebuilds. Combined with make -n preview, the plan is both minimal and inspectable.
• Cost — build cost is O(changed partitions + 1 rollup) instead of O(all partitions); the only overhead is the wildcard scan (microseconds) and keeping clean.py listed as a prerequisite so code changes still force a rebuild.

ETL
Topic — etl
ETL problems on incremental and partitioned loads

Practice →

SQL Topic — pandas Pandas problems on dependency graphs

Practice →

---

4. Taskfile (YAML) as a modern alternative

Taskfile.yml — named tasks, deps, and checksum fingerprints as a friendlier, cross-platform task runner

The mental model in one line: Taskfile is a task runner that keeps Make's dependency-graph model but swaps the syntax for YAML and the freshness check from mtime to content checksums — you write named tasks with deps (other tasks that must run first) and cmds (the commands), and you declare sources (inputs) and generates (outputs) so task fingerprints them and skips a task whose inputs have not actually changed. For teams that find Make's TAB-and-$$ quoting rules hostile, or who need first-class Windows support, Taskfile is the modern ergonomic upgrade — a single Go binary, one YAML file, and checksum-based incrementality that does not lie about touch.

Why reach for Taskfile over Make.

• No TAB trap. Taskfile is YAML — indentation is spaces, and there is no "missing separator" landmine. The single most common Make onboarding failure disappears.
• Cross-platform. task is one static Go binary that runs identically on Linux, macOS, and Windows. Make on Windows means MinGW/WSL gymnastics; task just works.
• Content fingerprints, not mtime. sources/generates are hashed; a task is skipped when the contents of its sources are unchanged, so a touch does not force a needless rebuild and a restored-but-different file is not missed.
• Readable structure. Named tasks, desc fields for task --list, deps, vars, env, and includes for splitting large Taskfiles — a more discoverable surface than a wall of Make rules.

Anatomy of a Taskfile.

• version and tasks. The file declares a version: and a tasks: map. Each key is a task name; task <name> runs it.
• cmds. A list of shell commands the task runs, top to bottom. Multi-line and templated commands are supported.
• deps. A list of tasks that must complete before this task's cmds. deps run in parallel by default — the DAG edge, like a Make prerequisite.
• desc. A one-line description surfaced by task --list, giving you a free, always-current command catalogue.

Incrementality — sources, generates, status, preconditions.

• sources + generates. Declare the input files and output files. task computes a checksum of the sources; if it matches the last run and the generates targets exist, the task is skipped (up to date).
• method. checksum (default) hashes contents; timestamp mimics Make's mtime behaviour; none always runs. Choose checksum for true content-based skipping.
• status. A list of shell checks; if all exit 0, the task is considered up to date. Useful for "skip if this table already has today's partition" style guards that files cannot express.
• preconditions. Shell checks that must pass or the task fails (not skips) — the fail-fast equivalent of Make's checksum gate (e.g. "abort unless the input checksum matches").

Variables, includes, and templating.

• vars. File-level or task-level variables (vars: { SEED: 42 }), referenced as {{.SEED}} in commands using Go template syntax.
• env. Environment variables for a task's commands; supports .env file loading via dotenv:.
• includes. Split a large Taskfile into per-domain files (includes: { db: ./db/Taskfile.yml }) and call task db:migrate — the modularity Make lacks.
• Templating. {{.VAR}}, {{.CLI_ARGS}} (pass-through CLI args), and functions like {{.ITEM}} in loops give more expressive commands than Make's raw variable expansion.

Common beginner mistakes.

• Expecting deps to run sequentially — they run in parallel; use cmds calling other tasks (or run: once) when order matters.
• Omitting generates, so checksum skipping cannot verify the output exists and the task reruns.
• Using status where preconditions was meant — status skips, preconditions fails; confusing them hides real errors.
• Forgetting that Taskfile variables use {{.VAR}} (Go templates), not Make's $(VAR).
• Assuming checksum fingerprints are stored in git — they live in a local .task/ cache; CI needs the cache or will rebuild (which is usually the safe default).

Worked example — a basic Taskfile with deps

Detailed explanation. Recreate the extract → transform → load chain as named Taskfile tasks with deps edges. This mirrors the Make DAG but in YAML, and shows how deps encode the dependency graph and how a top-level task composes the pipeline.

• Tasks. extract, transform, load, and a pipeline aggregate.
• Edges. transform deps on extract; load deps on transform.
• Entrypoint. task pipeline (or task default) runs the whole chain.

Question. Write a Taskfile that runs extract → transform → load with correct ordering, and explain how deps differ from cmds-that-call-tasks.

Input.

Task	deps	cmds
extract	—	download raw.csv
transform	extract	build clean.parquet
load	transform	write to warehouse
pipeline	load	echo done

Code.

version: '3'

tasks:
  extract:
    desc: Download the raw dataset
    cmds:
      - python extract.py --out data/raw.csv

  transform:
    desc: Clean the raw dataset
    deps: [extract]
    cmds:
      - python transform.py --in data/raw.csv --out data/clean.parquet

  load:
    desc: Load the cleaned data into the warehouse
    deps: [transform]
    cmds:
      - python load.py --in data/clean.parquet

  pipeline:
    desc: Run the full extract-transform-load pipeline
    deps: [load]
    cmds:
      - echo "pipeline OK"

Step-by-step explanation.

1. version: '3' selects the Taskfile schema; tasks: maps each task name to its definition. task --list prints every task with its desc — a free, always-current catalogue.
2. transform declares deps: [extract], so task transform runs extract first, then its own cmds. This is the DAG edge — the Taskfile equivalent of a Make prerequisite.
3. task pipeline follows the chain: pipeline deps on load, which deps on transform, which deps on extract. Task resolves the graph and runs them in order.
4. A subtlety: deps execute in parallel with each other. Here each task has a single dep, so ordering is linear. If a task listed two deps, both would run concurrently — fine for independent branches, wrong if one must precede the other.
5. When you need strict sequential ordering inside one task (not a fan-in), call tasks from cmds with - task: other-task instead of deps. cmds run top-to-bottom; deps run in parallel — this distinction is the most common Taskfile gotcha.

Output.

Command	Runs
task extract	extract only
task transform	extract, then transform
task pipeline	extract → transform → load → echo
task --list	catalogue of all tasks + descriptions

Rule of thumb. Model the pipeline as named tasks with deps edges, exactly like Make prerequisites — but remember deps run in parallel. When two steps must run in a fixed order within one task, sequence them in cmds with - task: calls, not deps.

Worked example — checksum fingerprints with sources and generates

Detailed explanation. The feature that makes Taskfile genuinely better than Make for reproducibility is content-based fingerprinting. Declare sources and generates; task hashes the sources and skips the task when they are byte-for-byte unchanged — immune to the touch false-positive and the restored-old-file false-negative that plague mtime.

• sources. The input files (globs allowed) whose contents determine freshness.
• generates. The output files the task produces; their existence is part of the up-to-date check.
• method: checksum. Hash contents (default). task stores the hash in .task/ and compares on the next run.

Question. Add sources/generates fingerprinting to the transform task so it skips when the input contents are unchanged, and contrast the behaviour with Make's mtime after a touch.

Input.

Field	Value
sources	data/raw.csv, transform.py
generates	data/clean.parquet
method	checksum (default)
touch data/raw.csv	Make: rebuilds; Task: skips (hash unchanged)

Code.

version: '3'

tasks:
  transform:
    desc: Clean raw.csv into clean.parquet (checksum-fingerprinted)
    sources:
      - data/raw.csv
      - transform.py
    generates:
      - data/clean.parquet
    # method: checksum is the default; hashes source CONTENTS
    cmds:
      - python transform.py --in data/raw.csv --out data/clean.parquet

Step-by-step explanation.

1. sources lists the input files (the data and the code). task computes a checksum over their contents and stores it under .task/. generates lists the output whose existence is also checked.
2. First run: no stored checksum, so the task runs and records the source hash. Second run with unchanged sources: the hash matches and data/clean.parquet exists, so task prints "task: Task 'transform' is up to date" and skips — the same incremental win as Make.
3. The decisive difference is touch data/raw.csv. Under Make, touching bumps the mtime and forces a rebuild even though bytes are identical. Under Taskfile's checksum method, the content hash is unchanged, so the task correctly skips — no wasted rebuild.
4. The inverse safety also holds: if you restore an older copy of raw.csv whose contents differ, mtime-based Make might see an older file and skip; Taskfile's hash differs, so it correctly rebuilds. Content fingerprints do not lie about change.
5. method: timestamp is available if you want Make-like mtime behaviour (faster on huge files where hashing is expensive), and method: none forces the task to always run. checksum is the reproducibility-friendly default.

Output.

Event	Make (mtime)	Taskfile (checksum)
unchanged rerun	skip	skip
touch raw.csv (same bytes)	rebuild (false positive)	skip (correct)
restore older, different file	may skip (false negative)	rebuild (correct)
edit transform.py	rebuild	rebuild

Rule of thumb. Declare sources and generates on every Taskfile task that produces a file, and keep the default checksum method. Content fingerprints skip exactly when the inputs are truly unchanged — immune to the touch and restored-file surprises that make raw Make mtime occasionally lie.

Worked example — variables, dotenv, and a status guard

Detailed explanation. Taskfile's vars, dotenv, and status fields cover the reproducibility knobs and the "skip if already done" checks that files alone cannot express. Wire a seed variable, load a .env, and add a status guard that skips loading when today's partition already exists.

• vars. { SEED: 42 }, referenced as {{.SEED}}.
• dotenv. Load .env so secrets/config are not hard-coded.
• status. A shell check; if it exits 0, the task is up to date and skips — for conditions files cannot capture.

Question. Write a load task that (a) threads a seed variable, (b) reads connection config from .env, and (c) skips if today's partition is already loaded.

Input.

Field	Value
vars	SEED: 42
dotenv	.env (holds WAREHOUSE_URL)
status	partition-exists.sh $(date +%F) exits 0 → skip
template	{{.SEED}}, {{.DATE}}

Code.

version: '3'

dotenv: ['.env']                   # loads WAREHOUSE_URL etc.

vars:
  SEED: 42

tasks:
  load:
    desc: Load today's partition (skips if already loaded)
    vars:
      DATE:
        sh: date +%F               # dynamic var from a shell command
    status:
      # If today's partition already exists, task is up to date -> skip
      - ./partition-exists.sh {{.DATE}}
    cmds:
      - python load.py --date {{.DATE}} --seed {{.SEED}} --url "$WAREHOUSE_URL"

Step-by-step explanation.

1. dotenv: ['.env'] loads key-value pairs from .env into the environment, so $WAREHOUSE_URL is available in cmds without hard-coding the connection string. This keeps secrets out of the Taskfile and makes the same file work across environments.
2. vars: { SEED: 42 } defines a file-level variable referenced as {{.SEED}} (Go template syntax). It threads a deterministic seed into the load command — the reproducibility knob, overridable with task load SEED=7.
3. The task-level DATE var uses sh: date +%F, a dynamic variable computed by running a shell command at task time. {{.DATE}} then expands to today's date in both the status check and the command.
4. status runs ./partition-exists.sh {{.DATE}}. If that script exits 0 (the partition is already loaded), task treats load as up to date and skips it entirely. This expresses a "skip if already done" condition that no file-timestamp check could — the state lives in the warehouse, not on disk.
5. The distinction from preconditions matters: status skips when satisfied (idempotent re-runs are free), whereas preconditions fails when unsatisfied (guarding against bad inputs). Using status here means re-running the pipeline after a partial failure re-loads only the missing partitions.

Output.

Scenario	Result
partition not yet loaded	status non-zero → task runs
partition already loaded	status exits 0 → task skips
task load SEED=7	overrides seed to 7
.env missing WAREHOUSE_URL	command sees empty var (add a precondition)

Rule of thumb. Use vars for reproducibility knobs ({{.SEED}}), dotenv to keep config out of the file, and status for "skip if already done" checks that live in a database or API rather than on disk. Reach for preconditions when a failed check should abort rather than skip.

Data engineering interview question on porting Make to Taskfile

A senior interviewer might ask: "Your team's Windows users can't run the project's Makefile, and people keep hitting the TAB-versus-spaces error. Port the extract → transform → load pipeline to a Taskfile, keep incrementality (skip unchanged steps), add a seed variable and a .env, and explain what you gain and lose versus Make."

Solution Using a Taskfile with deps, sources/generates checksums, vars, and dotenv

version: '3'

dotenv: ['.env']

vars:
  SEED: 42

tasks:
  default:
    desc: Run the full pipeline
    deps: [load]

  extract:
    desc: Download raw data
    sources: [extract.py]
    generates: [data/raw.csv]
    cmds:
      - python extract.py --out data/raw.csv

  transform:
    desc: Clean raw.csv into clean.parquet
    deps: [extract]
    sources: [data/raw.csv, transform.py]
    generates: [data/clean.parquet]
    cmds:
      - python transform.py --in data/raw.csv --out data/clean.parquet

  load:
    desc: Train/load with a deterministic seed
    deps: [transform]
    sources: [data/clean.parquet, load.py]
    generates: [model.pkl]
    cmds:
      - python load.py --in data/clean.parquet --out model.pkl --seed {{.SEED}} --url "$WAREHOUSE_URL"

Step-by-step trace.

Concern	Make	Taskfile (this port)
Windows support	MinGW/WSL needed	native (single Go binary)
TAB/spaces error	frequent	impossible (YAML)
Incrementality	mtime	sources/generates checksum
Seed knob	SEED ?= 42	vars: {SEED: 42} → {{.SEED}}
Config	env / include	dotenv: ['.env']
Discoverability	make help hack	task --list (built-in)

Running task (the default task) resolves default → load → transform → extract and runs the chain. Each task checksums its sources; a rerun with unchanged inputs prints "up to date" and skips — even after a touch, because the check is content-based. A Windows teammate runs the identical file with no WSL. Overriding the seed is task SEED=7, and the warehouse URL comes from .env rather than being hard-coded.

Output:

Metric	Make	Taskfile
Cross-platform	partial	full (win/mac/linux)
Freshness accuracy	mtime (touch lies)	checksum (content-true)
Onboarding failures	TAB errors	none
Command catalogue	manual help	task --list free
Trade-off	ubiquitous, zero-install	needs task installed

Why this works — concept by concept:

• YAML tasks with deps — named tasks and deps edges reproduce Make's DAG without the TAB-and-$$ syntax traps, so the dependency graph is identical but the onboarding failures vanish.
• sources/generates checksums — content fingerprints skip a task exactly when its inputs are byte-unchanged, fixing the touch false-positive and restored-file false-negative that mtime-based Make can hit.
• vars plus {{.SEED}} templating — the reproducibility knob is a first-class variable, overridable per invocation, threaded into the command via Go templates.
• dotenv config loading — connection strings live in .env, not the Taskfile, so the same file is portable across environments without editing.
• Cost — you gain cross-platform support, content-accurate incrementality, and a built-in command catalogue; you lose Make's zero-install ubiquity (every teammate and CI image must have the task binary). For Windows-inclusive teams the trade is almost always worth it; for a pure-Linux, Make-everywhere shop it may not be.

SQL
Topic — pandas
Pandas problems on task automation

Practice →

ETL Topic — etl ETL problems on pipeline configuration

Practice →

---

5. Patterns for reproducible pipelines and CI

The same Makefile locally and in CI — the patterns that make a data pipeline reproducible everywhere

The mental model in one line: reproducibility across machines comes from running the exact same task-runner file everywhere — laptop, teammate, CI runner — over pinned inputs and a pinned toolchain, with deterministic randomness, and then proving it with a checksum, so a green CI build is evidence the pipeline produces byte-identical outputs and not just that "some script exited 0". The task runner is the single source of truth; the CI job is just another caller of make pipeline. Everything that makes a run non-reproducible — floating dependency versions, ambient environment state, unseeded randomness, unverified inputs — is closed off in that one file.

Local + CI parity — one file, two callers.

• CI calls the runner. The CI config should be a thin wrapper: check out code, install the pinned toolchain, run make pipeline. All real logic lives in the Makefile so local and CI cannot diverge.
• No CI-only steps. The moment CI does something the Makefile does not (an extra flag, a special env), "works locally, fails in CI" incidents return. Push every step into the shared file.
• Reproduce CI locally. Because CI just runs make, an engineer can reproduce a CI failure with the same make command — no guessing what the runner did differently.
• Pin the runner image. Pin the CI base image and the toolchain (Python version, task/make version) so the environment itself is versioned.

Hermetic inputs and pinned toolchain.

• Locked dependencies. A lockfile (uv.lock, poetry.lock, requirements.txt with hashes) pins exact library versions. Invoke it from the Makefile (PYTHON := uv run python) so every run uses the same libraries.
• Pinned data snapshots. Inputs should be immutable, versioned snapshots (a dated S3 path, a DVC-tracked file), not "latest," so the same command sees the same bytes.
• Deterministic randomness. Thread a fixed seed (--seed 42, PYTHONHASHSEED=0) so stochastic steps produce identical outputs.
• Sorted / stable outputs. Write outputs deterministically — sort rows, fix float formatting, set a stable Parquet compression — so the bytes match, which is what a checksum verifies.

Parallelism and safe-fail flags.

• make -j. Builds independent DAG branches in parallel. make -j4 uses four jobs; make -j (no number) is unlimited — great locally, risky in CI (resource contention), so pin the job count.
• .NOTPARALLEL. A special target that forces sequential execution for a Makefile where parallelism is unsafe (e.g. shared temp files).
• .DELETE_ON_ERROR. A special target that deletes a target file if its recipe fails partway — without it, a half-written Parquet is left behind and looks "up to date" on the next run, silently shipping corrupt data.
• --output-sync=target. With -j, interleaved parallel logs are unreadable; --output-sync=target groups each target's output so CI logs stay legible.

Reproducibility guardrails.

• Checksum the output. After building, compute and compare a checksum of the final artifact against a committed expected value (make check). A mismatch is a hard failure — this is what turns "the build ran" into "the build produced the right bytes."
• make check in CI. Add a check target that verifies output checksums and run it as the CI gate. Green CI now means "reproducible," not just "no crash."
• Fail on drift. Verify input checksums before building (section 1) and output checksums after; drift on either side is a loud, early failure.
• Cache the fingerprint store. For Taskfile, cache .task/ in CI so unchanged steps skip; for Make, cache the artifact directory. But default to not trusting the cache for the final check — recompute the output hash fresh.

Common beginner mistakes.

• Putting real logic in the CI YAML instead of the Makefile, so local and CI diverge.
• Running make -j unbounded in CI and hitting OOM or flaky resource contention — pin -j to the runner's core count.
• Omitting .DELETE_ON_ERROR, leaving half-written outputs that look up to date and ship corrupt.
• Treating a green build as "reproducible" without a make check output-checksum gate.
• Forgetting PYTHONHASHSEED / a fixed seed, so set/dict ordering and model init drift between runs.

Worked example — one Makefile, laptop and CI parity

Detailed explanation. The parity pattern: the CI job installs the pinned toolchain and calls make pipeline, nothing more. All ordering, flags, and gates live in the Makefile, so the CI run is literally the local run on a different machine.

• CI job. Checkout → install pinned deps → make pipeline.
• Makefile. Owns the pinned interpreter, the build graph, and the checks.
• Local reproduction. The same make pipeline reproduces any CI result.

Question. Write a minimal CI config and the Makefile it calls so that local and CI runs are guaranteed to execute the same steps.

Input.

Layer	Responsibility
CI YAML	checkout, install pinned toolchain, call make
Makefile	pinned interpreter, build graph, checks
Parity guarantee	CI runs only make targets, no extra logic

Code.

# .github/workflows/pipeline.yml — a THIN wrapper
name: pipeline
on: [push]
jobs:
  build:
    runs-on: ubuntu-24.04           # pinned runner image
    steps:
      - uses: actions/checkout@v4
      - run: pipx install uv==0.5.11 # pinned tool version
      - run: uv sync --frozen        # install from the lockfile
      - run: make pipeline           # ALL real logic lives here
      - run: make check              # reproducibility gate

# Makefile — the single source of truth
PYTHON := uv run python
.PHONY: pipeline check

pipeline: model.pkl

model.pkl: data/features.parquet train.py
    $(PYTHON) train.py --in $< --out $@ --seed 42

check:                              ## verify the output is reproducible
    sha256sum -c model.pkl.sha256

Step-by-step explanation.

1. The CI YAML does four things and no logic: check out the code, install a pinned uv, install frozen (lockfile-exact) dependencies, and call make. There is no bespoke build command in the YAML that could drift from local behaviour.
2. uv sync --frozen installs the exact versions in the lockfile. Combined with PYTHON := uv run python in the Makefile, every python invocation — local or CI — uses identical library versions. The toolchain is pinned end to end.
3. make pipeline is the same command an engineer runs locally. Because all ordering and flags live in the Makefile, reproducing a CI failure is make pipeline on your laptop — no "what did CI do differently" investigation.
4. make check runs sha256sum -c against a committed hash of model.pkl. If the pinned inputs and toolchain truly produce a byte-identical model, the check passes; if anything drifted, CI fails loudly. This is the step that turns "green" into "reproducible."
5. The anti-pattern this prevents: teams that put python train.py --in ... --out ... --seed 42 directly in the CI YAML inevitably let it drift from the local command (a different flag, a different path), and "works locally, fails in CI" returns. One shared file, called identically, removes the divergence.

Output.

Aspect	Logic in CI YAML (bad)	Logic in Makefile (good)
Local == CI steps	not guaranteed	guaranteed
Reproduce CI failure	guess CI's commands	run make pipeline
Drift risk	high	low
Toolchain pin	ad hoc	lockfile + pinned tool

Rule of thumb. Keep the CI config a thin wrapper — checkout, install pinned toolchain, make pipeline, make check — and put every real step in the shared task-runner file. Local and CI then run the same commands by construction, so "works on my machine" stops being a category of bug.

Worked example — parallelism with safe-fail flags

Detailed explanation. Parallel builds speed up independent DAG branches, but naive -j introduces two hazards: half-written outputs on failure and unreadable interleaved logs. .DELETE_ON_ERROR, a pinned -j, and --output-sync make parallelism safe and legible.

• make -j4. Build up to four independent targets at once.
• .DELETE_ON_ERROR. Delete a target whose recipe failed, so no corrupt half-file is left "up to date."
• --output-sync=target. Group each target's log output instead of interleaving.

Question. Configure a Makefile and invocation so independent partitions build in parallel, a failed recipe never leaves a corrupt file, and logs stay readable.

Input.

Flag / target	Effect
-j4	4 parallel jobs
.DELETE_ON_ERROR	remove target on recipe failure
--output-sync=target	grouped per-target logs
.NOTPARALLEL	force sequential (if unsafe)

Code.

# Safe parallel builds
.DELETE_ON_ERROR:                  # <-- delete half-written targets on failure

CSVS     := $(wildcard in/*.csv)
PARQUETS := $(patsubst in/%.csv,out/%.parquet,$(CSVS))

.PHONY: all
all: $(PARQUETS)

out/%.parquet: in/%.csv clean.py | out
    python clean.py --in $< --out $@   # if this fails mid-write, target is deleted

out:
    mkdir -p out

# Invoke with pinned parallelism + grouped logs:
#   make -j4 --output-sync=target all

Step-by-step explanation.

1. .DELETE_ON_ERROR: is a special target (no recipe). It tells Make: if a recipe exits non-zero after starting to write its target file, delete that partial file. Without it, a clean.py that crashes after opening out/day.parquet leaves a truncated file that Make sees as newer than its input — so the next run treats corrupt data as "up to date."
2. The pattern rule builds partitions independently. Because the DAG proves they do not depend on each other, make -j4 runs up to four clean.py invocations concurrently, cutting wall time on a many-partition backfill roughly fourfold.
3. Pinning -j4 (not bare -j) bounds concurrency to the machine's capacity. Unbounded -j in CI can spawn a job per target and exhaust memory; a pinned number matched to the runner's cores is the safe choice.
4. --output-sync=target groups each target's stdout/stderr so the log reads as coherent per-file blocks instead of interleaved noise from four concurrent jobs — essential for debugging a parallel CI failure.
5. If some targets cannot run in parallel (they share a temp file or a non-reentrant resource), add the .NOTPARALLEL special target to force sequential execution for the whole Makefile, or restructure so the shared resource becomes an explicit prerequisite.

Output.

Situation	Without flags	With flags
recipe fails mid-write	corrupt file kept	file deleted (.DELETE_ON_ERROR)
8 partitions, 4 cores	sequential	~4x faster (-j4)
parallel logs	interleaved	grouped (--output-sync)
shared temp file	race	.NOTPARALLEL serialises

Rule of thumb. Always add .DELETE_ON_ERROR: so a failed recipe cannot leave a corrupt "up to date" artifact, pin -j to the core count rather than using unbounded parallelism, and add --output-sync=target to keep parallel logs readable. Reach for .NOTPARALLEL only when targets genuinely share a non-reentrant resource.

Worked example — checksum guardrail for reproducibility

Detailed explanation. The final reproducibility guardrail is an output checksum gate. Compute a hash of the final artifact, commit the expected value, and add a check target that fails on mismatch. This is the step that converts "the pipeline ran" into "the pipeline produced the agreed bytes."

• Generate expected hash. Once, on a known-good run: sha256sum model.pkl > model.pkl.sha256.
• Verify. make check runs sha256sum -c model.pkl.sha256.
• CI gate. Run make check after make pipeline; a mismatch fails the build.

Question. Add a reproducibility gate that fails CI when the pipeline output does not match a committed checksum, and explain what class of bug it catches.

Input.

Artifact	Value
Output	model.pkl
Expected hash file	model.pkl.sha256 (committed)
Gate	make check → sha256sum -c
Failure	non-zero exit → CI red

Code.

.PHONY: pipeline check freeze

pipeline: model.pkl

# Verify the output matches the committed hash (reproducibility gate)
check: model.pkl
    sha256sum -c model.pkl.sha256

# One-time / intentional update of the expected hash after a reviewed change
freeze: model.pkl
    sha256sum model.pkl > model.pkl.sha256
    @echo "expected hash updated -- commit model.pkl.sha256"

# CI step order
make pipeline          # build
make check             # fail if output drifted from the committed hash

Step-by-step explanation.

1. freeze is run once, deliberately, on a reviewed known-good build: it writes the current model.pkl hash into model.pkl.sha256, which you commit. This file becomes the contract for "what a correct output looks like."
2. check depends on model.pkl (so it builds first if needed) and runs sha256sum -c model.pkl.sha256. If the freshly built model hashes to the committed value, check exits 0; if it differs by a single byte, it exits non-zero.
3. In CI, running make check after make pipeline turns a green build into a reproducibility assertion. A passing pipeline that produces a different model now fails the gate — catching exactly the silent drift that unit tests miss.
4. The class of bug this catches: a floating dependency that reorders floats, an unseeded shuffle, a locale-dependent sort, a non-deterministic Parquet write. None of these throw an exception; all of them change the output bytes. The checksum is the only thing that notices.
5. When a change to the output is intentional (a real model improvement), you rerun freeze, review the new hash in the diff, and commit it — making every output change an explicit, reviewed event rather than an accident.

Output.

Event	make check result
reproducible build	pass (exit 0)
unseeded randomness	fail (hash differs)
dependency drift	fail (hash differs)
intentional change + freeze	pass (new committed hash)

Rule of thumb. Add a check target that verifies the final artifact against a committed sha256, and run it as the CI gate right after the build. It is the cheapest possible guard against silent non-reproducibility — the bugs that never throw an exception but quietly change your outputs.

Data engineering interview question on CI reproducibility

A senior interviewer might ask: "A data pipeline is green in CI but two engineers get different model files locally. There's no seed, dependencies float, and CI has a few build steps hard-coded in the YAML that differ from the README. Make it reproducible: one entrypoint, pinned toolchain, deterministic output, parallel-safe, and a gate that fails when the output drifts. Walk me through the Makefile and the CI wiring."

Solution Using a pinned toolchain, deterministic seeds, safe-fail parallelism, and a checksum gate

# Makefile — reproducible everywhere
.DELETE_ON_ERROR:                  # no corrupt half-written artifacts

PYTHON := uv run python            # pinned interpreter + locked deps
export PYTHONHASHSEED := 0         # stable set/dict ordering
SEED   := 42

.PHONY: pipeline check freeze clean

pipeline: verify model.pkl         ## verify inputs, build, deterministically

verify:                            ## input drift = fail fast
    sha256sum -c data/raw.csv.sha256

data/clean.parquet: data/raw.csv clean.py
    $(PYTHON) clean.py --in $< --out $@ --sort-rows

data/features.parquet: data/clean.parquet features.py
    $(PYTHON) features.py --in $< --out $@

model.pkl: data/features.parquet train.py
    $(PYTHON) train.py --in $< --out $@ --seed $(SEED)

check: model.pkl                   ## output drift = fail
    sha256sum -c model.pkl.sha256

freeze: model.pkl
    sha256sum model.pkl > model.pkl.sha256

clean:
    rm -f data/clean.parquet data/features.parquet model.pkl

# .github/workflows/pipeline.yml — thin wrapper, no logic
name: pipeline
on: [push]
jobs:
  build:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
      - run: pipx install uv==0.5.11
      - run: uv sync --frozen
      - run: make -j4 --output-sync=target pipeline
      - run: make check              # reproducibility gate

Step-by-step trace.

Non-reproducibility source	Fix in this solution
floating dependency versions	uv sync --frozen + uv run python
unseeded randomness	--seed 42 + PYTHONHASHSEED=0
unstable row order	--sort-rows deterministic output
CI steps drift from local	thin YAML calls make only
corrupt file on failure	.DELETE_ON_ERROR
silent output drift	make check vs committed sha256

Both engineers now run make -j4 pipeline against the same lockfile and the same seed, so train.py produces a byte-identical model.pkl; make check confirms it matches the committed hash. The CI YAML installs the pinned uv, syncs frozen deps, and calls the same make — so a CI failure is reproducible locally with one command. .DELETE_ON_ERROR guarantees a crashed recipe never leaves a corrupt artifact masquerading as up to date.

Output:

Metric	Before	After
Two engineers' model.pkl	differ	byte-identical
Dependency versions	floating	lockfile-frozen
Randomness	nondeterministic	seeded + PYTHONHASHSEED=0
CI == local steps	no	yes (thin wrapper)
Output drift detection	none	make check gate
Corrupt-artifact risk	present	removed (.DELETE_ON_ERROR)

Why this works — concept by concept:

• Pinned toolchain via uv sync --frozen — installing the exact lockfile versions and invoking uv run python from the Makefile means every run, everywhere, uses identical libraries, removing the "different version reorders floats" class of drift.
• Deterministic randomness — --seed 42 plus export PYTHONHASHSEED := 0 fixes both model initialisation and hash-based set/dict ordering, so stochastic and ordering-sensitive steps produce identical bytes.
• Deterministic output writes — --sort-rows (and stable float/compression settings) ensure the bytes are stable, which is the precondition for a checksum to mean anything.
• Thin CI wrapper — the workflow only checks out, pins the tool, syncs frozen deps, and calls make; with no logic in the YAML, local and CI cannot diverge, so CI failures reproduce locally.
• .DELETE_ON_ERROR + pinned -j4 — parallel builds are fast yet safe: a failed recipe's partial output is deleted rather than cached as up to date, and bounded parallelism avoids CI resource exhaustion.
• Cost — the guardrails add a lockfile, a seed, a sort flag, two checksum files, and a .DELETE_ON_ERROR line — a handful of lines. In return, build cost stays O(changed sub-graph), reproducibility becomes an enforced CI gate rather than a hope, and "works on my machine" is eliminated by construction.

Defensive
Topic — defensive-coding
Defensive-coding problems on reproducible, safe-fail pipelines

Practice →

ETL
Topic — etl
ETL problems on CI-driven pipeline runs

Practice →

---

Cheat sheet — Makefile and Taskfile recipes

• Rule syntax. Every Make rule is target: prerequisites then a TAB-indented recipe that writes the file named by the target. Bare make builds the first target, so make the first real target all (or pipeline). List the script as a prerequisite alongside the data so code edits trigger rebuilds. Diagnose the TAB trap with cat -A Makefile (recipes must show ^I, not spaces).
• Automatic variables. $@ = the target, $< = the first prerequisite, $^ = all prerequisites (deduped), $* = the pattern stem. Use them in every recipe so no path is written twice. Double a literal shell $ as $$ (e.g. $$HOME) so Make does not eat it.
• .PHONY list. Declare every action target phony: .PHONY: all clean test lint help pipeline. This forces the recipe to run regardless of a same-named file and kills the silent-no-op bug. Any target whose name is a verb, not a file, belongs here.
• Variable assignment. := simple (evaluate once — the default), = recursive (evaluate each use — usually avoid), ?= default-if-unset (PYTHON ?= python3, overridable by env/CLI). Keep tools and directories in top-of-file variables.
• Incremental file-target template. out.parquet: in.csv transform.py + a recipe writing $@ from $<. Make rebuilds only when a prerequisite is newer or the target is missing. Preview the stale sub-graph with make -n before a long run.
• Pattern-rule template. out/%.parquet: in/%.csv clean.py | out builds every partition from one rule; discover inputs with CSVS := $(wildcard in/*.csv) and map to targets with $(patsubst in/%.csv,out/%.parquet,$(CSVS)). Add a new input file and it builds automatically.
• Order-only prerequisites. List directories after a | (target: dep | out/) so the directory must exist but its mtime does not mark the target stale. Pair with an out: rule running mkdir -p out.
• Taskfile skeleton. version: '3'; tasks: map of named tasks; each with desc, deps (DAG edges, run in parallel), and cmds. task --list prints the catalogue. Remember: deps are parallel — sequence with - task: inside cmds when order matters.
• Taskfile fingerprinting. Declare sources: and generates: with the default method: checksum so a task skips only when its input contents are unchanged — immune to touch (unlike Make's mtime). Use status: to skip on an external condition (partition already loaded) and preconditions: to fail on a bad input.
• Taskfile config. vars: { SEED: 42 } referenced as {{.SEED}}; dotenv: ['.env'] to load config; includes: to split large Taskfiles into task db:migrate style namespaces; {{.CLI_ARGS}} to pass through command-line args.
• Parallel + safe-fail. make -j4 --output-sync=target for bounded, legible parallelism; add .DELETE_ON_ERROR: so a failed recipe cannot leave a corrupt "up to date" file; add .NOTPARALLEL only when targets share a non-reentrant resource. Never run unbounded -j in CI.
• CI parity + reproducibility gate. Keep CI a thin wrapper: checkout → install pinned tool → uv sync --frozen → make pipeline → make check. Pin the interpreter (PYTHON := uv run python), fix randomness (--seed, export PYTHONHASHSEED := 0), write deterministic outputs (sorted rows, stable compression), and gate on a committed output sha256 so green means reproducible, not just no crash.

Frequently asked questions

What is a task runner and why use one for data pipelines?

A task runner is a tool that reads a small declarative file — a Makefile for GNU Make or a Taskfile.yml for Taskfile — describing your pipeline as a dependency graph of named targets and the commands that build them, and then runs those commands in the correct order from a single invocation like make pipeline or task pipeline. For data workflows it replaces the "numbered scripts plus a stale README" anti-pattern with one machine-executable source of truth that is simultaneously the runbook, the automation, and the dependency documentation. The three concrete wins are one-command execution (new hires productive in minutes), incrementality (rebuild only what changed instead of the whole pipeline), and reproducibility (the same command over pinned inputs and toolchain yields the same output). Because the file is checked into git, the pipeline's structure is version-controlled next to the code instead of drifting in prose.

Makefile vs Taskfile — which should I pick?

Pick Make when you value zero-install ubiquity: it ships on every Linux and macOS box, needs nothing installed, and its file-target model maps perfectly onto data artifacts. Pick Taskfile when you want ergonomics and cross-platform support: it is a single Go binary that runs identically on Windows, uses YAML (so there is no TAB-versus-spaces trap), and fingerprints tasks by content checksum rather than file mtime — so a touch never forces a needless rebuild and a restored-but-different file is never missed. A Windows-inclusive team, or one that keeps hitting Make's quoting and tab rules, is usually happier with Taskfile; a pure-Linux shop where make is already everywhere may prefer the ubiquity. Both express the same dependency-graph model, so the mental model transfers either way — the trade is ubiquity (Make) versus friendlier syntax and content-accurate incrementality (Taskfile).

How does Make decide what to rebuild?

GNU Make is timestamp-driven. For each target it compares the target file's modification time against the modification time of every prerequisite; if the target is missing, or any prerequisite is newer, the recipe runs, otherwise it is skipped. Because each target's prerequisites are usually themselves targets, this forms a dependency DAG that Make evaluates bottom-up, rebuilding exactly the stale sub-graph and leaving up-to-date artifacts alone. The important caveat is that Make looks at mtime, not file contents: touch-ing a file forces a rebuild even if the bytes are identical, and copying in an older file with an older mtime can hide a real change. Use make -n to preview which recipes would run without executing them, and switch to Taskfile's checksum method when content-accurate freshness matters more than raw speed.

What are .PHONY targets and when do I need them?

A .PHONY target is a Make target that names an action rather than a file — clean, all, test, lint, help, or a pipeline entrypoint like pipeline. Normally Make treats a target name as a filename and considers it "up to date" if a file of that name exists and is newer than its prerequisites; so if a file or directory literally named clean ever appears, make clean silently does nothing. Declaring .PHONY: clean all test tells Make these targets are not files and their recipes must run every time, regardless of any same-named file on disk. The rule of practice is simple: the moment you write a target whose name is a verb (an action) instead of a noun (a file you produce), add it to a single .PHONY: line — it removes an entire class of silent no-op bugs and gives a tiny speedup by skipping the file-existence check.

Can Make replace Airflow or Dagster?

For a small, single-machine, on-demand pipeline — yes, and it often should, because a Makefile is far lighter than an orchestrator. But Make and orchestrators operate at different layers and usually compose rather than compete. A task runner builds a dependency graph now, here, on one machine and gives you incrementality; an orchestrator like Airflow, Dagster, or Prefect owns scheduling, retries with backoff, backfills over date partitions, cross-machine execution, and an observability UI. Make has none of those, so the moment you need a cron-style schedule, SLA alerting, backfills, or distributed execution, you graduate to an orchestrator — and the common senior pattern is to have the orchestrated task simply call make transform or task load, keeping the build logic identical between local debugging and production. Start with a task runner; add an orchestrator only when you actually need scheduling or cross-machine coordination.

How do I make a Makefile reproducible in CI?

Reproducibility means the same command, over the same inputs, with the same toolchain, produces the same output regardless of who runs it or where — and a Makefile is where you wire all three levers together. Keep the CI config a thin wrapper that only checks out code, installs a pinned tool version, syncs lockfile-frozen dependencies, and calls make pipeline — with no build logic in the YAML, local and CI cannot diverge. Inside the Makefile, invoke a pinned interpreter (PYTHON := uv run python), fix randomness (--seed 42 plus export PYTHONHASHSEED := 0), write deterministic outputs (sorted rows, stable compression), and add .DELETE_ON_ERROR: so a failed recipe never leaves a corrupt artifact. Finally, add a check target that verifies the final artifact against a committed sha256 and run it as the CI gate, so a green build proves the pipeline is reproducible, not merely that some script exited 0.

Practice on PipeCode

• Drill the pandas practice library → for the transformation, dependency-graph, and pipeline-glue problems a task runner encodes.
• Rehearse on the ETL practice library → for incremental loads, partitioned rebuilds, and CI-driven pipeline runs.
• Harden your scripts on the defensive-coding practice library → for the safe-fail, checksum, and reproducibility guardrails that keep a pipeline honest.
• Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the Make-versus-Taskfile decision and the incremental-build mental model against real graded inputs.

Lock in reproducible-pipeline muscle memory

Docs explain the syntax. PipeCode drills explain the decision — when a task runner beats a shell script, when mtime lies and checksums win, when `.PHONY` saves you from a silent no-op, and when a `make check` gate is the only thing standing between a green build and a non-reproducible one. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs data engineers actually face.

Practice pandas problems →
Practice ETL problems →