Papermill & Notebook Pipelines: Parametrized, Scheduled, Version-Controlled Notebooks

papermill notebook pipelines are what you get when you stop treating a Jupyter notebook as a throwaway scratchpad and start treating it as a runnable, parametrized job. Papermill is a small open-source tool — a Python library and a command-line program — that takes a notebook, injects a fresh set of parameters into it, runs every cell top to bottom, and writes a new notebook that contains the code, the injected values, and every output the run produced. The template is never mutated; each execution leaves behind its own fully-rendered artifact.

That one move changes the economics of the work an analyst already does in a notebook. Instead of copying a notebook, hand-editing the date at the top, running it, and screenshotting a chart into an email, you run papermill report.ipynb out/2026-09-15.ipynb -p run_date 2026-09-15 and the executed notebook — charts, tables, logs, and all — becomes the deliverable. This guide walks the five things an interviewer will actually probe: the parameters cell tag and papermill.execute_notebook, the executed output notebook as an artifact plus nbconvert, collecting structured results with scrapbook.glue, orchestrating notebooks on a schedule with Airflow's PapermillOperator, and version-controlling notebooks with nbdime. Each section pairs the concept with a Solution-Tail interview answer: code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works — including the honest part, which is when notebooks-as-pipelines is the wrong tool.

When you want hands-on reps immediately after reading, drill the pipeline-design practice library →, rehearse the run-cadence decisions on the scheduling practice set →, and shape the parametrized outputs on the data-transformation practice set →.

---

On this page

• Why notebooks-as-pipelines — Papermill's job and its limits
• The parameters cell tag & execute_notebook
• The executed output notebook as an artifact
• Collecting outputs with scrapbook & glue
• Orchestrating & version-controlling notebook pipelines
• Cheat sheet — Papermill recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Why notebooks-as-pipelines — Papermill's job and its limits

Papermill parametrizes and executes a notebook, producing a second notebook — that one fact decides where it fits

The one-sentence invariant: Papermill runs a notebook the way a function runs a body — you pass parameters in, it executes, and it returns a new notebook with the results baked in. The input notebook is read-only during a run; the value Papermill produces is an output notebook, an ordinary .ipynb in which every cell has been executed and its outputs captured. Everything else — scheduling, storing to S3, rendering to HTML — is built on top of that single behaviour.

What Papermill actually is (and is not).

• A parametrizer. Papermill's headline feature is injecting parameters into a notebook without editing the source. You mark one cell, and Papermill overrides its variables at run time.
• An executor. It runs the notebook cell-by-cell against a Jupyter kernel (python3 by default, but any installed kernel), saving outputs and cell timings as it goes.
• Not a scheduler. Papermill does not have a cron, a DAG, or a UI. You invoke it from a script, a CI job, or an orchestrator. "Scheduled notebooks" means something else triggers papermill.
• Not a transformation framework. Papermill neither knows nor cares what the notebook does; there is no dependency graph between cells beyond top-to-bottom order.

The good-idea case — when the artifact is the point.

• The output notebook is a self-documenting audit log. Inputs, code, logs, tables, and charts live in one file for one run — reproducible and reviewable months later without re-running anything.
• The author is the operator. An analyst who already lives in Jupyter can promote their notebook to a scheduled job with zero rewrite into a "real" script, which is often the difference between a report shipping and not shipping.
• Parametrized fan-out is trivial. Run the same notebook once per region, per date, or per customer by looping over parameter sets — each run is an independent artifact.

The anti-pattern case — when to reach for a script or task instead.

• Hidden execution state. Notebooks invite out-of-order cell runs and lingering variables; a notebook that only works if you run cell 7 before cell 3 is a landmine. Papermill always runs top-to-bottom, which helps, but complex control flow still belongs in tested modules.
• Heavy DAGs and shared logic. If tasks fan out, retry independently, or share a library, a notebook is the wrong container — extract the logic into a package and call it from Airflow tasks or a script.
• Team-edited notebooks. Concurrent edits to a JSON notebook merge badly; long-lived, multi-author pipeline logic wants plain .py files under review.

What interviewers listen for.

• Do you say "Papermill produces a new executed notebook, it does not mutate the input" early? — senior signal.
• Do you separate "Papermill executes" from "something else schedules"? — required framing.
• Can you name both the good case (the artifact is an audit log) and the anti-pattern (hidden state, heavy DAGs) unprompted? — the maturity signal.
• Do you mention that parameters are injected via a tagged cell, not by editing source? — the whole mechanism.

Worked example — five lines that turn a notebook into a job

Detailed explanation. The canonical Papermill "hello world" takes an existing notebook and runs it with one overridden parameter, writing the executed result somewhere new. It looks trivial, and that is the point: the same call that runs a toy report scales unchanged to a nightly job, because Papermill only ever sees "a notebook, some parameters, an output path." No cell is edited by hand; the override is injected.

Question. Run a report.ipynb for a specific date and land the executed notebook at a new path, without touching the original.

Input.

argument	value
input notebook	report.ipynb
output notebook	runs/report-2026-09-15.ipynb
parameter run_date	2026-09-15

Code.

import papermill as pm

pm.execute_notebook(
    "report.ipynb",                       # input template (never modified)
    "runs/report-2026-09-15.ipynb",       # executed output artifact
    parameters={"run_date": "2026-09-15"},
)

Step-by-step explanation. execute_notebook reads report.ipynb, finds the cell tagged parameters, and inserts a new injected-parameters cell directly beneath it that sets run_date = "2026-09-15", overriding the default. It then launches the python3 kernel and runs every cell in order, capturing each cell's outputs. Finally it writes the fully-executed notebook to runs/report-2026-09-15.ipynb. The original report.ipynb is untouched — you could run it a hundred times with a hundred dates and get a hundred artifacts.

Output.

Papermill produced	value
input notebook	report.ipynb (unchanged)
output notebook	runs/report-2026-09-15.ipynb (all cells executed)
injected cell	injected-parameters with run_date = "2026-09-15"
captured	every cell's outputs + per-cell execution time

Rule of thumb. If the work already lives in a notebook and the deliverable is "this notebook, run with today's inputs," Papermill is the smallest possible upgrade from manual to reproducible — one execute_notebook call, no rewrite.

---

2. The parameters cell tag & execute_notebook

One tagged cell is the whole parameter interface — Papermill injects overrides right beneath it

Papermill has exactly one mechanism for parametrizing a notebook, and an interviewer who asks "how does Papermill inject parameters?" wants it precisely. You tag one cell with the cell tag parameters. That cell holds default assignments. At run time Papermill inserts a second, machine-generated cell — labelled injected-parameters — immediately after it, containing the values you passed. Because Python executes top-to-bottom, the injected assignments run after the defaults and win.

The tagging mechanism.

• The parameters tag. In JupyterLab you add it via Property Inspector → Cell Tags → add tag parameters. It is metadata on the cell, not a comment; Papermill looks for exactly this tag.
• Defaults belong in the tagged cell. Assign every parameter a sensible default there (run_date = "2026-01-01", region = "us"). This keeps the notebook runnable standalone in Jupyter with no Papermill involved.
• The injected cell. Papermill writes injected-parameters right below the tagged cell. If no tagged cell exists, Papermill injects at the top and warns — your defaults then never run, a common footgun.

Passing parameters — library and CLI.

• Library. pm.execute_notebook(input, output, parameters={"region": "eu", "limit": 500}) passes a dict; types are preserved (int stays int, list stays list).
• CLI -p (typed). papermill in.ipynb out.ipynb -p limit 500 -p region eu — Papermill infers 500 as an int and eu as a string.
• CLI -r (raw string). -r zip 02139 forces a string, so a zero-padded code is not mangled into the integer 2139.
• CLI -y / -f (YAML / file). -y "region: eu\nskus: [a, b]" passes inline YAML; -f params.yaml reads a file — the clean way to pass nested or list parameters.
• -k kernel. -k python3 (or an R / Scala kernel) picks the kernel to execute against.

Why injection beats editing source.

• The template file is never modified, so it stays clean in version control and safe to run concurrently with different parameter sets.
• Types survive: a dict passed in Python arrives as a dict, not a re-parsed string.
• The injected cell is visible in the output notebook, so anyone reading the artifact sees exactly which values produced it.

Worked example — a parameters cell with defaults, overridden at run time

Detailed explanation. The everyday pattern is a first code cell tagged parameters that declares every input with a default, followed by cells that use those names. Papermill overrides a subset; the rest keep their defaults. This is what lets the same notebook run interactively (defaults) and as a job (overrides).

Question. A notebook's first cell is tagged parameters with region = "us" and limit = 100. You run it with Papermill passing region="eu" only. What values do the downstream cells see?

Input.

region = "us"    # first cell of report.ipynb, tagged "parameters"
limit = 100

Code.

import papermill as pm

pm.execute_notebook(
    "report.ipynb",
    "runs/report-eu.ipynb",
    parameters={"region": "eu"},        # override region; limit keeps its default
)

The CLI form is identical: papermill report.ipynb runs/report-eu.ipynb -p region eu.

Step-by-step explanation. Papermill locates the parameters-tagged cell and inserts injected-parameters right after it containing region = "eu". When the notebook runs, the tagged cell sets region = "us" and limit = 100, then the injected cell immediately reassigns region = "eu". Because limit was not passed, it keeps its default 100. Every downstream cell reads region == "eu" and limit == 100.

Output.

variable	source of final value	value at run time
region	injected-parameters (override)	"eu"
limit	parameters cell (default)	100

Rule of thumb. Give every parameter a default in the tagged cell, then override only what changes per run — the notebook stays runnable by hand and precise as a job.

Papermill interview question on parameter injection

Question. A teammate reports that their Papermill run "ignores the parameters" — the notebook always uses the defaults no matter what -p values they pass. The notebook opens fine and runs top-to-bottom in Jupyter. What is almost certainly wrong, and how would you prove and fix it?

Solution Using the parameters cell tag correctly

Code.

import papermill as pm
import nbformat

nb = nbformat.read("report.ipynb", as_version=4)
tagged = [                                       # prove it: which cells carry the tag?
    idx for idx, c in enumerate(nb.cells)
    if "parameters" in c.get("metadata", {}).get("tags", [])
]
print("cells tagged 'parameters':", tagged)     # [] means none — that's the bug

if not tagged:                                   # fix: tag the defaults cell, then re-run
    nb.cells[0]["metadata"].setdefault("tags", []).append("parameters")
    nbformat.write(nb, "report.ipynb")

pm.execute_notebook(
    "report.ipynb", "runs/report-eu.ipynb",
    parameters={"region": "eu"},
)

Step-by-step trace.

step	condition	Papermill behaviour
1	no cell tagged parameters	injects injected-parameters at top, emits a warning
2	injected cell runs first	later default cell reassigns region = "us", clobbering the override
3	tag the defaults cell	injected cell now lands after defaults
4	re-run with -p region eu	override runs last and wins → region == "eu"

1. Papermill only overrides variables via the injected-parameters cell it inserts right after the parameters-tagged cell.
2. With no tagged cell, Papermill still injects — but at the top — so any later plain cell that reassigns region overwrites the injected value; the defaults appear to "win."
3. The warning No cell tagged 'parameters' in the output notebook is the diagnostic; the empty tagged list is the proof.
4. Tagging the defaults cell places the injected cell after the defaults, so the override is the last assignment and takes effect.

Output:

state	tagged cells	region at run time
before fix	none	"us" (default wins)
after fix	cell 0	"eu" (override wins)

Why this works — concept by concept:

• Cell tag, not comment — Papermill keys off notebook metadata tags: ["parameters"]; a # parameters comment does nothing, which is the single most common Papermill mistake.
• Injection position — overrides are inserted after the tagged cell so they run last; with no tag, injection at the top is beaten by later reassignments.
• Defaults keep it runnable — the tagged cell's defaults let the notebook run in plain Jupyter, while Papermill supplies production values.
• Inspect to diagnose — reading cell.metadata.tags with nbformat turns a vague "it ignores params" into a one-line proof.
• Cost — injection and tag lookup are O(cells), trivial against the runtime of the notebook's actual work.

Pipelines
Topic — pipelines
Parametrized pipeline-design problems

Practice →

Transform Topic — data-transformation Parameter-driven transformation problems

Practice →

---

3. The executed output notebook as an artifact

The output notebook is the deliverable — a fully-run, self-contained record of one execution

The feature that makes Papermill a pipeline tool rather than a runner is that every execution produces a durable, human-readable artifact: the output notebook. The input template stays pristine; the output is the same notebook with every cell run, every output captured, per-cell timing recorded, and the injected parameters visible at the top. Months later you can open it and see exactly what code ran, on what inputs, with what result — no re-execution, no guessing.

What lands in the output notebook.

• Executed cells with outputs. Tables, stdout, logs, and rendered charts (as embedded images) are all stored inline, so the file is portable and complete.
• Execution metadata. Papermill records per-cell start/end times and a run status in the notebook metadata, so you can see which cell was slow or where it failed.
• The injected-parameters cell. The exact values that produced this run are right there in the artifact — the run is self-describing.
• Errors are captured, not hidden. By default a cell that raises stops the run and the traceback is written into the output notebook; the file is saved so you can debug the failure post-mortem.

Where the output can live.

• Path-based backends. The output path can be local, s3://bucket/key.ipynb, gs://bucket/key.ipynb, or Azure Blob — Papermill uses fsspec-style handlers, so the same call writes to object storage with no code change.
• Per-run paths. Templating the date or run id into the output path (runs/report-{date}.ipynb) keeps every execution's artifact instead of overwriting one file.

Turning the artifact into something to share.

• --report-mode. Runs Papermill with input cells hidden in the rendered result, so stakeholders see outputs (charts, tables) without the code.
• jupyter nbconvert. Converts the executed notebook to HTML or PDF: jupyter nbconvert --to html runs/report.ipynb. This is the standard "email a clean report" step — nbconvert renders, Papermill executes; they compose.
• --no-input. An nbconvert flag that strips code cells from the rendered HTML/PDF for a purely visual report.

Worked example — execute to S3, then render to HTML

Detailed explanation. A production pattern is: execute the notebook and land the artifact in object storage keyed by run date, then render a human-friendly HTML from that executed notebook. The execute step preserves the machine-readable record; the nbconvert step produces the shareable document. Neither step touches the template.

Question. Run report.ipynb for 2026-09-15, store the executed notebook to S3 under a dated key, and produce a code-free HTML report from it.

Input.

step	tool	target
execute	papermill	s3://reports/2026-09-15/report.ipynb
render	nbconvert	report-2026-09-15.html (no code)

Code.

import papermill as pm
import subprocess

out_nb = "s3://reports/2026-09-15/report.ipynb"

pm.execute_notebook(
    "report.ipynb", out_nb,
    parameters={"run_date": "2026-09-15"},
)

subprocess.run(                            # render the executed notebook to code-free HTML
    ["jupyter", "nbconvert", "--to", "html", "--no-input",
     out_nb, "--output", "report-2026-09-15"],
    check=True,
)

Step-by-step explanation. execute_notebook runs the parametrized notebook and writes the executed artifact straight to S3 — the s3:// prefix routes the write through the object-store handler, no extra upload code. The executed notebook contains every output. jupyter nbconvert --to html --no-input then reads that executed notebook and emits report-2026-09-15.html containing only the rendered outputs (charts, tables), because --no-input drops the code cells. The template report.ipynb remains unchanged.

Output.

artifact	contents
s3://reports/2026-09-15/report.ipynb	executed notebook: code + outputs + injected params + timings
report-2026-09-15.html	rendered outputs only (no code), ready to email

Rule of thumb. Execute once to a dated, durable path for the audit trail; convert that same artifact with nbconvert for humans — never re-run the notebook just to get a different format.

Papermill interview question on failure artifacts

Question. Your nightly Papermill job failed at cell 12 of 20. A teammate says "the run is gone, we have to reproduce it locally." Are they right? What did Papermill leave behind, and how do you use it?

Solution Using the saved output notebook on failure

Code.

import papermill as pm

try:
    pm.execute_notebook(
        "etl.ipynb",
        "runs/etl-2026-09-15.ipynb",     # saved even if a cell raises
        parameters={"run_date": "2026-09-15"},
    )
except pm.exceptions.PapermillExecutionError as err:
    # The output notebook is already on disk with the traceback in cell 12.
    print("failed cell:", err.cell_index, "->", err.ename)
    raise                                 # let the orchestrator mark the task failed

Step-by-step trace.

step	event	what Papermill does
1	cells 1–11 run	outputs captured into the output notebook in progress
2	cell 12 raises	traceback written into cell 12's output
3	run halts	output notebook saved to runs/etl-2026-09-15.ipynb
4	PapermillExecutionError	raised to the caller with cell_index=12 and error name

1. Papermill executes cells in order, persisting outputs as it goes, so partial progress is not lost.
2. When cell 12 raises, Papermill records the full traceback inside the output notebook rather than discarding it.
3. It then saves the output notebook to the specified path — the failure artifact exists on disk even though the run failed.
4. It re-raises PapermillExecutionError so the orchestrator sees a failed task, while you open the saved notebook to debug the exact failing cell with its inputs intact.

Output:

claim	reality
"the run is gone"	false — output notebook saved with the traceback
where to debug	open runs/etl-2026-09-15.ipynb, jump to cell 12

Why this works — concept by concept:

• Fail-with-artifact — Papermill saves the output notebook even on error, so a failed run is a debuggable record, not a void.
• Traceback in place — the exception is captured in the failing cell's output, so you see the error next to the code and the injected parameters that triggered it.
• PapermillExecutionError — a typed exception carrying cell_index and error name lets orchestration fail cleanly and alert precisely.
• Reproducibility for free — the injected-parameters cell means re-running the exact scenario is copy-paste, no guessing which date failed.
• Cost — persisting outputs incrementally is O(output size); the debugging time it saves is the real payoff.

Pipelines
Topic — pipelines
Artifact-and-observability pipeline problems

Practice →

Transform Topic — data-transformation Output-shaping and reporting problems

Practice →

---

4. Collecting outputs with scrapbook & glue

scrapbook.glue writes named values into the notebook, read_notebook reads them back — data escapes the notebook without a side database

An executed notebook is great for humans, but a pipeline usually needs to pull structured results back out — a row count, a model score, a small DataFrame — so a downstream step can use it. Papermill's original record / read_notebook API is deprecated; the supported way is the companion library scrapbook. The mechanism is symmetric: inside the notebook you sb.glue("name", value) to persist a value, and outside you sb.read_notebook(path).scraps to read it back. The values ("scraps") are stored inside the output notebook itself.

The glue side (inside the notebook).

• sb.glue("rows", 1240). Persists a JSON-serializable value under a name; it is stored in the cell's metadata as a scrap and survives in the output notebook.
• sb.glue("summary", df, encoder="pandas"). Encoders let you glue richer objects (a DataFrame, an Arrow table) that round-trip back to the same type.
• sb.glue("chart", fig, display=True). With display=True the value is also rendered visibly in the notebook, so it is both machine-readable data and a human-visible output.

The read side (downstream code).

• sb.read_notebook(path).scraps. Returns a name → scrap mapping; .scraps["rows"].data is the original value with its type restored.
• sb.read_notebooks(dir).scraps_report() / .papermill_dataframe. Read a whole directory of executed notebooks and assemble one row per run — exactly what you want after a parametrized fan-out.
• Type fidelity. Because scraps carry an encoder tag, a glued DataFrame comes back as a DataFrame, not a re-parsed string.

Why this beats the alternatives.

• No side channel. The result travels with the artifact — you do not need a separate table, file, or return-value plumbing to move a scalar from a notebook to the next step.
• Batch-friendly. After running one notebook per region, a single read_notebooks call collects every region's metrics into a DataFrame for comparison.
• Auditable. The glued value is visible in the same artifact that shows the code and inputs that produced it.

Worked example — glue a metric, read it in the next step

Detailed explanation. The everyday scrapbook pattern is: the notebook computes something and glues the number it wants to expose; an orchestrating script runs the notebook, then reads that number back to decide what to do next. This turns a notebook into a callable step with a return value.

Question. A notebook computes row_count. Glue it, run the notebook with Papermill, then read row_count back in the orchestrator and branch on it.

Input.

import scrapbook as sb        # inside etl.ipynb, after the load cell
row_count = len(df)          # e.g. 1240
sb.glue("row_count", row_count)

Code.

import papermill as pm
import scrapbook as sb

out = "runs/etl-2026-09-15.ipynb"
pm.execute_notebook("etl.ipynb", out, parameters={"run_date": "2026-09-15"})

nb = sb.read_notebook(out)                   # read the glued value back out
rows = nb.scraps["row_count"].data           # -> 1240, as an int

if rows == 0:
    raise ValueError("load produced zero rows — failing the pipeline")
print(f"loaded {rows} rows")

Step-by-step trace.

step	actor	effect
1	notebook	sb.glue("row_count", 1240) stores a scrap in the output notebook
2	papermill	writes the executed notebook (with the scrap) to out
3	orchestrator	sb.read_notebook(out).scraps["row_count"].data → 1240
4	orchestrator	rows != 0, so the pipeline proceeds

1. Inside the notebook, glue serializes row_count and stores it as a named scrap in that cell's metadata.
2. Papermill saves the executed notebook, so the scrap travels inside the artifact — no external write.
3. The orchestrator reads the notebook back and pulls scraps["row_count"].data, recovering the integer 1240 with its type intact.
4. Because the value is a real int, the == 0 guard works directly, and the pipeline branches on genuine notebook output.

Output:

scrap	stored in	recovered value	type
row_count	runs/etl-2026-09-15.ipynb	1240	int

Why this works — concept by concept:

• glue = persist-with-artifact — the value is written into the output notebook, so results and evidence never drift apart.
• scraps carry types — an encoder tag round-trips ints, dicts, and DataFrames back to their original types instead of strings.
• read_notebook as return value — the downstream step treats the notebook like a function that returned data, enabling real control flow (guards, branching).
• No side database — moving a scalar between steps needs no extra table or file, cutting a whole class of plumbing bugs.
• Cost — glue/read are O(value size); for scalars and small frames it is negligible next to the notebook's compute.

Transform
Topic — data-transformation
Extract-and-collect result problems

Practice →

Pipelines Topic — pipelines Step-to-step handoff pipeline problems

Practice →

---

5. Orchestrating & version-controlling notebook pipelines

Something else schedules Papermill, and nbdime tames the diff — the two things that make notebooks production-grade

Papermill executes; it does not schedule and it does not version-control. Making notebook pipelines production-grade means solving those two separately: an orchestrator (usually Airflow) triggers papermill on a cadence with per-run parameters, and a notebook-aware diff tool (nbdime) plus output stripping (nbstripout) make the JSON reviewable in git. Say it in one breath: Airflow's PapermillOperator runs the notebook on a schedule, and nbdime/nbstripout keep the source diffable.

Scheduling with Airflow.

• PapermillOperator. From airflow.providers.papermill, it wraps execute_notebook: you give it input_nb, output_nb, and parameters, and it becomes a task in a DAG.
• Templated per-run paths and dates. Use Jinja templating ({{ ds }}) so each scheduled run writes its own dated output notebook and passes the run date as a parameter — one artifact per run, no overwrites.
• Idempotent parameters. Parametrize by the logical run date, not datetime.now(), so a re-run of the same interval reproduces the same result — the property that makes retries and backfills safe.
• Operator-level retries. Retries, alerts, and SLAs live on the Airflow task, not in the notebook; Papermill just runs the notebook and fails loudly if a cell raises.

Version-controlling notebooks.

• The problem. A .ipynb is JSON containing source, outputs, execution counts, and metadata, so a one-line code change shows up as a huge, unreadable git diff full of base64 image blobs.
• nbstripout. A git filter that strips outputs and execution counts on commit, so the repo stores only source — small, mergeable diffs.
• nbdime. Content-aware diff/merge: nbdiff for the terminal, nbdiff-web for a rendered side-by-side, and nbdime config-git --enable to make git diff / git merge understand notebooks.
• jupytext pairing. Pair each .ipynb with a synced .py "percent" script so pull requests review clean Python while the notebook stays runnable — a common belt-and-suspenders setup.

Worked example — a scheduled PapermillOperator task

Detailed explanation. The standard way to schedule a notebook is a single Airflow task built from PapermillOperator, templated so each daily run injects that day's logical date and writes a dated artifact. The DAG owns cadence and retries; the operator owns the Papermill call.

Question. Define a daily Airflow task that runs etl.ipynb, passing the run's logical date as run_date and writing the executed notebook to a dated path.

Input.

field	value
schedule	daily (@daily)
input notebook	/nbs/etl.ipynb
output notebook	/runs/etl-{{ ds }}.ipynb
parameter	run_date = {{ ds }}

Code.

from airflow import DAG
from airflow.providers.papermill.operators.papermill import PapermillOperator
import pendulum

with DAG(
    dag_id="etl_notebook",
    schedule="@daily",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    catchup=False,
) as dag:
    run_etl = PapermillOperator(
        task_id="run_etl",
        input_nb="/nbs/etl.ipynb",
        output_nb="/runs/etl-{{ ds }}.ipynb",      # dated artifact per run
        parameters={"run_date": "{{ ds }}"},        # logical date, not now()
    )

Step-by-step explanation. Airflow schedules etl_notebook daily. For each run it renders the Jinja templates: {{ ds }} becomes that interval's logical date (e.g. 2026-09-15). PapermillOperator then calls execute_notebook("/nbs/etl.ipynb", "/runs/etl-2026-09-15.ipynb", parameters={"run_date": "2026-09-15"}). Because the date comes from the scheduler's logical date, re-running the task reproduces the same output; retries are safe. Each day leaves its own dated artifact.

Output.

run interval	output notebook	injected run_date
2026-09-15	/runs/etl-2026-09-15.ipynb	2026-09-15
2026-09-16	/runs/etl-2026-09-16.ipynb	2026-09-16

Rule of thumb. Parametrize by the orchestrator's logical run date, never wall-clock now() — that is what makes a scheduled notebook idempotent and backfillable.

Papermill interview question on notebooks in version control

Question. Reviewers refuse to approve notebook PRs because "the diffs are unreadable" — a two-line code change shows thousands of changed lines. The team still wants to keep working in notebooks. What do you set up so notebook changes review cleanly without abandoning Jupyter?

Solution Using nbstripout and nbdime as git integrations

Code.

pip install nbstripout nbdime
nbstripout --install                       # git clean filter: strip outputs on commit
nbdime config-git --enable --global        # make git diff/merge notebook-aware
nbdiff-web notebooks/etl.ipynb             # rendered side-by-side diff in the browser

Step-by-step trace.

step	tool	effect on the repo / review
1	nbstripout --install	commits drop outputs + execution_count → tiny diffs
2	nbdime config-git --enable	git diff shows cell-level source changes, not JSON
3	nbdiff-web	renders a two-column diff a reviewer can actually read
4	merge conflict	nbmerge resolves per-cell instead of per-JSON-line

1. nbstripout installs a git clean filter that removes outputs and execution counts before content is staged, so the stored notebook is mostly source — the base64 image blobs that bloated the diff are gone.
2. nbdime config-git --enable registers nbdime as git's diff and merge driver for .ipynb, so git diff compares notebooks by cell content rather than raw JSON lines.
3. nbdiff-web gives reviewers a rendered, side-by-side view where a two-line change looks like a two-line change.
4. When two branches edit the same notebook, nbmerge reconciles cell-by-cell, avoiding the unresolvable JSON conflicts that make notebook collaboration painful.

Output:

before	after
2-line change → thousands of diff lines	2-line change → 2 diff lines
unmergeable JSON conflicts	cell-level nbmerge resolution

Why this works — concept by concept:

• Strip outputs at the boundary — nbstripout removes the volatile, huge parts (outputs, counts) at commit time, so version control tracks intent, not render noise.
• Content-aware diff — nbdime compares the notebook's cell structure rather than its serialized JSON, turning an unreadable blob diff into a code review.
• Git integration — wiring both tools into git means reviewers get clean diffs with no change to how authors work in Jupyter.
• Mergeable notebooks — nbmerge resolves per cell, removing the "notebooks can't be collaborated on" objection.
• Cost — filters and diffs are O(notebook size) at commit/review time; the throughput cost is trivial next to the review friction they remove.

Scheduling
Topic — scheduling
Scheduled-run and idempotency problems

Practice →

Pipelines Topic — pipelines Orchestration and DAG-design problems

Practice →

---

Cheat sheet — Papermill recipes

Minimal execute.

import papermill as pm
pm.execute_notebook("in.ipynb", "out.ipynb", parameters={"run_date": "2026-09-15"})

CLI parametrized run.

papermill in.ipynb out.ipynb -p region eu -p limit 500 -r zip 02139 -k python3

YAML params file.

papermill in.ipynb out.ipynb -f params.yaml

where params.yaml contains:

region: eu
skus: [a, b, c]

Glue a value and read it back.

import scrapbook as sb                # inside the notebook
sb.glue("row_count", len(df))

nb = sb.read_notebook("out.ipynb")   # downstream: read it back
rows = nb.scraps["row_count"].data

Airflow PapermillOperator task.

from airflow.providers.papermill.operators.papermill import PapermillOperator
PapermillOperator(
    task_id="run_etl",
    input_nb="/nbs/etl.ipynb",
    output_nb="/runs/etl-{{ ds }}.ipynb",
    parameters={"run_date": "{{ ds }}"},
)

Clean notebook diffs in git.

nbstripout --install                 # strip outputs on commit
nbdime config-git --enable --global  # notebook-aware diff/merge
jupyter nbconvert --to html --no-input out.ipynb   # code-free report

Choosing the pattern.

Situation	Use
Run the same notebook per date / region	Papermill parameters + dated output path
Get a scalar / frame back to the caller	scrapbook.glue + read_notebook
Trigger on a schedule with retries	Airflow PapermillOperator
Share outputs without code	nbconvert --no-input (HTML/PDF)
Keep notebook PRs reviewable	nbstripout + nbdime

---

Frequently asked questions

What is Papermill?

Papermill is an open-source tool (a Python library and a CLI) for parametrizing and executing Jupyter notebooks. You pass parameters into a notebook without editing its source, Papermill runs every cell against a kernel, and it writes a new output notebook containing the code, the injected parameters, and every output the run produced. It does not schedule or transform — it executes a notebook and hands you the executed artifact.

How do I parametrize a Jupyter notebook with Papermill?

Tag one cell with the cell tag parameters and put default variable assignments in it. At run time Papermill injects a new injected-parameters cell immediately after the tagged cell, overriding those defaults with the values you pass via pm.execute_notebook(..., parameters={...}) or the CLI -p/-r/-y/-f flags. The defaults keep the notebook runnable by hand; the injected values win because they run last.

What is the output notebook in Papermill?

It is the executed artifact Papermill writes for each run — the input notebook with every cell run, all outputs captured inline (tables, logs, charts), per-cell timings recorded, and the injected-parameters cell visible. The input template is never modified. If a cell raises, Papermill still saves the output notebook with the traceback in place, so a failed run remains fully debuggable.

How do I get values back out of a Papermill notebook?

Use the companion library scrapbook (Papermill's old record/read_notebook API is deprecated). Inside the notebook call sb.glue("name", value) to store a value in the output notebook; downstream, call sb.read_notebook(path).scraps["name"].data to read it back with its type intact. For a batch of runs, sb.read_notebooks(dir) assembles one row per notebook so you can compare metrics across a parametrized fan-out.

How do I schedule Papermill notebooks?

Papermill has no scheduler, so you trigger it from an orchestrator. In Airflow, the PapermillOperator (from airflow.providers.papermill) wraps execute_notebook: give it input_nb, a templated output_nb, and parameters, and template the run's logical date ({{ ds }}) instead of now() so re-runs are idempotent. Retries, alerts, and SLAs live on the Airflow task; Papermill just runs the notebook and fails loudly if a cell errors.

When are notebooks-as-pipelines a bad idea?

When the logic is complex, shared, or heavily branched. Notebooks encourage hidden execution state, they are awkward to unit-test, and their JSON format makes multi-author collaboration and code review painful. Papermill mitigates the run-order problem by always executing top-to-bottom, but for reusable libraries, fan-out DAGs, or team-owned pipeline logic, extract the code into tested .py modules and call them from tasks — keep notebooks for the reporting and exploratory jobs where the rendered artifact is the deliverable.

Practice on PipeCode

Pipecode.ai is Leetcode for Data Engineering — every Papermill idea above, from the parameters cell tag to the executed output artifact, scrapbook glue, and the scheduled PapermillOperator, maps to a hands-on practice room where you design the pipeline against real graded inputs. PipeCode pairs each reading with 450+ DE-focused problems and a real-time scoring engine, so your answer to "how would you make this scheduled notebook idempotent?" holds up under a senior interviewer's depth probes.

Practice pipeline problems now →
Scheduling drills →