Temporal for Data Workflows: Durable Execution, Retries & Long-Running Pipelines

Every data engineer eventually writes the same doomed script: a multi-step job that extracts, transforms, and loads, wrapped in try/except, sprinkled with a status table and a few "resume from step 3" flags — and then a machine reboots mid-run and nobody can say what actually happened. temporal for data workflows is the answer to that failure class. Temporal is a durable execution engine: you write your pipeline as an ordinary function, and the platform guarantees that function runs to completion exactly as written even if the process crashes, the pod is rescheduled, you deploy new code, or the whole worker fleet disappears for an hour. The state of the running program — which step finished, what each step returned, which timer is pending — is not held in fragile memory or a hand-rolled checkpoint table; it is reconstructed, deterministically, from a durable log.

That single idea reshapes how you build reliable pipelines. Instead of gluing together cron, a queue, a state column, and a pile of manual retry logic, you express the whole thing as code and lean on the engine for the hard parts: automatic retries with backoff, durable timers that survive restarts, and the coordination primitives that long-running pipelines need — waiting for a human approval for three days, reacting to an external event, or undoing three completed steps when the fourth fails. This guide walks the model end to end: what durable execution is and why workflows-as-code beat cron and DAG-of-tasks workflow orchestration, the workflow/activity split and the determinism rules that make it all work, the reliability toolkit of retries, timeouts, idempotency, and heartbeats, the long-running toolkit of signals, queries, child workflows, continue-as-new, and the saga pattern, and finally a clear-eyed comparison of when to reach for Temporal versus Airflow. Every section pairs a teaching block with a worked example, and every domain closes on an interview scenario with the trace, the output, and a concept-by-concept breakdown.

When you want hands-on reps alongside the reading, drill pipeline design on the ETL practice library →, rehearse orchestration trade-offs on the system design practice library →, and go deeper with the ETL system design course →.

---

On this page

• Durable execution — why workflows-as-code beat cron and Airflow DAGs
• Workflows, activities & determinism — the split that makes replay work
• Retries, timeouts & idempotency — reliability without a retry table
• Long-running pipelines — signals, timers, child workflows & the saga pattern
• Temporal vs Airflow for data pipelines — when to use each
• Cheat sheet — Temporal decision recipes
• Frequently asked questions
• Practice on PipeCode

---

1. Durable execution — why workflows-as-code beat cron and Airflow DAGs

Durable execution means your program's state survives crashes without a checkpoint table

The one-sentence framing that changes how you build pipelines: durable execution is the guarantee that a function runs to completion exactly once as written — surviving process crashes, deployments, and machine loss — because the engine records every step and its result in an append-only event history and can replay that history to reconstruct the program's exact state. You write normal-looking code; Temporal makes that code durable. There is no status column to keep in sync, no "which step did we die on" forensics, and no bespoke resume logic — the history is the resume logic.

What the event history is, and why it matters. The durability comes from one data structure, not magic.

• Append-only log. Every meaningful thing the workflow does — a task scheduled, an activity completed, a timer started, a signal received — is written as an event to a per-workflow history in Temporal's persistence store before the workflow proceeds.
• Deterministic replay. When a worker picks up a workflow (after a crash, a deploy, or just load-balancing), it re-runs your workflow function from the top, feeding it the recorded history. Because the code is deterministic, it takes the same branches and reaches the exact point it was at — then continues from there.
• No lost work. A completed activity is in the history, so replay does not re-run it; it just hands back the recorded result. That is why a crash halfway through a five-step pipeline resumes at step four, not step one.

The problem durable execution kills. Everything on this list is code you no longer write.

• Hand-rolled state machines — the status IN ('extracted','transformed','loaded') column and the branching that reads it.
• Checkpoint / resume tables — "last processed offset", "last successful step", updated transactionally with the work.
• Manual retry bookkeeping — attempt counters, backoff sleeps, and dead-letter tables scattered across jobs.
• "Where did it die?" forensics — the log-grepping you do at 3 a.m. when a batch half-ran.

How this differs from cron and from Airflow DAGs. Both are orchestration tools, but neither gives you durable code.

• Cron runs a command on a schedule and knows nothing about state, retries, or partial progress. A crash means "run it again from scratch and hope it is idempotent."
• Airflow (and DAG-of-tasks tools) orchestrate tasks wired by dependencies, driven by a central scheduler. State lives between tasks in a metadata DB and in whatever XCom/external store you use; within a task you are back to writing your own reliability logic, and expressing per-entity, long-lived, branching logic as a static DAG is awkward.
• Temporal makes the program itself durable: loops, conditionals, variables, await — ordinary control flow — all survive restarts, so a pipeline that waits three days for approval or retries a flaky partner API is just code.

What durable execution buys you in production. The value shows up in the incidents you no longer have.

• Zero partial-run forensics. A deploy in the middle of a batch is a non-event — in-flight workflows are picked up by the new workers and continue from their history, so "did the 2 a.m. job finish before the rollout?" stops being a question you answer by grepping logs.
• Reliability logic you write once, centrally. Retries, backoff, timeouts, and dead-lettering live in policy on each activity rather than scattered across every job, so a new pipeline inherits the same battle-tested behaviour for free.
• Observability by construction. Because every step is an event, the history is the audit trail — you can see exactly which activities ran, what they returned, and where a run is currently blocked, without adding bespoke logging.
• Testable orchestration. Workflow logic is deterministic code, so you can unit-test the branching and replay recorded histories to catch non-determinism before it reaches production.

The through-line is that durable execution moves reliability from something you bolt onto each pipeline to something the platform guarantees for all of them — which is why teams adopting temporal for data workflows usually delete more code than they add.

The crash-resilient pipeline — a worked teaching example

Detailed explanation. Picture a nightly sync that pulls orders from a partner API, transforms them, and loads them into a warehouse. In a plain script, a crash between "transform" and "load" leaves you with transformed data in memory (gone) and no record of how far you got; the safe move is to re-run everything, which re-hits the API and risks double-loading. As a durable workflow, the same three steps are recorded as they complete, so a crash after "transform" resumes directly at "load" using the already-computed transformed result.

• Fragile version. State is in local variables and process memory; a crash loses it and forces a full restart.
• Durable version. Each step is an activity whose result is written to history; a restart replays the recorded results and continues.
• The payoff. You get exactly-once effect semantics for the pipeline (given idempotent activities) with zero bespoke checkpointing.

Question. A three-step ETL job crashes right after the transform step completes. What does each approach do on restart?

Input.

Step	Plain cron script	Durable workflow
extract	re-runs (re-hits API)	replays recorded result, skips
transform	re-runs	replays recorded result, skips
load	runs	runs (resumes here)
net effect	whole job re-executes	resumes at load only

Code.

# Durable version: the pipeline is one workflow function.
from datetime import timedelta
from temporalio import workflow

@workflow.defn
class DailyOrderSync:
    @workflow.run
    async def run(self, since: str) -> int:
        orders = await workflow.execute_activity(
            fetch_orders, since,
            start_to_close_timeout=timedelta(minutes=5),
        )
        clean = await workflow.execute_activity(
            transform_orders, orders,
            start_to_close_timeout=timedelta(minutes=10),
        )
        written = await workflow.execute_activity(
            load_to_warehouse, clean,
            start_to_close_timeout=timedelta(minutes=15),
        )
        return written

Step-by-step trace.

1. fetch_orders runs; its return value is written to history as an ActivityTaskCompleted event.
2. transform_orders runs; its result is likewise recorded in history.
3. The worker process crashes before load_to_warehouse starts.
4. A worker picks the workflow back up and replays the function: it re-executes the Python line by line, but each execute_activity for a completed step returns the recorded result instead of re-running the activity.
5. Execution reaches load_to_warehouse — the first not-yet-completed step — and runs it for the first time.

Output:

Restart moment	Activities actually re-executed	Data re-fetched?
after extract	transform, load	no
after transform	load only	no
after load	none (workflow already complete)	no

Rule of thumb. If your reliability plan includes a status column and a "resume from step N" branch, that is exactly the code durable execution deletes — model the pipeline as a workflow and let the history be the checkpoint.

Reading an event history — what replay reconstructs

Detailed explanation. The mental model that makes Temporal click is that a workflow is a deterministic function of its event history. The worker holds no long-lived state in memory between tasks; it rebuilds state by replaying history whenever it needs to make progress. Understanding what the history contains — and what it deliberately does not re-do — is the difference between trusting the system and fighting it.

• Command/event pairs. Your workflow code issues commands ("schedule this activity", "start this timer"); the server turns each into recorded events ("activity scheduled", "activity completed", "timer fired").
• Results are cached in history. A completed activity's return value is in the history, so replay hands it straight back — the activity's side effects happen once.
• Pending work is durable too. A 24-hour timer or an unfulfilled wait_condition is an event in history; the workflow can be evicted from memory entirely and re-created when the timer fires.

Question. After a crash, how does a worker know a pipeline had finished "extract" and "transform" but not "load"?

Input.

History event	Meaning
WorkflowExecutionStarted	run began with input since
ActivityTaskCompleted (fetch_orders)	extract result recorded
ActivityTaskCompleted (transform_orders)	transform result recorded
ActivityTaskScheduled (load_to_warehouse)	load requested but not yet completed

Code.

# Conceptual replay loop the worker performs:
state = initial
for event in history:                 # deterministic re-execution of your code
    if event is ActivityTaskCompleted: # feed recorded result back into the awaiting call
        resume_await_with(event.result)
    # timers, signals, child workflows replay the same way
# after history is exhausted -> continue running new code from here

Step-by-step trace.

1. The worker loads the workflow's history from persistence.
2. It re-runs run(); at the first execute_activity it sees a matching ActivityTaskCompleted in history and returns that recorded value instead of calling the API.
3. Same for the second activity — recorded result returned, no re-execution.
4. At the third execute_activity it finds only ActivityTaskScheduled with no completion, so this is the live frontier: the worker issues the command to actually run load_to_warehouse.
5. From here the function executes new code and appends new events.

Output:

Reconstructed fact	Source in history
extract done, value known	ActivityTaskCompleted #1
transform done, value known	ActivityTaskCompleted #2
load not done yet	Scheduled with no Completed
resume point	the load activity

Rule of thumb. Trust the history: never store workflow progress in an external table "for safety" — the history already is the single source of truth, and a parallel table just drifts.

---

2. Workflows, activities & determinism — the split that makes replay work

The whole model rests on one split: workflows orchestrate deterministically, activities do the side effects

The invariant to burn in: workflow code is deterministic orchestration that must replay to an identical result every time, so it may contain only logic whose outcome is fixed by the event history; every interaction with the outside world — HTTP calls, database writes, file reads, random numbers, the wall clock — must be pushed into an activity, whose result is recorded once and replayed thereafter. Get this split right and everything else in Temporal follows; get it wrong and you hit non-determinism errors on the first redeploy.

What lives in a workflow. Deterministic orchestration only.

• Control flow and state — loops, conditionals, local variables, and await on Temporal futures. This is the "shape" of your pipeline.
• Calls into activities and child workflows — workflow.execute_activity(...), workflow.execute_child_workflow(...).
• Deterministic replacements for non-deterministic operations — workflow.now() instead of datetime.now(), workflow.uuid4() instead of uuid.uuid4(), workflow.sleep() / durable timers instead of time.sleep(), and workflow.wait_condition(...) to block on state.

What lives in an activity. Everything with a side effect or non-determinism.

• All IO — reading from and writing to databases, calling REST/gRPC APIs, reading files or object storage, publishing to a queue.
• Anything non-deterministic — random values, current time you actually need from the environment, reading a config that can change.
• Long or heavy work — an activity can run for seconds, minutes, or hours; it heartbeats to prove liveness (Section 3) and is retried independently.

Why the split exists. It exists so replay is safe.

• Replay re-runs workflow code, not activities. If a database write lived in workflow code, every replay would repeat the write; putting it in an activity means the result is recorded once and the write happens once.
• Determinism is a contract, not a suggestion. On replay, if your workflow code takes a different branch or issues commands in a different order than the recorded history, Temporal raises a non-determinism error to protect you from corrupting the run.
• Versioning is how you change deterministic code safely. When you must change a workflow's logic for in-flight runs, you gate the change with workflow.patched("v2") (or the Worker Versioning feature) so old histories replay the old path and new runs take the new one.

Defining a workflow and an activity — a worked teaching example

Detailed explanation. The physical layout mirrors the logical split: activities are plain functions decorated with @activity.defn and are free to do IO; the workflow is a class decorated with @workflow.defn whose @workflow.run method calls those activities. A worker process registers both and polls a task queue for work. The important discipline is that the workflow file imports no IO libraries and performs no side effects directly.

• Activity = capability. "Fetch orders", "load rows" — each is an independently retryable unit of side-effecting work.
• Workflow = choreography. It decides the order, passes results between steps, and sets timeouts and retry policies.
• Worker = host. It runs both and connects them to a task queue.

Question. Write the workflow/activity split for a two-step "fetch then load" sync, keeping all IO out of the workflow.

Input.

Component	Responsibility	May do IO?
fetch_orders activity	call the partner API	yes
load_to_warehouse activity	upsert rows	yes
DailyOrderSync workflow	order the two steps, set timeouts	no

Code.

# activities.py — all side effects live here
from temporalio import activity
import httpx

@activity.defn
async def fetch_orders(since: str) -> list[dict]:
    async with httpx.AsyncClient() as client:
        resp = await client.get(f"https://api.example.com/orders?since={since}")
        resp.raise_for_status()
        return resp.json()

@activity.defn
async def load_to_warehouse(rows: list[dict]) -> int:
    # idempotent upsert keyed on order_id; returns rows written
    return warehouse.merge("orders", rows, key="order_id")

# workflow.py — deterministic orchestration only, no imports that touch IO
from datetime import timedelta
from temporalio import workflow

@workflow.defn
class DailyOrderSync:
    @workflow.run
    async def run(self, since: str) -> int:
        orders = await workflow.execute_activity(
            fetch_orders, since,
            start_to_close_timeout=timedelta(minutes=5),
        )
        return await workflow.execute_activity(
            load_to_warehouse, orders,
            start_to_close_timeout=timedelta(minutes=15),
        )

Step-by-step trace.

1. A client starts DailyOrderSync on a task queue; a worker polling that queue picks up the workflow task.
2. The workflow issues a command to schedule fetch_orders; the worker running the activity makes the HTTP call and returns JSON, which is recorded in history.
3. The workflow resumes, now holding orders, and schedules load_to_warehouse with the fetched rows.
4. The load activity upserts and returns a count; the workflow returns it and completes.
5. Nothing in workflow.py ever opened a socket — every side effect happened inside an activity.

Output:

Event ordering in history	Emitted by
schedule fetch_orders	workflow command
fetch_orders completed (JSON)	activity worker
schedule load_to_warehouse	workflow command
load_to_warehouse completed (count)	activity worker

Rule of thumb. If a line in your workflow file could return a different value on a different day — a clock read, a random number, an API call — it belongs in an activity, not the workflow.

What breaks replay — deterministic vs non-deterministic code

Detailed explanation. The most common beginner bug is smuggling non-determinism into workflow code, where it works fine on the first run and then explodes on the first replay (a crash, a deploy, or a worker restart), because the replayed value no longer matches history. The fix is always the same: replace the non-deterministic call with Temporal's deterministic equivalent, or move the operation into an activity so its result is recorded.

• Forbidden in workflow code — datetime.now() / time.time(), random.*, uuid.uuid4(), reading env vars or mutable config, spawning threads, direct network/DB calls, iterating an unordered set/dict in a way that changes order.
• Deterministic replacements — workflow.now(), a seeded/recorded workflow.random(), workflow.uuid4(), and moving genuine environment reads into an activity.
• The tell — a NonDeterminismError (or a mismatched-command panic) after a redeploy almost always means workflow code branched on something not in history.

Question. Rewrite a workflow that assigns a random batch id and branches on the wall clock so it is replay-safe.

Input.

Non-deterministic line	Why it breaks replay	Deterministic fix
random.randint(...)	new value each replay	workflow.uuid4()
time.time() branch	different clock on replay	workflow.now()
requests.get(...) in workflow	re-runs on every replay	move into an activity

Code.

# BAD — non-deterministic: replay produces different values than the first run
@workflow.defn
class BadWorkflow:
    @workflow.run
    async def run(self) -> str:
        import random, time                       # forbidden inside workflow code
        batch_id = f"batch-{random.randint(0, 9999)}"   # differs on replay
        if time.time() % 2 == 0:                  # wall-clock branch -> mismatch
            return await workflow.execute_activity(
                path_a, batch_id, start_to_close_timeout=timedelta(minutes=1))
        return await workflow.execute_activity(
            path_b, batch_id, start_to_close_timeout=timedelta(minutes=1))

# GOOD — deterministic: Temporal APIs record their result in history
@workflow.defn
class GoodWorkflow:
    @workflow.run
    async def run(self) -> str:
        batch_id = f"batch-{workflow.uuid4()}"    # recorded once, replays identically
        now = workflow.now()                      # deterministic clock from history
        chosen = path_a if now.minute % 2 == 0 else path_b
        return await workflow.execute_activity(
            chosen, batch_id, start_to_close_timeout=timedelta(minutes=1))

Step-by-step trace.

1. BadWorkflow runs once; random.randint returns 4271 and the clock branch takes path_a. History records the path_a command.
2. A redeploy triggers replay; random.randint now returns 118 and time.time() lands on the other branch, so the code wants to schedule path_b.
3. Replayed commands no longer match history (recorded path_a, now issuing path_b) → Temporal raises a non-determinism error and refuses to corrupt the run.
4. GoodWorkflow instead derives batch_id from workflow.uuid4() (recorded in history) and branches on workflow.now() (also from history), so replay reproduces the identical command sequence.

Output:

Workflow	First run	Replay after deploy	Result
BadWorkflow	path_a	wants path_b	non-determinism error
GoodWorkflow	path_a	path_a	replays cleanly

Rule of thumb. Inside a workflow, treat now(), randomness, ids, and sleep as Temporal APIs — never the standard library — and push everything else into an activity.

Interview scenario on the workflow/activity split

You are asked in a system-design round to design a pipeline that, per uploaded file, validates it, calls a third-party enrichment API, and writes results to a warehouse. The interviewer asks where each piece of logic lives in a Temporal design and how you keep it replay-safe across deploys.

Solution Using a thin deterministic workflow over idempotent activities

Answer choices (as an interviewer would frame them).

• A. Put validation, the API call, and the warehouse write all inside the workflow method for simplicity.
• B. Put orchestration in the workflow and each side-effecting step (validate-read, enrich-call, warehouse-write) in its own activity, gating any logic change with workflow.patched.
• C. Put everything in one giant activity and skip the workflow.
• D. Do the API call in the workflow but the DB write in an activity.

Code.

Elimination:
A  IO inside workflow -> re-runs on every replay, non-deterministic     [reject: correctness]
C  one mega-activity  -> loses per-step retries, timeouts, visibility    [reject: no orchestration]
D  API call in workflow -> still non-deterministic IO in workflow code    [reject: correctness]
B  thin workflow + activities per side effect + patched() for changes     [ACCEPT]

Step-by-step trace.

1. Constraint keywords: "per uploaded file" → one workflow per file; "third-party API + warehouse write" → side effects → activities; "replay-safe across deploys" → determinism + versioning.
2. A and D place IO in workflow code, which replays non-deterministically and re-executes side effects — eliminate on correctness.
3. C collapses everything into one activity, throwing away per-step retry policies, per-step timeouts, and step-level history visibility — eliminate; you lose the whole point of orchestration.
4. B keeps the workflow deterministic (it only sequences steps and passes data), isolates each side effect in a retryable activity, and uses workflow.patched("v2") so in-flight runs replay the old logic while new runs use the new path.

Output:

Concern	Where it goes in choice B
validate / read file	activity
enrichment API call	activity
warehouse write	idempotent activity
ordering + retries + timeouts	workflow
safe logic change	workflow.patched gate

Why this works — concept by concept:

• Workflow/activity split — keeping the workflow to pure orchestration is what makes replay deterministic; every side effect sits in an activity whose result is recorded once.
• Per-step activities — one activity per side effect gives you independent retry policies, independent timeouts, and step-level visibility in history — impossible with a single mega-activity.
• Idempotent writes — the warehouse activity may run more than once under retries, so it upserts on a business key rather than blindly inserting.
• Versioning with patched — workflow.patched (or Worker Versioning) lets you evolve deterministic logic without breaking the histories of in-flight runs.
• Cost — the design adds essentially no runtime cost over a script (activities are just your functions) while removing all bespoke checkpoint/retry code; the marginal cost is running the Temporal service, amortised across every workflow.

ETL
Topic — etl
Pipeline orchestration and activity-design problems

Practice →

Design Topic — design Workflow architecture and system-design problems

Practice →

---

3. Retries, timeouts & idempotency — reliability without a retry table

Temporal retries activities for you — which means every activity must be idempotent

The invariant: Temporal automatically retries a failed activity according to its RetryPolicy — exponential backoff, capped attempts, with certain error types marked non-retryable — which is a gift for reliability and a trap for correctness, because "retried" means an activity can execute more than once, so every activity that has a side effect must be idempotent. Reliability and idempotency are two sides of the same coin here; you cannot have automatic retries without designing for at-least-once execution.

The RetryPolicy, field by field. These knobs live on the activity, not the workflow.

• initial_interval — how long to wait before the first retry (e.g. 1s).
• backoff_coefficient — the multiplier between attempts (2.0 doubles the wait each time).
• maximum_interval — a ceiling so backoff does not grow unbounded (e.g. 100s).
• maximum_attempts — the cap; 0 means retry forever, a positive N stops after N tries.
• non_retryable_error_types — error classes that should fail immediately (a ValueError from bad input will never succeed on retry, so do not waste attempts on it).

Activity retries vs workflow retries — a key distinction.

• Activities retry by default. A transient failure (network blip, 503 from a partner) is retried transparently; the workflow just sees the eventual success or a final failure after the cap.
• Workflows do not retry per-step on failure the same way — a workflow "retries" only via replay for infrastructure failures (which is transparent) or via an explicit workflow-level retry policy for the whole run. The everyday retry knob you tune is the activity's.
• Implication. Push flaky operations down into activities so the fine-grained retry policy applies where failures actually happen.

The four activity timeouts — know all four cold. This is a classic interview question.

• schedule-to-start — max time an activity task may sit on the queue before a worker picks it up (detects a starved/undersized worker pool).
• start-to-close — max time a single activity attempt may run before it is considered failed (the one you almost always set).
• schedule-to-close — max time across all attempts, from first schedule to final result (an overall deadline including retries).
• heartbeat — max time allowed between heartbeats from a long-running activity (detects a stuck or dead worker mid-activity).

Idempotency — the correctness half. Because retries mean at-least-once, design every side effect to be safe to repeat.

• Upsert / MERGE on a business key instead of blind INSERT — a retried load overwrites rather than duplicates.
• Dedupe keys — carry a stable id (order_id, batch_key + chunk_id) so the sink can ignore a second delivery.
• Conditional writes — "write only if not already present" (e.g. INSERT ... ON CONFLICT DO NOTHING, a compare-and-set, or an idempotency key on the partner API).

RetryPolicy with backoff and non-retryable errors — a worked teaching example

Detailed explanation. Consider an activity that calls a flaky partner API which returns transient 503s but also occasionally a 400 for malformed input. You want to retry the 503s with exponential backoff up to five attempts, but fail fast on the 400 because no number of retries fixes bad input. This is exactly what RetryPolicy plus non_retryable_error_types expresses.

• Transient errors (timeouts, 5xx) → retry with backoff.
• Permanent errors (4xx bad input, validation failures) → non-retryable, fail immediately.
• The cap → after five attempts, surface the failure to the workflow so it can compensate or alert.

Question. Configure retries so 503s back off over five attempts but a BadRequest fails on the first try.

Input.

Failure	Desired behaviour
HTTP 503 (transient)	retry, exponential backoff, up to 5
HTTP 429 (rate limit)	retry with backoff
BadRequest (HTTP 400)	fail immediately, no retry
final failure after cap	propagate to workflow

Code.

from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

retry = RetryPolicy(
    initial_interval=timedelta(seconds=1),
    backoff_coefficient=2.0,
    maximum_interval=timedelta(seconds=100),
    maximum_attempts=5,
    non_retryable_error_types=["BadRequest", "ValueError"],
)

result = await workflow.execute_activity(
    call_partner_api, payload,
    start_to_close_timeout=timedelta(seconds=30),   # per-attempt deadline
    schedule_to_close_timeout=timedelta(minutes=10),# overall deadline incl. retries
    retry_policy=retry,
)

Step-by-step trace.

1. Attempt 1 fails with 503 → Temporal waits initial_interval = 1s.
2. Attempt 2 fails with 503 → waits 1s × 2.0 = 2s; attempt 3 waits 4s, attempt 4 waits 8s (each capped at maximum_interval).
3. Suppose attempt 3 instead returns a BadRequest; because it is in non_retryable_error_types, Temporal stops immediately and fails the activity — no more retries.
4. If all five attempts are transient failures, the activity fails after the cap and the error propagates to the workflow, which can compensate or alert.

Output:

Scenario	Attempts used	Final state
503 then success on attempt 3	3	success
503 on all attempts	5	fail after cap
BadRequest on attempt 2	2	fail immediately (non-retryable)

Rule of thumb. Retry transient failures with capped exponential backoff and mark input/validation errors non-retryable — retrying a deterministic failure just burns time and rate limits.

Idempotent activity with heartbeats — a worked teaching example

Detailed explanation. A load activity streams a large file into the warehouse in chunks and can take twenty minutes. Two things must be true: it must heartbeat so Temporal can tell a stuck worker from a slow one (and can retry on a different worker if this one dies), and its writes must be idempotent so a retry that re-runs some chunks does not double-insert. Heartbeats also carry progress, so a retried attempt can resume from the last reported chunk instead of restarting.

• Heartbeat = liveness + progress. activity.heartbeat(i) both proves the worker is alive and records the last completed chunk index.
• Heartbeat timeout — set it, so a dead worker is detected within seconds and the activity is retried elsewhere.
• Idempotent chunk writes — MERGE on (batch_key, chunk_id) so re-processing a chunk overwrites rather than duplicates.

Question. Write a long load activity that heartbeats progress, resumes on retry, and never double-writes a chunk.

Input.

Requirement	Mechanism
detect a dead worker mid-load	heartbeat + heartbeat_timeout
resume instead of restart	read last heartbeat detail
no duplicate rows on retry	MERGE on (batch_key, chunk_id)
cap total time	schedule_to_close_timeout

Code.

from temporalio import activity

@activity.defn
async def load_large_file(uri: str, batch_key: str) -> int:
    # resume from the last chunk this activity reported, if it was retried
    start = 0
    info = activity.info()
    if info.heartbeat_details:
        start = info.heartbeat_details[0] + 1

    total = 0
    for chunk_id, chunk in enumerate(read_chunks(uri)):
        if chunk_id < start:
            continue                              # already loaded on a prior attempt
        warehouse.merge(                          # idempotent upsert -> safe to repeat
            "events", chunk, key=["batch_key", "chunk_id"],
        )
        total += len(chunk)
        activity.heartbeat(chunk_id)              # liveness + progress checkpoint
    return total

The workflow sets the matching timeouts:

await workflow.execute_activity(
    load_large_file, uri, batch_key,
    start_to_close_timeout=timedelta(minutes=30),
    heartbeat_timeout=timedelta(seconds=30),
    retry_policy=RetryPolicy(maximum_attempts=5),
)

Step-by-step trace.

1. The activity starts on worker A, loading chunks 0…120 and heartbeating each chunk_id.
2. Worker A crashes after chunk 120; because no heartbeat arrives within heartbeat_timeout (30s), Temporal marks the attempt failed and schedules a retry.
3. Worker B picks up attempt 2; activity.info().heartbeat_details reports 120, so it sets start = 121 and skips the already-loaded chunks.
4. Even if a chunk near the boundary is re-processed, the MERGE on (batch_key, chunk_id) overwrites the same row instead of inserting a duplicate.
5. The activity finishes and returns the total rows written.

Output:

Attempt	Worker	Chunks processed	Duplicates written
1	A (crashes)	0–120	0
2	B (resumes)	121–end	0 (MERGE dedupes)

Rule of thumb. Any activity that runs longer than a few seconds should heartbeat, and any activity that writes should be idempotent — the two together turn a mid-load crash into a resume instead of a restart or a double-load.

Interview scenario on retries and idempotency

You are told a nightly load occasionally double-counts revenue because a transient warehouse timeout causes the loader to re-run and re-insert rows. The interviewer asks how Temporal's retries could make this worse and how you would design the activity so retries are safe.

Solution Using capped retries plus an idempotent MERGE on a business key

Answer choices (as an interviewer would frame them).

• A. Turn off retries entirely so nothing ever re-runs.
• B. Keep automatic retries with backoff, but make the load activity idempotent via MERGE on order_id and carry a stable dedupe key.
• C. Add a global lock so only one loader can ever run.
• D. Retry forever with no cap until it eventually succeeds.

Code.

Elimination:
A  no retries     -> a transient blip now fails the whole load (fragile)   [reject: reliability]
C  global lock    -> serialises everything, kills throughput, doesn't dedupe [reject: wrong fix]
D  infinite retry -> a poison batch retries forever, no idempotency either   [reject: correctness]
B  capped retries + idempotent MERGE on business key                         [ACCEPT]

Step-by-step trace.

1. Constraint keywords: "double-counts on re-run" → non-idempotent write; "transient timeout" → retries are correct, the write is not.
2. A removes retries, so a normal transient failure now fails the pipeline — trading a duplicate bug for a fragility bug — eliminate.
3. C serialises loaders behind a lock, tanking throughput and still double-inserting if one loader retries — eliminate; it does not address idempotency.
4. D retries forever with the same non-idempotent write — the duplicates persist and a poison batch never gives up — eliminate.
5. B keeps capped exponential-backoff retries (so transient failures self-heal) and makes the write idempotent by MERGE-ing on order_id, so a re-run overwrites the same rows rather than adding new ones.

Output:

Concern	Result under choice B
transient timeout	retried transparently with backoff
re-run of loader	MERGE overwrites, no duplicates
poison batch	fails after maximum_attempts, alerts
revenue count	correct (exactly-once effect)

Why this works — concept by concept:

• At-least-once execution — Temporal guarantees an activity runs at least once, never exactly once, so the write must be safe to repeat by construction.
• Idempotent MERGE — upserting on a business key (order_id) turns a second execution into a no-op overwrite, which is what actually removes the double-count.
• Capped backoff — retrying transient failures with exponential backoff and a cap heals blips without hammering a struggling warehouse or looping forever.
• Non-retryable errors — a malformed batch is marked non-retryable so it fails fast to a dead-letter path instead of consuming all attempts.
• Cost — idempotency is a design choice with near-zero runtime cost (a MERGE instead of an INSERT), while the retries you get for free replace an entire hand-built retry/dead-letter subsystem.

ETL
Topic — etl
Retry-safe, idempotent pipeline problems

Practice →

Validation Topic — data-validation Deduplication and data-validation problems

Practice →

---

4. Long-running pipelines — signals, timers, child workflows & the saga pattern

Long-running work needs to receive input, expose state, wait for days, fan out, and undo — Temporal has a primitive for each

The invariant: a long-running workflow is a durable program that can live for days, weeks, or forever, so Temporal gives it primitives to interact with the world without breaking determinism — signals push data in, queries read state out, durable timers wait real time across restarts, child workflows fan work out, continue-as-new keeps history bounded, and the saga pattern undoes completed steps when a later step fails. These are the tools that separate Temporal from a batch scheduler: the workflow is alive and reactive, not a fire-once DAG.

Signals — external input into a running workflow.

• A signal is an asynchronous message sent to a specific running workflow (by id) that mutates its state — "approval granted", "new data available", "cancel".
• Signal handlers are @workflow.signal methods; they update fields the run method is waiting on.
• Signals are recorded in history, so a signal received before a crash is replayed after it.

Queries — read state without changing it.

• A query is a synchronous, read-only call that returns a workflow's current state ("what stage is order 42 in?") without mutating history.
• Query handlers are @workflow.query methods and must not mutate state or call activities.
• Queries are how dashboards and APIs inspect an in-flight pipeline.

Durable timers — waiting that survives restarts.

• workflow.sleep(duration) and workflow.wait_condition(pred, timeout=...) create durable timers: the wait is an event in history, so a workflow can sleep for 30 days across countless deploys and wake exactly on time.
• This is what makes "wait 3 days for approval, then escalate" trivial — no external scheduler, no polling job.

Child workflows — fan-out and modularity.

• A workflow can start child workflows (one per file, per partition, per tenant) and await them, giving you parallelism and independent retry/timeout scopes per child.
• Children have their own histories, keeping the parent's history small.

continue-as-new — bounding infinite histories.

• Event history is not infinite; a workflow that loops forever (or millions of iterations) would grow an unbounded history.
• continue_as_new atomically completes the current run and starts a fresh run with the same id and carried-forward state — resetting history to near-empty while preserving continuity. Use it for long polling loops, entity workflows, and cron-like perpetual workflows.

Saga pattern — distributed transactions without 2PC.

• Many data pipelines touch several systems with no shared transaction (reserve inventory, charge payment, ship). If a later step fails, you must compensate the earlier ones (release inventory, refund).
• The saga pattern records a compensation for each completed step and, on failure, runs the compensations in reverse order — Temporal makes this a natural try/except around a growing list.

Signals, queries and a durable timer — a worked teaching example

Detailed explanation. A dataset-publishing pipeline must wait for a human to approve a dataset before it goes live, but auto-reject if nobody approves within 24 hours; meanwhile a dashboard needs to show the current status. That is a signal (approve), a query (status), and a durable timer (24-hour deadline) in one workflow — and all three survive worker restarts because they are backed by history.

• Signal approve flips an internal flag.
• Query get_status returns the current stage for the dashboard.
• Durable timer via wait_condition(..., timeout=24h) bounds the wait.

Question. Model "wait up to 24h for approval, else auto-reject; expose status the whole time."

Input.

Interaction	Temporal primitive	Mutates state?
human approves	signal approve	yes
dashboard reads stage	query get_status	no
24h deadline	durable timer (wait_condition timeout)	n/a
publish on approval	activity	via activity

Code.

from datetime import timedelta
from temporalio import workflow

@workflow.defn
class DatasetPublish:
    approved: bool = False
    status: str = "created"

    @workflow.run
    async def run(self, dataset: str) -> str:
        self.status = "awaiting-approval"
        try:
            # durable timer: block until approved OR 24h elapses
            await workflow.wait_condition(
                lambda: self.approved, timeout=timedelta(hours=24)
            )
        except TimeoutError:
            self.status = "rejected-timeout"
            return self.status

        self.status = "publishing"
        await workflow.execute_activity(
            publish_dataset, dataset,
            start_to_close_timeout=timedelta(minutes=10),
        )
        self.status = "published"
        return self.status

    @workflow.signal
    async def approve(self) -> None:
        self.approved = True

    @workflow.query
    def get_status(self) -> str:
        return self.status

Step-by-step trace.

1. The workflow starts, sets status = "awaiting-approval", and blocks on wait_condition with a 24-hour durable timer.
2. A dashboard calls the get_status query at any time and gets "awaiting-approval" without disturbing the run.
3. If a reviewer sends the approve signal within 24h, self.approved flips, wait_condition unblocks, and the workflow publishes the dataset.
4. If nobody approves, the 24-hour timer fires, wait_condition raises TimeoutError, and the workflow sets "rejected-timeout" and returns.
5. Every state transition and the pending timer are in history, so a worker restart mid-wait resumes exactly where it left off.

Output:

Event	status after
workflow start	awaiting-approval
approve signal at 3h	publishing → published
no signal, 24h timer fires	rejected-timeout

Rule of thumb. Reach for a signal when the world needs to push data into a running workflow, a query when something needs to read state out, and a durable timer whenever you would otherwise write a polling job or an external scheduler.

Saga with compensations and continue-as-new — a worked teaching example

Detailed explanation. An order-fulfilment pipeline reserves inventory, charges payment, then ships — three side effects across three systems with no shared transaction. If shipping fails, you must refund the payment and release the inventory. The saga pattern records a compensation after each successful step and runs them in reverse on failure. Separately, a perpetual per-customer aggregator that never "ends" uses continue_as_new to keep its history bounded.

• Saga = forward steps + a stack of compensations, unwound on error.
• Reverse order — compensate the most recent successful step first.
• continue-as-new — reset history on a long/infinite loop while carrying state forward.

Question. Implement the reserve → charge → ship saga so a shipping failure cleanly undoes the charge and the reservation.

Input.

Forward step	Compensation
reserve_inventory	release_inventory
charge_payment	refund_payment
ship_order	(final step — nothing after it)

Code.

from datetime import timedelta
from temporalio import workflow

@workflow.defn
class OrderSaga:
    @workflow.run
    async def run(self, order: dict) -> str:
        compensations = []                         # stack of (activity, arg)
        opts = dict(start_to_close_timeout=timedelta(seconds=30))
        try:
            await workflow.execute_activity(reserve_inventory, order, **opts)
            compensations.append((release_inventory, order))

            await workflow.execute_activity(charge_payment, order, **opts)
            compensations.append((refund_payment, order))

            await workflow.execute_activity(ship_order, order, **opts)
            return "shipped"
        except Exception:
            for activity_fn, arg in reversed(compensations):   # unwind in reverse
                await workflow.execute_activity(activity_fn, arg, **opts)
            return "compensated"

And a perpetual aggregator that bounds its history:

@workflow.defn
class CustomerAggregator:
    @workflow.run
    async def run(self, cursor: str, processed: int = 0) -> None:
        for _ in range(1000):                      # bounded batch of iterations
            batch = await workflow.execute_activity(
                pull_events, cursor,
                start_to_close_timeout=timedelta(minutes=2),
            )
            cursor = batch["next_cursor"]
            processed += batch["count"]
        # history is long now -> start a fresh run, carry state forward
        workflow.continue_as_new(args=[cursor, processed])

Step-by-step trace.

1. reserve_inventory succeeds → push release_inventory onto the compensation stack.
2. charge_payment succeeds → push refund_payment onto the stack.
3. ship_order raises (carrier API down after all retries) → control enters except.
4. The workflow runs compensations in reverse: refund_payment first, then release_inventory, leaving the systems consistent.
5. Separately, the aggregator processes 1,000 batches then calls continue_as_new, which completes this run and starts a new one with the same id and the carried-forward cursor and processed — history resets to near-empty.

Output:

Failure point	Compensations run (in order)	Final state
ship fails	refund_payment, release_inventory	compensated
all succeed	none	shipped
aggregator after 1000 batches	n/a	continue-as-new, bounded history

Rule of thumb. Whenever a pipeline mutates multiple external systems without a shared transaction, build a saga — record a compensation as you complete each step and unwind in reverse — and reach for continue-as-new the moment a workflow loops without a natural end.

Interview scenario on long-running orchestration

You are designing an onboarding pipeline: for each new tenant it provisions resources, waits up to 7 days for the customer to upload their first dataset, then runs a validation-and-load job; if the upload never comes, it must clean up the provisioned resources. The interviewer asks how you would structure it in Temporal.

Solution Using a durable timer plus a saga for cleanup

Answer choices (as an interviewer would frame them).

• A. A cron job that polls a "tenant status" table every hour to check for uploads and timeouts.
• B. One workflow per tenant: provision (with a recorded compensation), wait_condition up to 7 days for an upload signal, then load — and on timeout run the compensation to de-provision.
• C. Keep the whole thing in one activity that sleeps for 7 days.
• D. Fire three separate cron jobs (provision, check-upload, cleanup) coordinated by a status column.

Code.

Elimination:
A  hourly poller  -> external scheduler + status table, the thing we're replacing  [reject: fragile]
C  activity sleeps 7 days -> activities aren't durable timers; a crash loses it     [reject: wrong tool]
D  three crons + status column -> hand-rolled state machine, race conditions        [reject: fragile]
B  per-tenant workflow: durable timer + signal + saga compensation                  [ACCEPT]

Step-by-step trace.

1. Constraint keywords: "per tenant" → one workflow per tenant; "wait up to 7 days" → durable timer; "clean up if no upload" → compensation/saga.
2. A rebuilds an external poller and status table — precisely the fragile pattern durable execution removes — eliminate.
3. C sleeps inside an activity, but activities are not durable timers and have start-to-close limits; a worker crash loses the wait — eliminate; sleeping belongs in the workflow via a durable timer.
4. D splits the logic across cron jobs coordinated by a status column, reintroducing races and manual state — eliminate.
5. B models each tenant as a workflow that provisions (pushing a de-provision compensation), waits on a 7-day wait_condition for the upload signal, loads on arrival, and on timeout runs the compensation to tear down resources — all durable across restarts.

Output:

Event	Workflow action
tenant created	provision + record compensation
upload signal within 7d	run validate-and-load
no upload, 7d timer fires	run de-provision compensation
worker restarts mid-wait	resumes timer from history

Why this works — concept by concept:

• Per-entity workflow — one workflow instance per tenant gives each its own durable state, timer, and history, which a shared cron/status-table design cannot cleanly provide.
• Durable timer — the 7-day wait is an event in history, so it survives any number of deploys and worker restarts and fires exactly once on time.
• Signal-driven progress — the upload event is a signal into the live workflow, replacing a polling loop with a push.
• Saga compensation — recording a de-provision step as the compensation for provisioning guarantees clean teardown on timeout without a separate cleanup job.
• Cost — a waiting workflow consumes essentially no compute (it is just history plus a pending timer), so millions of long-lived per-tenant workflows are cheap compared with a fleet of always-on pollers.

Events
Topic — event-processing
Event-driven and signal-based workflow problems

Practice →

ETL Topic — etl Long-running, multi-step pipeline problems

Practice →

---

5. Temporal vs Airflow for data pipelines — when to use each

The real distinction is task orchestrator vs durable-execution engine — pick by the shape of the work

The invariant: Airflow (and DAG-of-tasks schedulers) and Temporal solve overlapping but different problems — Airflow is a scheduler that runs directed acyclic graphs of tasks on a timetable and shines at batch ELT, DAG-of-SQL, and backfills; Temporal is a durable-execution engine that runs long-lived, stateful, reactive code and shines at per-entity workflows, human-in-the-loop waits, sagas, and sub-second reactions — so the right question is never "which is better" but "is this work a scheduled DAG or a durable program?" Senior engineers reach for both, in the same platform, for different jobs.

Where Airflow (task schedulers) wins.

• Scheduled batch ELT — "run this DAG at 2 a.m." is Airflow's native shape; time-based scheduling and calendar logic are first-class.
• DAG-of-SQL / transform pipelines — dbt-style transformation graphs with clear task dependencies map cleanly to operators.
• Backfills and reprocessing — Airflow's date-partitioned runs and backfill tooling are built for "re-run last month".
• Data-team ergonomics — a large operator ecosystem, a mature UI for run history, and a model analysts already understand.

Where Temporal wins.

• Long-running / stateful — anything that waits hours or days (approvals, SLAs, retries with long backoff) is trivial with durable timers and awkward as a DAG.
• Per-entity workflows — one workflow per order/tenant/user, each with its own state and lifecycle, rather than one giant DAG for all of them.
• Human-in-the-loop and reactive — signals let a running pipeline react to external events and sub-second inputs.
• Sagas / distributed transactions — compensation logic across multiple systems is native.
• Complex control flow — loops, conditionals, and dynamic branching are just code, not a statically-declared graph.

Temporal's operational model — know this for interviews.

• Temporal Server is a set of services — Frontend (API gateway), History (owns workflow histories and drives progress), Matching (dispatches tasks to workers via task queues), and internal Worker service (system workflows) — backed by a persistence store (Cassandra, PostgreSQL, or MySQL) and, optionally, Elasticsearch for advanced visibility.
• Your workers are your processes that host workflow and activity code; they poll task queues and execute tasks. They are stateless and scale horizontally, independently of the server.
• The separation that matters — the server durably stores state and coordinates; your workers do the compute. You can deploy new worker code without losing in-flight workflows.
• Self-host or Temporal Cloud — you can run the open-source server yourself or use the managed Temporal Cloud; the SDK and programming model are identical either way.

Interview signals — what a strong answer sounds like.

• Frames the choice as scheduled DAG vs durable program, not "old vs new".
• Names durable execution, event history, and determinism as the core of Temporal — not just "it does retries".
• Knows activities must be idempotent because retries are at-least-once.
• Can place signals/queries/timers/saga against concrete needs.

Mapping an Airflow DAG to a Temporal workflow — a worked teaching example

Detailed explanation. A linear extract → transform → load Airflow DAG scheduled nightly maps almost one-to-one onto a Temporal workflow, with two upgrades: the dependency wiring becomes ordinary sequential code, and each task's ad-hoc retry config becomes a per-activity RetryPolicy. The scheduling itself (run nightly) becomes a Temporal Schedule or a cron workflow.

• DAG edges → sequential awaits in the workflow.
• Operator retries → per-activity RetryPolicy.
• DAG schedule → Temporal Schedule / cron.

Question. Translate a nightly linear ETL DAG into a Temporal workflow.

Input.

Airflow concept	Temporal equivalent
extract >> transform >> load	sequential await execute_activity
operator retries=3	activity RetryPolicy(maximum_attempts=3)
schedule="0 2 * * *"	Temporal Schedule / cron workflow
XCom to pass data	function return values

Code.

# Airflow: a DAG of tasks wired by dependencies, run on a schedule
with DAG("daily_etl", schedule="0 2 * * *") as dag:
    extract   = PythonOperator(task_id="extract",   python_callable=extract_fn)
    transform = PythonOperator(task_id="transform", python_callable=transform_fn)
    load      = PythonOperator(task_id="load",       python_callable=load_fn)
    extract >> transform >> load

# Temporal: the same pipeline as one deterministic workflow function
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

@workflow.defn
class DailyETL:
    @workflow.run
    async def run(self, run_date: str) -> dict:
        raw = await workflow.execute_activity(
            extract, run_date,
            start_to_close_timeout=timedelta(minutes=10),
            retry_policy=RetryPolicy(maximum_attempts=3))
        clean = await workflow.execute_activity(
            transform, raw,
            start_to_close_timeout=timedelta(minutes=20))
        return await workflow.execute_activity(
            load, clean,
            start_to_close_timeout=timedelta(minutes=15))

Step-by-step trace.

1. The DAG's >> edges become plain sequential awaits — the dependency order is the code order.
2. XCom hand-off (extract's output feeding transform) becomes ordinary return values passed between activities.
3. The operator-level retries=3 becomes RetryPolicy(maximum_attempts=3) on the extract activity.
4. The schedule="0 2 * * *" moves to a Temporal Schedule that starts DailyETL nightly.
5. The upgrade: if the process crashes after transform, Temporal replays and resumes at load, whereas the Airflow task would re-run from its own retry state and depends on your idempotency.

Output:

Property	Airflow DAG	Temporal workflow
step wiring	>> operators	sequential await
data passing	XCom	return values
mid-run crash	task-level retry	replay resumes at next step
scheduling	DAG schedule	Temporal Schedule

Rule of thumb. A simple scheduled linear ELT is comfortable in either tool; the moment steps need to wait for external events, branch dynamically, or run per-entity for days, the workflow-as-code model pulls decisively ahead.

The decision matrix applied — a worked teaching example

Detailed explanation. The fastest way to choose is to score the workload on a few axes: is it scheduled or reactive, batch or per-entity, short or long-lived, and does it need cross-system compensation? A "scheduled, batch, short, no-compensation" job leans scheduler; anything "reactive, per-entity, long-lived, or saga" leans Temporal. Many platforms run both — Airflow for the nightly warehouse builds, Temporal for the order/onboarding/streaming-reaction workflows.

• Scheduled + batch + short → task scheduler.
• Reactive or per-entity or long-lived or saga → Temporal.
• Mixed estate → use each where it fits; they are not mutually exclusive.

Question. For each pipeline, pick scheduler or Temporal and say why.

Input.

Pipeline	Shape	Best fit
Nightly warehouse rebuild (DAG of SQL)	scheduled, batch	scheduler
Per-order fulfilment with refunds	per-entity, saga	Temporal
Wait 3 days for KYC approval then load	long-lived, human-in-loop	Temporal
Monthly backfill of a partitioned table	scheduled, backfill	scheduler
React to each Kafka event with a stateful multi-step process	reactive, per-entity	Temporal

Code.

# One-line decision rule:
if scheduled and batch and short_lived and no_compensation:
    use a task scheduler (Airflow-style DAG)
elif reactive or per_entity or long_lived or needs_saga:
    use Temporal (durable execution)
# and it is normal to run both in the same platform for different jobs

Step-by-step trace.

1. The nightly rebuild is time-triggered and stateless between runs → scheduler.
2. Per-order fulfilment holds per-order state and needs compensations on failure → Temporal saga.
3. The 3-day KYC wait is long-lived and human-in-the-loop → Temporal durable timer + signal.
4. The monthly backfill is a date-partitioned batch re-run → scheduler's backfill tooling.
5. The per-event stateful reaction is reactive and per-entity → Temporal, one workflow per key.

Output:

Pipeline	Verdict
nightly warehouse rebuild	scheduler
per-order fulfilment	Temporal
3-day KYC wait	Temporal
monthly backfill	scheduler
per-event stateful reaction	Temporal

Rule of thumb. Score the workload on scheduled-vs-reactive, batch-vs-per-entity, and short-vs-long-lived; if any answer is "reactive, per-entity, long-lived, or saga," that is a durable-execution job.

Versioning a running workflow safely — a worked teaching example

Detailed explanation. Because Temporal replays a workflow's full history to rebuild state, changing the workflow code while executions are mid-flight is dangerous: an in-progress workflow started on the old code path must still replay deterministically, or it will hit a non-determinism error. The fix is explicit versioning — the workflow asks the runtime "which version am I?" and branches, so old histories keep taking the old path while new executions take the new one. This is the durable-execution analogue of a backwards-compatible migration, and interviewers probe whether you know you cannot just edit-and-redeploy a workflow the way you would a stateless service.

• The hazard — reordering, adding, or removing activity calls changes the command sequence and breaks replay of in-flight runs.
• patched / get_version — a guarded branch that returns the old path for pre-existing histories and the new path for fresh ones.
• Replay tests — run new code against captured production histories in CI to catch non-determinism before deploy.
• Retire the branch — once no old-version histories remain, drop the guard.

Question. You must insert a new validate() activity before an existing charge() activity in a workflow that has thousands of in-flight executions. How do you deploy without breaking them?

Input.

Concern	Mechanism
In-flight runs must keep replaying	version guard (old path)
New runs get the new step	version guard (new path)
Catch breakage pre-deploy	replay test vs prod histories
Eventually simplify	remove guard when old runs drain

Code.

# Python SDK: guard the new step so old histories stay deterministic
@workflow.defn
class PaymentWorkflow:
    @workflow.run
    async def run(self, order):
        v = workflow.patched("add-validate-step")   # True for new runs, False when replaying old history
        if v:
            await workflow.execute_activity(validate, order, start_to_close_timeout=timedelta(seconds=30))
        await workflow.execute_activity(charge, order, start_to_close_timeout=timedelta(seconds=30))
        # After all old-version runs complete, replace with workflow.deprecate_patch(...) then delete the guard.

Step-by-step trace.

1. An execution already past charge() replays with patched returning False for the "add-validate-step" marker, so it never expects the validate command — replay stays deterministic.
2. A brand-new execution gets True, runs validate() then charge().
3. A CI replay test runs the new code against a sample of captured production histories; any non-determinism fails the build before deploy.
4. Once metrics show no old-version executions remain, the guard is deprecated and then deleted, collapsing back to a single clean path.

Output:

Execution started on	patched returns	Path taken
old code (in-flight)	False	charge only (deterministic replay)
new code	True	validate → charge

Rule of thumb. Never edit a workflow's activity sequence in place — guard the change with a version marker, prove it against real histories with a replay test, and remove the guard only after old runs drain.

Interview scenario on choosing the tool

An interviewer describes a payments pipeline: each transaction must be authorised, risk-checked (which can take up to an hour and may need a manual review), then settled, with a full refund path if settlement fails — thousands of concurrent transactions, each with its own state. They ask whether you would build this on Airflow or Temporal, and why.

Solution Using Temporal for a per-transaction durable workflow

Answer choices (as an interviewer would frame them).

• A. An Airflow DAG per transaction, dynamically generated, scheduled immediately.
• B. A Temporal workflow per transaction: authorise → risk-check (durable wait + optional manual-review signal) → settle, with saga compensations for refunds.
• C. A single cron job that scans a transactions table every minute and advances states.
• D. A stream processor that reacts to events but stores no per-transaction state.

Code.

Elimination:
A  DAG-per-transaction -> Airflow isn't built for thousands of tiny per-entity DAGs
                          with hour-long human waits                                 [reject: wrong shape]
C  minute cron scan     -> hand-rolled state machine, races, no durable waits         [reject: fragile]
D  stateless stream      -> loses per-transaction state and compensation logic         [reject: no state]
B  Temporal workflow per transaction: durable wait + signal + saga                    [ACCEPT]

Step-by-step trace.

1. Constraint keywords: "each transaction its own state" → per-entity workflow; "up to an hour + manual review" → durable timer + signal; "refund if settlement fails" → saga; "thousands concurrent" → cheap waiting workflows.
2. A tries to model thousands of per-entity, hour-long, human-gated flows as scheduled DAGs — the wrong shape for Airflow — eliminate.
3. C rebuilds a polling state machine over a table, reintroducing the races and forensics durable execution removes — eliminate.
4. D is stateless, so it cannot hold per-transaction progress or run compensations — eliminate.
5. B runs one workflow per transaction: authorise as an activity, risk-check as a durable wait that a manual-review signal can resolve, settle as an activity, and a saga that refunds on settlement failure — all durable, all cheap while waiting.

Output:

Requirement	Temporal mechanism
per-transaction state	one workflow per transaction
up-to-1h risk check + manual review	durable timer + signal
settle	idempotent activity
refund on failure	saga compensation
thousands concurrent	cheap waiting workflows

Why this works — concept by concept:

• Durable execution engine — the payments flow is long-lived, stateful, and reactive, which is exactly the shape durable execution is built for and exactly the shape a batch scheduler is not.
• Per-entity workflows — modelling each transaction as its own workflow gives isolated state and lifecycle, so one stuck review never blocks the others.
• Durable timer + signal — the hour-long risk check waits with no compute cost and resolves either automatically or via a manual-review signal.
• Saga compensation — the refund path is a compensation for the settlement step, giving cross-system consistency without a distributed transaction.
• Cost — thousands of waiting workflows cost almost nothing (state plus pending timers), and workers scale independently of the server, so throughput scales by adding stateless worker pods.

Design
Topic — design
Orchestration and durable-execution design problems

Practice →

Course
Course — ETL system design
ETL System Design for Data Engineering Interviews

Practice →

---

Cheat sheet — Temporal decision recipes

Concept → Temporal primitive (memorise this table).

You need to…	Temporal primitive
Survive crashes/deploys mid-pipeline	durable execution (event history + replay)
Do IO / side effects	an activity (@activity.defn)
Orchestrate steps deterministically	a workflow (@workflow.defn)
Retry a flaky step with backoff	activity RetryPolicy
Fail fast on bad input	non_retryable_error_types
Bound a single attempt's runtime	start_to_close_timeout
Bound total time incl. retries	schedule_to_close_timeout
Detect a stuck worker mid-activity	heartbeat + heartbeat_timeout
Wait real time (hours/days)	durable timer (workflow.sleep / wait_condition)
Push external data into a run	signal (@workflow.signal)
Read a running workflow's state	query (@workflow.query)
Fan work out in parallel	child workflows
Keep an infinite loop's history bounded	continue_as_new
Undo completed steps on failure	saga pattern (compensations)
Deterministic clock / id inside workflow	workflow.now() / workflow.uuid4()

RetryPolicy defaults & the four activity timeouts.

Knob	What it controls
initial_interval	wait before the first retry
backoff_coefficient	multiplier between retries (2.0 = double)
maximum_interval	cap on the backoff wait
maximum_attempts	retry cap (0 = unlimited)
schedule-to-start	queue wait before a worker starts it
start-to-close	one attempt's max runtime
schedule-to-close	total time across all attempts
heartbeat	max gap between heartbeats

Determinism — do / don't inside workflow code.

• Do: control flow, workflow.now(), workflow.uuid4(), workflow.sleep, wait_condition, execute_activity, execute_child_workflow.
• Don't: datetime.now() / time.time(), random.*, uuid.uuid4(), direct DB/HTTP/file IO, threads, iterating unordered collections in order-dependent ways.
• Change logic safely: gate with workflow.patched(...) or Worker Versioning so in-flight histories still replay.

Temporal vs task scheduler — the decision line. Scheduled + batch + short-lived + no compensation → task scheduler (Airflow-style). Reactive, per-entity, long-lived, or saga → Temporal (durable execution). Mixed estates run both.

Saga / compensation checklist.

• Record a compensation immediately after each forward step succeeds.
• Run compensations in reverse order on failure.
• Make both forward steps and compensations idempotent.
• Prefer continue_as_new for any workflow that loops without a natural end.

---

Frequently asked questions

What is durable execution in Temporal, in one sentence?

Durable execution is the guarantee that your workflow function runs to completion exactly as written — surviving process crashes, deploys, and machine loss — because Temporal records every step and its result in an append-only event history and replays that history to reconstruct the program's state. In practice it means you write ordinary code and get automatic checkpointing, so a crash halfway through a pipeline resumes at the next unfinished step instead of restarting. That is why temporal for data workflows removes the status tables and resume flags you would otherwise hand-roll.

Do activities need to be idempotent if Temporal already retries?

Yes — precisely because Temporal retries. Temporal guarantees at-least-once execution, never exactly-once, so a transient failure, a worker crash, or a heartbeat timeout can cause an activity to run more than once. If that activity has a side effect (a warehouse write, a payment, an email), you must make it idempotent — upsert on a business key, use a dedupe id, or a conditional write — so a second execution is a safe no-op rather than a duplicate.

What is continue-as-new and when do I need it?

continue_as_new atomically completes the current workflow run and starts a fresh one with the same workflow id and carried-forward state, resetting the event history to near-empty. You need it whenever a workflow loops many thousands of times or runs forever — a long polling loop, a per-entity workflow that lives indefinitely, or a cron-style perpetual workflow — because event history is not unbounded and a very long history slows replay and hits size limits. It is the standard way to keep long-running pipelines healthy.

Is Temporal a replacement for Airflow?

Not exactly — they overlap but target different shapes of work. Airflow is a scheduler for directed-acyclic-graph batch pipelines and excels at nightly ELT, DAG-of-SQL, and backfills; Temporal is a durable-execution engine for long-lived, stateful, reactive code and excels at per-entity workflows, human-in-the-loop waits, sagas, and sub-second reactions. Many teams run both — Airflow for scheduled warehouse builds and Temporal for the per-order, onboarding, and streaming-reaction workflow orchestration that DAGs handle awkwardly.

What breaks deterministic replay in a workflow?

Any non-determinism in workflow code: reading the wall clock with datetime.now() or time.time(), generating randomness or UUIDs with the standard library, doing direct network or database IO, spawning threads, or iterating an unordered collection in an order-dependent way. On replay these produce different values or a different command order than the recorded history, and Temporal raises a non-determinism error. The fixes are to use the deterministic equivalents (workflow.now(), workflow.uuid4(), durable timers) and to move every genuine side effect into an activity.

Do I need Temporal Cloud or can I self-host?

Both are supported and the programming model is identical. You can run the open-source Temporal Server yourself — the Frontend, History, Matching, and internal Worker services backed by Cassandra, PostgreSQL, or MySQL (plus optional Elasticsearch for visibility) — or use the managed Temporal Cloud and skip operating the server. Either way you write and run your own workers hosting workflow and activity code; only the server's operational burden differs.

---

Practice on PipeCode

Turn durable-execution theory into pipeline muscle memory

Reading about Temporal explains the primitives. PipeCode drills build the reflex interviews and real systems test — modelling a pipeline as workflows and activities, making every write idempotent, and choosing durable execution over a DAG when the work is stateful and long-running. Pipecode.ai is Leetcode for Data Engineering — scenario-first practice on ETL, streaming, and system design tuned to the trade-offs Temporal-style orchestration rewards.

Practice ETL problems →
Practice system-design problems →