Gowtham Potureddi

Posted on Sep 12

AWS Step Functions: Serverless Orchestration for Data

#python #sql #interview #dataengineering

aws step functions is the managed workflow engine that turns a pile of independently-deployed serverless pieces — a Lambda that validates a file, a Glue job that transforms it, an Athena query that aggregates it, an SNS topic that announces it — into a single, durable, observable pipeline that either finishes or tells you exactly where it stopped. The moment a data pipeline grows past "one Lambda triggered by one S3 event," you inherit the hard problems of distributed systems: what happens when step three throttles, how do you retry only the failed branch, how do you fan a thousand files out across concurrent workers without melting a downstream API, and how do you know six weeks later why last Tuesday's run produced half the rows it should have. Those problems do not live in the individual functions — each Lambda is fine — they live in the orchestration between them, and hand-wiring that orchestration with SQS queues, cron schedules, and Lambdas-invoking-Lambdas is how teams accidentally build a distributed monolith nobody can debug.

This guide is the walkthrough you wished existed the first time an interviewer asked "how would you orchestrate a serverless ETL pipeline on AWS without standing up an Airflow cluster," or "walk me through the difference between a Standard and an Express workflow," or "you need to process 200,000 files from an S3 prefix in parallel — what does the state machine look like." It opens the engine in layers: why a workflow orchestrator earns its keep over choreography, how the Amazon States Language describes a state machine as plain JSON, how the Map and Parallel states express fan-out ETL, how Retry and Catch make each step resilient to transient failure, and how the Standard-versus-Express choice trades durability against cost. Each section pairs a teaching block with a worked interview answer — code, a step-by-step trace, an output table, then a concept-by-concept breakdown of why it works.

When you want hands-on reps immediately after reading, drill the ETL practice library →, rehearse fan-out patterns on the data-processing practice library →, and tune the cost and concurrency knobs on the optimization practice library →.

On this page

Why orchestrate serverless work
States & the Amazon States Language
Map & Parallel for fan-out ETL
Retries, Catch & error handling
Standard vs Express + cost & patterns
Cheat sheet — AWS Step Functions recipes
Frequently asked questions
Practice on PipeCode

1. Why orchestrate serverless work

Orchestration beats choreography once a pipeline has more than two steps that can fail

The one-sentence invariant: aws step functions is a managed state machine that owns the control flow of a multi-step serverless pipeline — it holds the durable state between steps, retries and catches failures declaratively, records a per-execution history you can replay months later, and turns a tangle of event-driven Lambdas into a single named workflow with one place to look when something breaks. The alternative — choreography, where each Lambda knows which Lambda or queue comes next and fires it directly — works beautifully for two steps and becomes an un-debuggable distributed monolith at ten, because the DAG only exists implicitly, smeared across a dozen function handlers, IAM policies, and EventBridge rules that no single diagram captures.

Orchestration vs choreography — the axis that starts every discussion.

Choreography. Each component reacts to events and emits events; there is no central coordinator. S3 triggers Lambda A, which drops a message on SQS, which triggers Lambda B, which publishes to SNS. Loose coupling is the selling point; the cost is that the end-to-end flow is emergent, retries are per-component and inconsistent, and "where did execution 7f3a stall" has no answer because no component owns the whole run.
Orchestration. A central coordinator — the state machine — explicitly defines the order, the branching, the parallelism, and the error handling. Every step reports back; the coordinator decides what runs next. You trade a little coupling for a single source of truth about the workflow, uniform retry/catch semantics, and a visual execution history.
The senior framing. "Use choreography for decoupling independent services; use orchestration for a business process that must complete as a unit." A nightly ETL that must ingest, transform, quality-check, and publish as one durable transaction-like unit is an orchestration problem. Step Functions is AWS's orchestrator.

What a workflow orchestrator actually buys you.

Durable state between steps. The state machine persists the JSON that flows from one state to the next. A Standard workflow can pause for up to a year waiting on a callback; the state survives Lambda cold starts, deploys, and regional blips. You never hand-roll a "where was I" table.
Declarative error handling. Retry and Catch blocks live in the workflow definition, not scattered through function code. Every step gets consistent exponential backoff and consistent failure routing without a single try/except retry loop in your Lambdas.
Observability for free. Each execution is a first-class object with an input, an output, a status, and a step-by-step event history rendered as a graph in the console. When an interviewer asks "how do you debug a failed run," the answer is "open the execution, find the red state, read its input and error" — not "grep sixteen CloudWatch log groups."
Native service integrations. Step Functions can call 220+ AWS services directly through the SDK integration — start a Glue job, run an Athena query, put a DynamoDB item, publish to SNS — without a proxy Lambda per call. Fewer Lambdas means less code to own.

The 2026 reality — Step Functions is the default AWS-native orchestrator, but it is not the only one.

Step Functions is the default for serverless-first, AWS-native pipelines: event-driven ingestion, fan-out file processing, ML pipelines, saga-style microservice transactions. It shines when your steps are already AWS services.
Managed Airflow (MWAA) is the pick when you need a rich Python DAG ecosystem, hundreds of community operators, cross-cloud tasks, complex backfills, and a scheduler your data team already knows. It is a running cluster with a running cost; Step Functions is pay-per-use.
Glue Workflows orchestrate Glue-only crawlers and jobs; narrow but simple if your whole pipeline is Glue.
EventBridge / EventBridge Pipes handle pure event routing and light choreography — great glue between services, not a substitute for a durable multi-step workflow with branching and retries.

What interviewers listen for.

Do you say "orchestration, not choreography" and explain the trade-off? — senior signal.
Do you name durable state, retry/catch, and execution history as the concrete wins over hand-wired Lambdas? — required answer.
Do you reach for direct SDK integrations instead of a proxy Lambda per service call? — senior signal.
Do you frame the Standard vs Express choice as a workload decision (durability vs volume/cost) rather than a default? — senior signal.
Do you name idempotency and at-least-once delivery as the contract every task must honor? — required answer.

Worked example — the choreography-to-orchestration refactor

Detailed explanation. The most common real-world entry point to Step Functions is inheriting a choreographed Lambda chain that has become impossible to operate, and refactoring it into a single state machine. Walk through a four-step ingestion pipeline — validate, transform, load, notify — first as choreography, then as orchestration, and count the failure modes each design leaves open.

The pipeline. A partner drops a CSV in S3; it must be validated, transformed to Parquet, loaded into Redshift, and a Slack notification sent.
Choreographed version. S3 → Lambda validate → SQS → Lambda transform → SQS → Lambda load → SNS → Lambda notify.
Orchestrated version. S3 → EventBridge → Step Functions state machine with four Task states and shared error handling.

Question. List the failure modes the choreographed design leaves unhandled and show which single Step Functions feature closes each one.

Input.

Failure mode	Choreographed handling	Step Functions feature
`transform` throttles on a cold downstream	ad-hoc retry in code (often missing)	declarative `Retry` with backoff
`load` fails; partial data in Redshift	no coordinated rollback	`Catch` → compensation state
"where did run X stop?"	grep 4 log groups	execution history graph
one bad file blocks the SQS queue	poison-message stuck	per-execution isolation

Code.

{
  "Comment": "Ingestion pipeline as a single orchestrated workflow",
  "StartAt": "ValidateFile",
  "States": {
    "ValidateFile": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "validate", "Payload.$": "$" },
      "ResultPath": "$.validation",
      "Next": "TransformToParquet"
    },
    "TransformToParquet": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": { "JobName": "csv-to-parquet" },
      "Next": "LoadRedshift"
    },
    "LoadRedshift": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "load-redshift", "Payload.$": "$" },
      "Next": "Notify"
    },
    "Notify": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": { "TopicArn": "arn:aws:sns:us-east-1:111:ingest", "Message.$": "$" },
      "End": true
    }
  }
}

Step-by-step explanation.

The choreographed design smears the DAG across four Lambdas and three queues. The order "validate → transform → load → notify" exists nowhere as a single artifact — you reconstruct it by reading every handler. The orchestrated design writes the DAG down once, in the States map, where Next fields make every transition explicit.
In choreography, retry is whatever each author remembered to code. If the transform author forgot a retry loop, a transient Glue throttle drops the message to a dead-letter queue and the run silently stalls. In orchestration, a Retry block on the TransformToParquet state applies uniformly regardless of who wrote it.
load failing after transform succeeded is the classic partial-failure: Parquet exists in S3, but Redshift never got it. Choreography has no coordinated cleanup. Orchestration adds a Catch on LoadRedshift that routes to a compensation state (delete the orphaned Parquet, alert on-call).
"Where did execution X stop?" is unanswerable in choreography because no component owns the whole run. In Step Functions the execution is a single object; the console renders the graph with the failed state in red and its exact input/error attached.
A poison message — one malformed CSV — can block an SQS-driven consumer and stall every downstream file. Each Step Functions execution is isolated: a bad file fails its execution and no others.

Output.

Metric	Choreographed	Orchestrated
DAG visibility	implicit (read 4 handlers)	explicit (one JSON)
Retry consistency	per-author, uneven	declarative, uniform
Partial-failure cleanup	none	`Catch` → compensation
Debuggability	grep N log groups	execution graph
Blast radius of one bad file	queue-wide stall	single execution

Rule of thumb. The instant a serverless pipeline has more than two steps that can independently fail, stop wiring Lambdas to Lambdas and put a state machine in the middle. You are not adding a component — you are extracting the control flow that was already there, implicitly, into something you can see and operate.

Worked example — Step Functions vs MWAA (Airflow) on a real decision

Detailed explanation. The second-most-common interview probe is "why did you pick Step Functions over Airflow (or vice versa)?" The senior answer is never "Step Functions is better" — it is a workload-driven comparison. Walk the decision for two concrete pipelines: an event-driven S3 ingestion and a complex nightly analytics DAG with 80 interdependent Spark jobs and frequent backfills.

Pipeline A. Event-driven: a file lands, a short workflow runs, thousands of times a day, unpredictably. Latency and per-run cost matter; there is no fixed schedule.
Pipeline B. A scheduled analytics DAG: 80 tasks, complex dependencies, dynamic task generation, backfills over date ranges, a data team fluent in Python and Airflow operators.

Question. Recommend an orchestrator for each pipeline and defend the trade-off on cost, latency, and operational fit.

Input.

Dimension	Pipeline A (event-driven)	Pipeline B (nightly DAG)
Trigger	S3 event, bursty	cron, nightly
Task count	4–6	~80
Backfill needs	rare	frequent, date-ranged
Team skillset	AWS-native	Python / Airflow
Cost profile	pay-per-execution	steady, predictable

Code.

Decision heuristic (say this out loud)
======================================

Pick Step Functions when:
  - steps are already AWS services (Lambda, Glue, ECS, Athena, DynamoDB)
  - trigger is event-driven / bursty and you want pay-per-use, no idle cluster
  - you want managed durable state + built-in retry/catch + execution history
  - the workflow is a business process that completes as a unit

Pick MWAA / Airflow when:
  - you need rich Python DAGs, dynamic task mapping, hundreds of operators
  - complex backfills over date ranges are routine
  - the team already runs Airflow and cross-cloud/on-prem tasks exist
  - a steady always-on scheduler cost is acceptable

Anti-pattern:
  - running an always-on MWAA cluster to fire a 4-step event-driven flow
  - hand-building an 80-task backfill matrix in raw ASL JSON

Step-by-step explanation.

Pipeline A is bursty and event-driven with a handful of AWS-native steps. An always-on Airflow cluster bills 24/7 to serve traffic that arrives in unpredictable spikes — you pay for idle. Step Functions bills per state transition (Standard) or per request (Express), which matches a bursty load exactly.
Pipeline A's steps are Lambda and Glue — services Step Functions integrates with natively. There is no Python-operator ecosystem you need; the direct SDK integrations cover it. This is the archetypal Step Functions fit.
Pipeline B has 80 interdependent tasks and routine date-range backfills. Airflow's scheduler, dynamic task mapping, and backfill CLI are purpose-built for this; expressing it in raw ASL would be painful and the Distributed Map fan-out does not map cleanly onto "re-run 2024-01-01 through 2024-03-31."
Pipeline B's team already speaks Airflow. Operational fit — the team can read, debug, and extend the DAGs — often outweighs a marginal cost difference. Orchestrator choice is a socio-technical decision, not only a technical one.
The two anti-patterns are the tells of a weak answer: paying for an idle MWAA cluster to run a tiny event-driven flow, or torturing raw ASL into an 80-task backfill matrix. Naming both anti-patterns unprompted is the senior signal.

Output.

Pipeline	Recommendation	Primary reason
A (event-driven ingestion)	Step Functions (Express or Standard)	pay-per-use, AWS-native steps, no idle cluster
B (80-task nightly DAG)	MWAA / Airflow	rich DAGs, backfills, team fluency
Hybrid	Airflow triggers Step Functions	Airflow schedules; SFN owns the sub-workflow

Rule of thumb. Choose the orchestrator by the shape of the work, not by fashion. Event-driven, AWS-native, bursty → Step Functions. Scheduled, Python-heavy, backfill-heavy → Airflow. And remember the hybrid: Airflow can call Step Functions for the AWS-native leaf sub-workflows, letting each tool do what it is best at.

Data engineering interview question on serverless orchestration

A senior interviewer often opens with: "You have a serverless ingestion pipeline hand-wired as Lambda-invokes-Lambda through SQS. It works most days, but when it breaks, on-call spends an hour finding where. The business now wants a coordinated rollback when the Redshift load fails. Walk me through how you would re-architect this with AWS Step Functions, and justify why the orchestrator earns its place."

Solution Using an orchestrated state machine with coordinated compensation

{
  "Comment": "Re-architected ingestion with coordinated rollback (saga)",
  "StartAt": "Validate",
  "States": {
    "Validate": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "validate", "Payload.$": "$" },
      "ResultPath": "$.validation",
      "Retry": [
        { "ErrorEquals": ["States.TaskFailed"], "MaxAttempts": 2, "IntervalSeconds": 2, "BackoffRate": 2.0 }
      ],
      "Next": "Transform"
    },
    "Transform": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": { "JobName": "csv-to-parquet", "Arguments": { "--input.$": "$.s3key" } },
      "ResultPath": "$.transform",
      "Next": "Load"
    },
    "Load": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "load-redshift", "Payload.$": "$" },
      "ResultPath": "$.load",
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "CompensateCleanup" }
      ],
      "Next": "Notify"
    },
    "CompensateCleanup": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "delete-orphaned-parquet", "Payload.$": "$" },
      "Next": "FailRun"
    },
    "FailRun": {
      "Type": "Fail",
      "Error": "LoadFailed",
      "Cause": "Redshift load failed; orphaned Parquet cleaned up"
    },
    "Notify": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": { "TopicArn": "arn:aws:sns:us-east-1:111:ingest-done", "Message.$": "$" },
      "End": true
    }
  }
}

Step-by-step trace.

Step	Input state	What happens
1. Validate	`{s3key}`	Lambda checks the CSV; on transient error, retries twice with backoff
2. Transform	`{s3key, validation}`	Glue runs `.sync`; SFN waits for the job to finish
3. Load	`{..., transform}`	Lambda loads Redshift; success → Notify
4. Load fails	error object	`Catch` routes to CompensateCleanup with `$.error` preserved
5. CompensateCleanup	`{..., error}`	deletes the orphaned Parquet written in step 2
6. FailRun	—	execution ends as Failed with a clear Error/Cause

Walking a run that fails at Load: Validate and Transform succeed, so a Parquet file now sits in S3. Load throws; the Catch on Load fires, attaches the error under $.error, and transitions to CompensateCleanup, which deletes the orphaned Parquet so the next run starts clean. The execution then hits FailRun and terminates as Failed with Error: LoadFailed — on-call opens one execution and sees the whole story.

Output:

Concern	Before (choreography)	After (orchestration)
Time to locate failure	~1 hour across log groups	one execution graph
Retry on transient error	inconsistent per-Lambda	declarative on Validate
Rollback on Load failure	none (orphaned data)	compensation deletes Parquet
Run status	inferred	explicit Succeeded/Failed
Ownership of the DAG	none	the state machine

Why this works — concept by concept:

State machine as coordinator — the States map is the single durable definition of the control flow. Every transition is explicit via Next, so the DAG is an artifact you read, not a behavior you reverse-engineer from handlers.
Declarative Retry — the Retry on Validate gives uniform exponential backoff without a retry loop in Lambda code. Transient errors self-heal; consistent semantics apply to every step you add later.
Catch + compensation (saga) — Catch on Load routes failure to CompensateCleanup, which undoes the side effect (Transform's Parquet). This is the saga pattern: no distributed transaction, just a compensating action per step that can fail.
ResultPath preservation — each Task writes its result under a distinct key ($.validation, $.transform, $.load, $.error) so the accumulating JSON carries the full context into the cleanup and notify steps without clobbering the original input.
Cost — a handful of state transitions per run on a Standard workflow (fractions of a cent), plus the underlying Lambda/Glue cost you were already paying. The eliminated cost is the on-call hour per incident and the data-corruption risk of orphaned partial loads. O(steps) transitions per run versus O(engineer-hours) per mystery failure.

ETL
Topic — etl
ETL orchestration and pipeline-design problems

Practice →

Data Topic — data-processing Data-processing workflow problems

Practice →

2. States & the Amazon States Language

A state machine is JSON — named states, one active at a time, wired by transitions

The mental model in one line: the amazon states language (ASL) is a JSON dialect where a state machine is an object with a StartAt entry point and a States map, each state has a Type that determines its behavior, and control flows from state to state along Next transitions until a terminal state ends the execution — the entire workflow, including its branching, waiting, parallelism, and error handling, is data, not code. Because the workflow is declarative JSON, it is versionable, diffable, and rendered as a graph by the console; because each state has a typed contract, the engine — not your Lambda — owns the control flow.

The eight state types every ASL author must know.

Task. Does work — invokes a Lambda, starts a Glue job, runs an Athena query, or calls any of 220+ services via the SDK integration. The only state type that touches the outside world.
Choice. Branches on the input JSON. A list of rules (BooleanEquals, NumericGreaterThan, StringMatches, And/Or/Not) each with a Next; the first matching rule wins, with a Default fallback.
Wait. Pauses for a fixed Seconds, until a Timestamp, or until a value from the input (SecondsPath/TimestampPath). Costs nothing while waiting on a Standard workflow.
Parallel. Runs a fixed set of Branches concurrently and collects their outputs into an array.
Map. Runs the same sub-workflow once per item of an input array — the fan-out primitive.
Pass. Injects or reshapes JSON without doing work; handy for building fixtures and restructuring state.
Succeed / Fail. Terminal states. Succeed ends the execution successfully; Fail ends it with an Error and Cause.

Input and output processing — the plumbing that trips everyone up.

InputPath. Selects a slice of the raw state input to work with ($.detail instead of the whole event). Applied first.
Parameters. Constructs the exact payload passed to the resource, using .$ to pull from state ("FunctionName": "x", "Payload.$": "$").
ResultSelector. Reshapes the raw result the resource returns, keeping only what you want.
ResultPath. Decides where the (selected) result is merged into the running state — "$.load" nests it, "$" replaces the whole state, and null discards it.
OutputPath. Selects the final slice passed to the next state. Applied last. The order is fixed: InputPath → Parameters → (task runs) → ResultSelector → ResultPath → OutputPath.

Task integration patterns — three ways a Task waits.

Request/response (default). SFN calls the resource and immediately moves on with the response. Good for quick synchronous calls like lambda:invoke or sns:publish.
Run a job (.sync). SFN calls the resource and blocks until the job completes — glue:startJobRun.sync, ecs:runTask.sync, batch:submitJob.sync. This is how you wait for a long-running Glue or ECS job without polling.
Wait for callback (.waitForTaskToken). SFN passes a task token to the resource and pauses — up to a year on Standard — until something calls SendTaskSuccess/SendTaskFailure with that token. This is the human-in-the-loop and third-party-callback pattern.

Common interview probes on ASL.

"What are the state types?" — name all eight; Task and Choice and Map are the load-bearing three.
"In what order do InputPath, Parameters, ResultPath, and OutputPath apply?" — the fixed six-stage order above.
"How do you wait for a Glue job to finish?" — the .sync integration pattern, not a polling loop.
"How do you pause for human approval?" — .waitForTaskToken with SendTaskSuccess.

Worked example — a minimal Task → Choice → Succeed/Fail machine

Detailed explanation. The smallest useful state machine validates an input and branches on the result: a Task computes a score, a Choice routes on it, and the run ends in Succeed or Fail. Build it end-to-end and trace one execution so the input/output plumbing is concrete.

Task. A Lambda that returns a numeric score for an incoming record.
Choice. If score >= 80, go to Approved; otherwise Rejected.
Terminals. Approved is a Succeed; Rejected is a Fail with a clear reason.

Question. Write the ASL for a score-and-branch machine and trace the state as a record with score = 92 flows through.

Input.

Field	Value
Input event	`{ "recordId": "r-1", "amount": 4200 }`
Lambda result	`{ "score": 92 }`
Choice threshold	`score >= 80`
Expected path	Score → Choice → Approved (Succeed)

Code.

{
  "StartAt": "Score",
  "States": {
    "Score": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "score-record", "Payload.$": "$" },
      "ResultSelector": { "score.$": "$.Payload.score" },
      "ResultPath": "$.result",
      "Next": "CheckScore"
    },
    "CheckScore": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.result.score", "NumericGreaterThanEquals": 80, "Next": "Approved" }
      ],
      "Default": "Rejected"
    },
    "Approved": { "Type": "Succeed" },
    "Rejected": {
      "Type": "Fail",
      "Error": "ScoreTooLow",
      "Cause": "Record scored below the approval threshold"
    }
  }
}

Step-by-step explanation.

The execution starts at Score with the raw input {recordId, amount}. Parameters builds the Lambda payload with Payload.$: "$", passing the whole input to the function.
The Lambda returns { "Payload": { "score": 92 }, ... } (the lambda:invoke integration wraps the function's return under Payload). ResultSelector plucks just $.Payload.score and reshapes it to { "score": 92 }.
ResultPath: "$.result" merges that selected result under a new result key rather than overwriting the input. State is now { recordId, amount, result: { score: 92 } } — the original fields survive.
CheckScore is a Choice. Its single rule tests $.result.score >= 80. Because 92 satisfies it, the first matching rule wins and control transitions to Approved. If no rule matched, Default: "Rejected" would fire.
Approved is a Succeed terminal, so the execution ends successfully with the full state as output. Had the score been 55, the Choice would fall through to Default → Rejected, a Fail state that ends the run with Error: ScoreTooLow.

Output.

State	State JSON on entry	Transition
Score	`{recordId, amount}`	→ CheckScore
CheckScore	`{recordId, amount, result:{score:92}}`	rule matches → Approved
Approved	same	Succeed (end)

Rule of thumb. Reach for ResultSelector + ResultPath on every Task so results land under a named key and never clobber the input you still need downstream. A Choice should test a value the previous Task deliberately placed there — not a raw, unshaped resource response.

Worked example — ResultPath plumbing that preserves accumulating context

Detailed explanation. The single most common ASL bug is a Task overwriting the running state because ResultPath was left at its default ($, replace everything). In a multi-step pipeline each Task's output must accumulate so later steps still see earlier context. Walk through a three-Task pipeline that builds up a state object without ever losing a field.

The rule. ResultPath: "$" replaces the whole state with the result (usually wrong). ResultPath: "$.someKey" nests the result under someKey (usually right). ResultPath: null discards the result, keeping the input unchanged.
The goal. After three Tasks, the state should contain the original input plus each Task's result under its own key.

Question. Show how three Tasks accumulate context via distinct ResultPath keys, and contrast with the default that destroys it.

Input.

Task	Result returned	ResultPath	Effect
Fetch	`{rows: 1000}`	`$.fetch`	nests under `fetch`
Transform	`{parquet: "s3://..."}`	`$.transform`	nests under `transform`
Load	`{loaded: 1000}`	`$.load`	nests under `load`
(anti-pattern)	any	`$` (default)	wipes prior context

Code.

{
  "StartAt": "Fetch",
  "States": {
    "Fetch": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "fetch", "Payload.$": "$" },
      "ResultSelector": { "rows.$": "$.Payload.rows" },
      "ResultPath": "$.fetch",
      "Next": "Transform"
    },
    "Transform": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": { "JobName": "transform" },
      "ResultSelector": { "parquet.$": "$.JobRunState" },
      "ResultPath": "$.transform",
      "Next": "Load"
    },
    "Load": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "load",
        "Payload": { "source.$": "$.transform.parquet", "count.$": "$.fetch.rows" }
      },
      "ResultPath": "$.load",
      "End": true
    }
  }
}

Step-by-step explanation.

Fetch returns a row count; ResultPath: "$.fetch" nests it, so the state becomes { ...input, fetch: { rows: 1000 } }. The original input fields are untouched.
Transform runs a Glue job .sync and nests its result under $.transform. Crucially, $.fetch still exists — the state is now { ...input, fetch: {...}, transform: {...} }.
Load needs both the Parquet location from Transform and the row count from Fetch. Because both were preserved under distinct keys, Parameters can reference $.transform.parquet and $.fetch.rows in the same payload.
Had any Task used the default ResultPath: "$", it would have replaced the entire state with just its own result — $.fetch would vanish, and Load would fail to find the row count. This is the bug that produces States.Runtime "invalid path" errors at the last step.
The pattern generalizes: give every Task a unique ResultPath namespace, and the accumulating state document becomes a reliable context object every downstream state can read.

Output.

After state	State JSON
Fetch	`{...input, fetch:{rows:1000}}`
Transform	`{...input, fetch:{...}, transform:{parquet:"s3://..."}}`
Load	`{...input, fetch, transform, load:{...}}`

Rule of thumb. Treat the state document as an append-only context object: every Task writes under its own ResultPath key and reads other Tasks' keys by name. The default ResultPath: "$" is a footgun — use it only when you deliberately want to discard everything but the latest result.

Worked example — waitForTaskToken for a human-in-the-loop approval

Detailed explanation. Some steps cannot complete synchronously — they wait on a human approval, a third-party webhook, or an out-of-band job. The .waitForTaskToken integration pauses the execution and hands out a token; the workflow resumes only when someone calls SendTaskSuccess (or SendTaskFailure) with that token. Build an approval gate that pauses until a reviewer clicks a link.

The pause. A Task with Resource: arn:aws:states:::lambda:invoke.waitForTaskToken passes TaskToken.$: "$$.Task.Token" to the Lambda, which emails the reviewer a link embedding the token, then returns — but the state stays paused.
The resume. The reviewer's click hits an API that calls SendTaskSuccess(taskToken, {approved:true}); SFN injects that output and moves to the next state.
The guard. A TimeoutSeconds (or HeartbeatSeconds) ensures the paused state cannot wait forever.

Question. Write the approval-gate Task and explain how the token round-trip resumes the paused execution.

Input.

Element	Value
Integration	`lambda:invoke.waitForTaskToken`
Token source	`$$.Task.Token` (context object)
Resume call	`SendTaskSuccess(token, {approved})`
Timeout	86400 s (24 h)

Code.

{
  "StartAt": "RequestApproval",
  "States": {
    "RequestApproval": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Parameters": {
        "FunctionName": "email-approval-link",
        "Payload": {
          "recordId.$": "$.recordId",
          "taskToken.$": "$$.Task.Token"
        }
      },
      "TimeoutSeconds": 86400,
      "ResultPath": "$.approval",
      "Catch": [
        { "ErrorEquals": ["States.Timeout"], "ResultPath": "$.error", "Next": "AutoReject" }
      ],
      "Next": "CheckApproval"
    },
    "CheckApproval": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.approval.approved", "BooleanEquals": true, "Next": "Proceed" }
      ],
      "Default": "AutoReject"
    },
    "Proceed":    { "Type": "Succeed" },
    "AutoReject": { "Type": "Fail", "Error": "NotApproved", "Cause": "Timed out or rejected" }
  }
}

# The resume API handler (Lambda behind API Gateway)
import boto3
sfn = boto3.client("stepfunctions")

def handler(event, _ctx):
    token = event["queryStringParameters"]["token"]
    decision = event["queryStringParameters"]["decision"]  # "approve" | "reject"
    if decision == "approve":
        sfn.send_task_success(taskToken=token,
                              output='{"approved": true}')
    else:
        sfn.send_task_failure(taskToken=token,
                              error="Rejected", cause="Reviewer rejected")
    return {"statusCode": 200, "body": "Recorded"}

Step-by-step explanation.

RequestApproval uses lambda:invoke.waitForTaskToken. SFN generates a unique task token exposed on the context object as $$.Task.Token and passes it into the Lambda payload. The Lambda emails the reviewer a link containing the token, then returns — but the state machine does not advance.
The execution is now paused, durably, on a Standard workflow. It can wait hours or days; the TimeoutSeconds: 86400 guard ensures it will not wait past 24 hours.
When the reviewer clicks the link, API Gateway invokes the resume handler, which calls send_task_success(taskToken=..., output='{"approved": true}'). SFN matches the token to the paused execution and injects {approved:true} as the Task's result.
ResultPath: "$.approval" nests that result, so CheckApproval can test $.approval.approved. True → Proceed; anything else → AutoReject.
If nobody clicks within 24 hours, the States.Timeout error fires, the Catch routes to AutoReject, and the record is safely rejected rather than hung forever. The token-plus-timeout pair is the whole pattern.

Output.

Event	Effect on execution
Task starts	token minted; email sent; state paused
Reviewer approves	`SendTaskSuccess` → `$.approval.approved = true` → Proceed
Reviewer rejects	`SendTaskFailure` → Catch/Choice → AutoReject
24 h elapse	`States.Timeout` → Catch → AutoReject

Rule of thumb. Use .waitForTaskToken whenever a step depends on something outside the workflow's control — human approval, a partner callback, a long external job with no .sync integration. Always pair the token with a TimeoutSeconds and a Catch on States.Timeout so a lost callback fails the run cleanly instead of pausing it forever.

Data engineering interview question on the Amazon States Language

A senior interviewer might ask: "Design a state machine that ingests a record, calls an enrichment API, and — depending on a quality score — either loads it to the warehouse or routes it to a manual-review queue that waits for a human decision. Show the ASL, the input/output plumbing so context is preserved, and how you would keep the manual-review step from hanging forever."

Solution Using typed states, ResultPath accumulation, and a waitForTaskToken gate

{
  "StartAt": "Enrich",
  "States": {
    "Enrich": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "enrich", "Payload.$": "$" },
      "ResultSelector": { "score.$": "$.Payload.qualityScore", "enriched.$": "$.Payload.record" },
      "ResultPath": "$.enrich",
      "Next": "QualityGate"
    },
    "QualityGate": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.enrich.score", "NumericGreaterThanEquals": 90, "Next": "LoadWarehouse" },
        { "Variable": "$.enrich.score", "NumericLessThan": 50, "Next": "AutoDrop" }
      ],
      "Default": "ManualReview"
    },
    "ManualReview": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Parameters": {
        "FunctionName": "queue-for-review",
        "Payload": { "record.$": "$.enrich.enriched", "taskToken.$": "$$.Task.Token" }
      },
      "TimeoutSeconds": 172800,
      "ResultPath": "$.review",
      "Catch": [ { "ErrorEquals": ["States.Timeout"], "ResultPath": "$.error", "Next": "AutoDrop" } ],
      "Next": "ReviewDecision"
    },
    "ReviewDecision": {
      "Type": "Choice",
      "Choices": [ { "Variable": "$.review.approved", "BooleanEquals": true, "Next": "LoadWarehouse" } ],
      "Default": "AutoDrop"
    },
    "LoadWarehouse": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "load-warehouse", "Payload.$": "$.enrich.enriched" },
      "ResultPath": "$.load",
      "End": true
    },
    "AutoDrop": { "Type": "Fail", "Error": "RecordDropped", "Cause": "Low score or review timeout/reject" }
  }
}

Step-by-step trace.

Step	State JSON	Transition
1	`{record}`	Enrich runs
2	`{record, enrich:{score, enriched}}`	QualityGate evaluates score
3a (score 95)	same	→ LoadWarehouse → End
3b (score 70)	same	→ ManualReview (pauses on token)
4	`{..., review:{approved:true}}`	ReviewDecision → LoadWarehouse
5 (score 30)	same	→ AutoDrop (Fail)

Trace a record scoring 70: Enrich nests {score:70, enriched:{...}} under $.enrich. QualityGate finds neither >=90 nor <50, so it falls to Default: ManualReview, which mints a task token and pauses. A reviewer approves two hours later via SendTaskSuccess({approved:true}); $.review.approved becomes true; ReviewDecision routes to LoadWarehouse, which loads $.enrich.enriched. Had the reviewer never responded, the 48-hour TimeoutSeconds would trip States.Timeout → AutoDrop.

Output:

Score	Path taken	Terminal
≥ 90	Enrich → LoadWarehouse	Succeed
50–89 (approved)	Enrich → ManualReview → LoadWarehouse	Succeed
50–89 (timeout/reject)	Enrich → ManualReview → AutoDrop	Fail
< 50	Enrich → AutoDrop	Fail

Why this works — concept by concept:

Typed states — Task does the work, Choice branches, Fail terminates. Each state's Type tells the engine how to run it; the workflow logic is declarative JSON, not imperative code.
ResultSelector + ResultPath accumulation — Enrich selects only score and enriched and nests them under $.enrich, so both the Choice and the eventual load can read structured, named values without touching the raw Lambda envelope.
Choice ordering — Choice rules are evaluated top-to-bottom; the first match wins. Placing the >=90 and <50 rules explicitly and letting the mid-band fall to Default cleanly expresses "auto-load, auto-drop, else review."
waitForTaskToken + timeout — the manual-review gate pauses durably on a token and is bounded by TimeoutSeconds, with a Catch on States.Timeout routing to a safe default. A lost human decision fails the record instead of leaking a paused execution.
Cost — a Standard workflow bills per state transition (roughly 5–7 transitions here), pennies per thousand executions, and nothing while the review step is paused. The engine holds durable state for up to a year at no compute cost — the economic reason .waitForTaskToken beats a polling loop. O(states) transitions per run, O(1) idle cost while waiting.

ETL
Topic — etl
ETL branching and state-flow problems

Practice →

Data Topic — data-processing Data-processing pipeline-modeling problems

Practice →

3. Map & Parallel for fan-out ETL

`Map` fans one workflow across many items; `Parallel` fans many workflows across one input — both collect results as an array

The mental model in one line: the Map state runs the same sub-workflow once per element of an input array and gathers the per-item results into an output array, while the Parallel state runs a fixed set of different branches concurrently against the same input and gathers their outputs into an array — Map is dynamic fan-out over data, Parallel is static fan-out over tasks, and the two together are how Step Functions expresses parallel ETL without a single thread of your own concurrency code. The distinction interviewers test: Map's branch count depends on the data at runtime; Parallel's branch count is fixed in the definition.

Parallel — fixed concurrent branches.

What it is. A Parallel state has a Branches array; each branch is a full sub-state-machine with its own StartAt/States. All branches receive the same input and run concurrently.
The output. An array with one element per branch, in branch order. If any branch fails (and is not caught), the whole Parallel state fails.
When to use. Fan-out over different tasks: load the same transformed data to Redshift and Elasticsearch and a feature store simultaneously; or run independent quality checks in parallel and collect all results.

Inline Map — dynamic fan-out, modest scale.

What it is. Type: Map with an ItemsPath pointing at an input array and an ItemProcessor sub-workflow. SFN runs the processor once per item, up to MaxConcurrency at a time.
The limits. Inline Map keeps all iterations' state in the execution and caps concurrency around 40. It fits arrays of hundreds to low thousands of items — a batch of files from a single event.
The output. An array of per-item results, positionally aligned with the input array.

Distributed Map — massive fan-out, S3-scale.

What it is. Map in Distributed mode. It reads items from an ItemReader (an S3 object list, a CSV/JSON manifest, or a JSON array in S3), batches them with ItemBatcher, runs up to 10,000 concurrent child executions, and writes results with ItemWriter/ResultWriter back to S3.
Why it exists. Inline Map's ~40-concurrency and in-execution state cannot process 200,000 S3 objects. Distributed Map spawns child executions, so the state is not held in the parent, and concurrency scales to five figures.
The knobs. MaxConcurrency (cap the fan-out to protect downstream), ItemBatcher.MaxItemsPerBatch (amortize per-invocation overhead), ToleratedFailurePercentage (let a few items fail without killing the run).

Common interview probes on Map/Parallel.

"Map vs Parallel?" — Map = same workflow per array item (dynamic count); Parallel = fixed different branches (static count).
"Inline vs Distributed Map?" — inline ≈ 40 concurrency, in-execution state, thousands of items; Distributed → 10,000 concurrency, child executions, S3-scale.
"How do you stop a fan-out from overwhelming a downstream API?" — MaxConcurrency.
"How do you tolerate a few bad items in a million-item run?" — ToleratedFailurePercentage / ToleratedFailureCount.

Worked example — inline Map over a batch of files

Detailed explanation. An S3 event delivers a manifest listing ~500 files that each need a small transform. Inline Map is the right tool: the array is modest, and you want a bounded concurrency so the downstream service is not hammered. Build the Map state and trace the fan-out.

Input. { "files": ["a.csv", "b.csv", ... ] } — an array of ~500 keys.
Processor. A one-Task sub-workflow that invokes a transform-file Lambda per key.
Concurrency. MaxConcurrency: 20 to protect a rate-limited downstream.

Question. Write an inline Map that transforms each file with bounded concurrency and collects the per-file results.

Input.

Field	Value
ItemsPath	`$.files`
Item count	~500
MaxConcurrency	20
Per-item Task	`transform-file` Lambda

Code.

{
  "StartAt": "TransformAll",
  "States": {
    "TransformAll": {
      "Type": "Map",
      "ItemsPath": "$.files",
      "MaxConcurrency": 20,
      "ItemSelector": { "key.$": "$$.Map.Item.Value", "index.$": "$$.Map.Item.Index" },
      "ItemProcessor": {
        "ProcessorConfig": { "Mode": "INLINE" },
        "StartAt": "TransformOne",
        "States": {
          "TransformOne": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": { "FunctionName": "transform-file", "Payload.$": "$" },
            "ResultSelector": { "rows.$": "$.Payload.rows", "out.$": "$.Payload.outKey" },
            "End": true
          }
        }
      },
      "ResultPath": "$.results",
      "End": true
    }
  }
}

Step-by-step explanation.

TransformAll is a Map with ItemsPath: "$.files", so SFN iterates the 500-element array. MaxConcurrency: 20 means at most 20 iterations run at once — the 21st waits for a slot.
ItemSelector builds the payload for each iteration from the context object: $$.Map.Item.Value is the current file key and $$.Map.Item.Index is its position. Each iteration therefore starts with { key, index }.
The ItemProcessor sub-workflow (Mode: INLINE) has a single TransformOne Task that invokes transform-file with the per-item payload and selects rows and out from the result.
As iterations finish, SFN collects each processor's output into an array positionally aligned with $.files. ResultPath: "$.results" nests that array under results, preserving the original input.
If one iteration fails and is not caught inside the processor, the Map state fails — but you can add a Catch inside ItemProcessor, or (in Distributed mode) tolerate a failure percentage. Inline Map holds all 500 iterations' state in the single execution, which is why it caps out in the low thousands.

Output.

Field	Value after Map
`$.files`	unchanged 500-element array
`$.results[0]`	`{rows: 1200, out: "a.parquet"}`
`$.results[499]`	`{rows: 980, out: "zz.parquet"}`
Peak concurrency	20 iterations

Rule of thumb. Use inline Map for fan-out over an in-memory array of hundreds to a few thousand items, and always set MaxConcurrency — an unbounded Map will launch every iteration at once and can throttle or overwhelm whatever the per-item Task calls. Reach for Distributed Map the moment the item count crosses into the tens of thousands or the items live in S3.

Worked example — Distributed Map over an S3 prefix at scale

Detailed explanation. A daily job must process every object under s3://raw/events/2026-09-05/ — often 100,000+ files. Inline Map cannot: its ~40 concurrency and in-execution state do not scale. Distributed Map reads the S3 listing directly, batches items, runs thousands of child executions, tolerates a small failure rate, and writes results back to S3. Build it.

ItemReader. An S3 object list under a prefix — no need to materialize the manifest yourself.
ItemBatcher. Batch 100 keys per child execution to amortize Lambda invocation overhead.
Tolerance. Allow up to 1% of items to fail without failing the run.

Question. Write a Distributed Map that processes an S3 prefix with batching, capped concurrency, and a tolerated failure percentage, writing results to S3.

Input.

Knob	Value
Mode	DISTRIBUTED
ItemReader	S3 list of `raw/events/2026-09-05/`
MaxItemsPerBatch	100
MaxConcurrency	1000
ToleratedFailurePercentage	1

Code.

{
  "StartAt": "ProcessPrefix",
  "States": {
    "ProcessPrefix": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": { "Mode": "DISTRIBUTED", "ExecutionType": "EXPRESS" },
        "StartAt": "HandleBatch",
        "States": {
          "HandleBatch": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": { "FunctionName": "process-batch", "Payload.$": "$" },
            "End": true
          }
        }
      },
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": { "Bucket": "raw", "Prefix": "events/2026-09-05/" }
      },
      "ItemBatcher": { "MaxItemsPerBatch": 100 },
      "MaxConcurrency": 1000,
      "ToleratedFailurePercentage": 1,
      "ResultWriter": {
        "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": { "Bucket": "processed", "Prefix": "events/2026-09-05/results/" }
      },
      "End": true
    }
  }
}

Step-by-step explanation.

ProcessorConfig.Mode: DISTRIBUTED switches the Map into distributed mode, where each batch runs as its own child execution rather than an iteration held in the parent. This is what removes the in-execution state ceiling. ExecutionType: EXPRESS makes each child a cheap Express workflow.
ItemReader with s3:listObjectsV2 streams the object keys directly from the prefix — SFN handles pagination over 100,000+ keys; you never build a manifest by hand.
ItemBatcher.MaxItemsPerBatch: 100 groups 100 keys into one child execution, so process-batch handles 100 files per Lambda invocation. Batching amortizes cold-start and invocation overhead across many items — critical at six-figure scale.
MaxConcurrency: 1000 caps concurrent child executions at 1,000 (Distributed Map supports up to 10,000). Capping protects downstream systems and stays within Lambda concurrency limits.
ToleratedFailurePercentage: 1 lets up to 1% of items fail without failing the whole run — essential when processing a million files where a handful are inevitably corrupt. ResultWriter aggregates per-child results to S3 for a downstream reconcile.

Output.

Metric	Value
Items discovered	~120,000 keys
Batches (100/each)	~1,200 child executions
Peak concurrency	1,000
Tolerated failures	up to ~1,200 items (1%)
Results	written to `processed/.../results/`

Rule of thumb. When the item set lives in S3 and numbers in the tens of thousands or more, use Distributed Map with ItemBatcher to amortize per-invocation overhead, MaxConcurrency to protect downstream, and ToleratedFailurePercentage so a few bad items do not sink a million-item run. Inline Map is for the batch that fits in one execution; Distributed Map is for the prefix that does not.

Worked example — Parallel branches for multi-target load

Detailed explanation. After transforming data once, you often must load it to several targets at once — a warehouse, a search index, and a cache — each an independent task. Parallel runs all three branches concurrently and collects their results; if any fails, you catch it and decide. Build a three-branch Parallel with per-branch error handling.

Branches. Load Redshift, index into OpenSearch, warm a DynamoDB cache — three different sub-workflows.
Collection. Output is a three-element array, one per branch.
Resilience. A Catch on the Parallel state routes any branch failure to a partial-failure handler.

Question. Write a Parallel state that loads three targets concurrently and collects their results, with a catch for partial failure.

Input.

Branch	Target	Task
0	Redshift	`load-redshift`
1	OpenSearch	`index-opensearch`
2	DynamoDB cache	`warm-cache`

Code.

{
  "StartAt": "FanOutLoad",
  "States": {
    "FanOutLoad": {
      "Type": "Parallel",
      "Branches": [
        { "StartAt": "Redshift", "States": {
            "Redshift": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
              "Parameters": { "FunctionName": "load-redshift", "Payload.$": "$" }, "End": true } } },
        { "StartAt": "OpenSearch", "States": {
            "OpenSearch": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
              "Parameters": { "FunctionName": "index-opensearch", "Payload.$": "$" }, "End": true } } },
        { "StartAt": "Cache", "States": {
            "Cache": { "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
              "Parameters": { "FunctionName": "warm-cache", "Payload.$": "$" }, "End": true } } }
      ],
      "ResultPath": "$.loads",
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "PartialFailure" }
      ],
      "Next": "Done"
    },
    "PartialFailure": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": { "TopicArn": "arn:aws:sns:us-east-1:111:load-alerts", "Message.$": "$.error" },
      "Next": "Done"
    },
    "Done": { "Type": "Succeed" }
  }
}

Step-by-step explanation.

FanOutLoad is a Parallel with three Branches, each a self-contained sub-state-machine. All three receive the same input JSON and start concurrently — there is no ordering between them.
Each branch runs its single load Task. Because the branches are independent tasks (not iterations over data), this is Parallel, not Map: the branch count is fixed at three in the definition.
When all branches succeed, SFN assembles a three-element array [redshiftResult, openSearchResult, cacheResult] in branch order and nests it under $.loads via ResultPath.
If any branch throws — say OpenSearch is down — the whole Parallel state fails. The Catch on States.ALL intercepts it, preserves the error under $.error, and routes to PartialFailure, which alerts on-call rather than silently dropping the run.
The trade-off to name in an interview: Parallel is all-or-nothing at the state level. If you need "load whatever targets you can and report the rest," you either add a Catch inside each branch (so a branch failure becomes a recorded result instead of a state failure) or accept the all-or-nothing semantics and compensate in PartialFailure.

Output.

Outcome	`$.loads` / `$.error`	Next
all succeed	`[rs, os, cache]` under `$.loads`	Done
OpenSearch fails	error under `$.error`	PartialFailure → alert
per-branch Catch	failed branch records its own error	Done (degraded)

Rule of thumb. Use Parallel for a fixed set of independent tasks against one input, and Map for a variable set of identical tasks over an array. Remember Parallel is all-or-nothing at the state boundary — if you need partial success, catch inside each branch so a failure becomes data, not a state-level abort.

Data engineering interview question on fan-out ETL

A senior interviewer might ask: "You need to reprocess every file under an S3 prefix — roughly 500,000 objects — through a transform Lambda, write the results back to S3, and finish within an hour without throttling a downstream rate-limited API. A colleague reached for an inline Map and it failed. Design the correct fan-out, justify the concurrency and batching knobs, and explain how you keep a few corrupt files from failing the whole job."

Solution Using Distributed Map with batching, capped concurrency, and failure tolerance

{
  "StartAt": "ReprocessPrefix",
  "States": {
    "ReprocessPrefix": {
      "Type": "Map",
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": { "Bucket": "raw", "Prefix": "events/reprocess/" }
      },
      "ItemBatcher": { "MaxItemsPerBatch": 200 },
      "MaxConcurrency": 500,
      "ToleratedFailurePercentage": 2,
      "ItemProcessor": {
        "ProcessorConfig": { "Mode": "DISTRIBUTED", "ExecutionType": "EXPRESS" },
        "StartAt": "TransformBatch",
        "States": {
          "TransformBatch": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": { "FunctionName": "transform-batch", "Payload.$": "$" },
            "Retry": [
              { "ErrorEquals": ["Lambda.TooManyRequestsException", "States.TaskFailed"],
                "IntervalSeconds": 2, "MaxAttempts": 4, "BackoffRate": 2.0 }
            ],
            "End": true
          }
        }
      },
      "ResultWriter": {
        "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": { "Bucket": "processed", "Prefix": "events/reprocess/results/" }
      },
      "End": true
    }
  }
}

Step-by-step trace.

Knob	Value	Reasoning
Mode	DISTRIBUTED / EXPRESS children	removes in-execution state ceiling; cheap children
ItemReader	s3:listObjectsV2	streams 500k keys, paginated by SFN
MaxItemsPerBatch	200	~2,500 child executions instead of 500k
MaxConcurrency	500	protects the rate-limited downstream API
ToleratedFailurePercentage	2	a few corrupt files do not sink the run
Retry	4× backoff on throttle	absorbs transient 429s per batch

Walking the run: ItemReader streams ~500,000 keys; ItemBatcher groups 200 per batch → ~2,500 child Express executions; MaxConcurrency: 500 keeps at most 500 batches (100,000 files) in flight, which — combined with the per-batch Retry on throttling — holds the downstream API under its rate limit. Corrupt files fail their batch, but ToleratedFailurePercentage: 2 lets up to ~10,000 items fail before the run aborts. Results land in processed/.../results/ for a downstream reconcile.

Output:

Metric	Value
Objects processed	~500,000
Child executions	~2,500 (200 items each)
Peak in-flight batches	500
Tolerated failures	up to ~10,000 items (2%)
Wall-clock	well under 1 hour
Downstream throttling	absorbed by Retry + MaxConcurrency

Why this works — concept by concept:

Distributed Map with child executions — each batch runs as its own Express child execution, so the parent never holds 500k iterations of state. This is the only Map mode that scales past inline's ~40-concurrency, in-execution ceiling.
ItemReader over S3 — s3:listObjectsV2 streams and paginates the key listing natively; you fan out over the prefix without materializing a manifest, and SFN handles the pagination bookkeeping.
ItemBatcher amortization — batching 200 keys per child cuts 500,000 invocations to ~2,500, amortizing cold-start and per-invocation overhead. Batch size trades per-invocation efficiency against blast radius if a batch fails.
MaxConcurrency as a throttle valve — capping in-flight batches at 500 is how you respect a downstream rate limit without application-level rate-limiting code. The cap plus per-batch Retry with backoff turns bursty fan-out into a steady, downstream-safe flow.
Cost — Distributed Map bills the child executions (Express: per request + duration) plus the underlying Lambda; batching and concurrency caps keep both bounded. Compared to inline Map (impossible at this scale) or a hand-rolled thread pool (your own concurrency bugs), it is O(items/batch) child executions with engine-managed fan-out. Tolerated-failure and retry mean the run finishes once, not on the third manual re-kick.

Data
Topic — data-processing
Data-processing fan-out and parallelism problems

Practice →

ETL Topic — etl ETL batch-reprocessing problems

Practice →

4. Retries, Catch & error handling

`Retry` self-heals transient failures with backoff; `Catch` routes the unrecoverable ones to a cleanup branch

The mental model in one line: retry catch is Step Functions' two-layer resilience model — a Retry array on a state re-runs it with exponential backoff for the error classes you expect to be transient (throttles, timeouts, brief unavailability), and a Catch array on the same state routes any error that exhausts its retries (or was never retried) to a named recovery state, with the failing input and error preserved so the recovery path can compensate — together they let each step fail safely instead of aborting the whole execution. The interview distinction: Retry gives a step more chances; Catch gives the workflow a different path when the chances run out.

Retriers — the fields that shape backoff.

ErrorEquals. The list of error names this retrier matches — Lambda.TooManyRequestsException, States.Timeout, States.TaskFailed, a custom error your Lambda throws, or the catch-all States.ALL. Retriers are evaluated in order; the first match applies.
IntervalSeconds. The wait before the first retry. BackoffRate multiplies it each attempt: interval 2, rate 2.0 → waits of 2s, 4s, 8s, 16s.
MaxAttempts. How many retries (not counting the first try). 0 means "matched, but do not retry" — useful to shadow a broader retrier.
MaxDelaySeconds / JitterStrategy. Cap the exponential growth (MaxDelaySeconds) and add randomness (JitterStrategy: FULL) so a fleet of executions retrying in lockstep does not create a thundering herd.

Catchers — the fields that route failure.

ErrorEquals. Same error-name matching as Retry. A common shape is a specific catcher for a known error plus a States.ALL catcher last as a backstop.
Next. The recovery state to transition to — a cleanup, a compensation, a notify, or a Fail.
ResultPath. Where the error object ({Error, Cause}) is merged into the state. "$.error" preserves the original input and attaches the error, so the recovery state has full context. "$" would replace the state with only the error — usually wrong.

Error names — the vocabulary you must recognize.

States.ALL. Matches any error; only valid as the last entry in a Retry/Catch list.
States.TaskFailed. Any Task failure not matched by a more specific name.
States.Timeout. The state exceeded TimeoutSeconds or a heartbeat lapsed.
States.Runtime. An unrecoverable engine error (bad path, malformed output) — generally not retryable.
Custom errors. A Lambda that throws with a specific error type surfaces that type name (e.g. RecordNotFound), which you can match precisely.

Timeouts, heartbeats, and the saga pattern.

TimeoutSeconds. Always set one on Tasks that call something that can hang; without it a stuck Task can block for the workflow's max duration.
HeartbeatSeconds. For long .waitForTaskToken work, the worker calls SendTaskHeartbeat periodically; miss the window and States.Timeout fires — detecting a dead worker fast.
Saga / compensation. For multi-step workflows that mutate external state, each step has a compensating action; a Catch chains backward through compensations to undo partial work. There is no distributed transaction — just declaratively-ordered undo.

Common interview probes on error handling.

"Retry vs Catch?" — Retry re-runs the same state with backoff; Catch transitions to a different state after retries are exhausted.
"How do you avoid a retry storm?" — BackoffRate, MaxDelaySeconds, and JitterStrategy: FULL.
"How do you preserve the input when catching an error?" — ResultPath: "$.error", never "$".
"How do you roll back partial work?" — the saga pattern: a compensating action per step, chained via Catch.

Worked example — exponential backoff on a throttled API

Detailed explanation. A Task calls a third-party API that returns HTTP 429 under load. The right response is not to fail — it is to back off and retry, with jitter so a thousand concurrent executions do not all retry at the same instant. Build a retrier tuned for throttling and trace the wait schedule.

The error. The Lambda throws ApiThrottled on a 429.
The schedule. interval 2s, backoff 2.0, max 5 attempts, capped at 30s, full jitter.
The fallthrough. A separate States.ALL retrier with fewer attempts for anything else.

Question. Write a Retry configuration that absorbs API throttling with jittered exponential backoff and trace the retry waits.

Input.

Field	Value
Throttle error	`ApiThrottled`
IntervalSeconds	2
BackoffRate	2.0
MaxAttempts	5
MaxDelaySeconds	30
JitterStrategy	FULL

Code.

{
  "CallPartnerApi": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Parameters": { "FunctionName": "call-partner-api", "Payload.$": "$" },
    "TimeoutSeconds": 15,
    "Retry": [
      {
        "ErrorEquals": ["ApiThrottled"],
        "IntervalSeconds": 2,
        "BackoffRate": 2.0,
        "MaxAttempts": 5,
        "MaxDelaySeconds": 30,
        "JitterStrategy": "FULL"
      },
      {
        "ErrorEquals": ["States.TaskFailed", "States.Timeout"],
        "IntervalSeconds": 1,
        "BackoffRate": 2.0,
        "MaxAttempts": 2
      }
    ],
    "Catch": [
      { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "ApiFailed" }
    ],
    "Next": "Continue"
  }
}

Step-by-step explanation.

The Task has a 15-second TimeoutSeconds so a hung API call surfaces as States.Timeout rather than blocking indefinitely.
The first retrier matches ApiThrottled. On a 429, SFN waits IntervalSeconds (2s) before attempt 1, then multiplies by BackoffRate (2.0) each subsequent attempt: nominal waits of 2, 4, 8, 16, then 30 (capped by MaxDelaySeconds).
JitterStrategy: FULL randomizes each wait between 0 and the computed value, so concurrent executions spread their retries instead of hammering the API in synchronized waves — the fix for retry storms.
The second retrier matches generic States.TaskFailed/States.Timeout with only 2 attempts — a lighter touch for non-throttle failures. Retriers are evaluated in order, so a 429 matches the first, and only other errors reach the second.
If all retries for a matched error are exhausted, the error falls through to Catch. The States.ALL catcher preserves the input plus error under $.error and routes to ApiFailed, giving the failure path full context.

Output.

Attempt	Nominal wait	With FULL jitter
retry 1	2s	0–2s
retry 2	4s	0–4s
retry 3	8s	0–8s
retry 4	16s	0–16s
retry 5	30s (capped)	0–30s

Rule of thumb. Match transient errors (throttles, timeouts) with a dedicated retrier using exponential backoff, cap the growth with MaxDelaySeconds, and always add JitterStrategy: FULL when many executions can retry at once. Keep a lighter generic retrier after it, and a States.ALL Catch as the final backstop.

Worked example — Catch to a cleanup branch with error context

Detailed explanation. When a Task fails unrecoverably, the workflow should not just abort — it should run a cleanup that undoes side effects and records what happened. The key is ResultPath on the Catch, which decides whether the cleanup step sees the original input plus the error, or only the error. Build a load Task whose failure triggers a context-rich cleanup.

The failure. A load-warehouse Lambda fails after a transform already wrote a staging file.
The catch. Route to Cleanup with ResultPath: "$.error" so both the input and the error survive.
The cleanup. Delete the staging file (compensation) and alert with the error cause.

Question. Write the Catch that preserves context and the cleanup state that uses it, and contrast with the ResultPath: "$" mistake.

Input.

Field	Value
Failing Task	`load-warehouse`
Catch match	`States.ALL`
ResultPath	`$.error` (correct) vs `$` (wrong)
Cleanup	delete staging + SNS alert

Code.

{
  "LoadWarehouse": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Parameters": { "FunctionName": "load-warehouse", "Payload.$": "$" },
    "Catch": [
      { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "Cleanup" }
    ],
    "Next": "Done"
  },
  "Cleanup": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Parameters": {
      "FunctionName": "delete-staging",
      "Payload": {
        "stagingKey.$": "$.transform.stagingKey",
        "reason.$": "$.error.Cause"
      }
    },
    "Next": "FailRun"
  },
  "FailRun": { "Type": "Fail", "Error": "LoadFailed", "Cause": "Warehouse load failed; staging cleaned" }
}

Step-by-step explanation.

LoadWarehouse fails. Its Catch matches States.ALL and, critically, uses ResultPath: "$.error". SFN merges the error object {Error, Cause} under $.error while leaving the rest of the state — including $.transform.stagingKey written earlier — intact.
Cleanup can therefore read both the staging key ($.transform.stagingKey) from the earlier transform and the failure reason ($.error.Cause). It deletes the orphaned staging file and passes the reason along for the alert.
Had the Catch used ResultPath: "$" (the default), the state would be replaced by just {Error, Cause} — $.transform.stagingKey would vanish, and Cleanup would have nothing to delete. This is the single most common error-handling bug.
After compensation, FailRun terminates the execution as Failed with a clear Error/Cause. The run is honestly marked failed, but the side effect (orphaned staging data) has been undone.
The generalization is the saga: each mutating step gets a compensating cleanup, and a Catch routes to it with ResultPath: "$.error" so the compensation always has the context it needs.

Output.

ResultPath on Catch	State entering Cleanup	Result
`$.error` (correct)	input + `$.error`	staging deleted, alert has cause
`$` (wrong)	only `{Error, Cause}`	staging key lost, cleanup no-ops

Rule of thumb. On every Catch, set ResultPath: "$.error" so the recovery state inherits the full input and the error. Reserve the default ResultPath: "$" for the rare case where the recovery path genuinely needs nothing but the error object.

Worked example — the saga pattern for coordinated rollback

Detailed explanation. A booking workflow reserves inventory, charges a card, and books a slot — three external mutations. If the slot booking fails, the charge and the reservation must be undone. There is no distributed transaction across three services; the saga pattern chains a compensating action per step via Catch. Build the rollback chain.

Forward path. Reserve → Charge → Book.
Compensations. CancelBooking (none needed if Book failed), RefundCharge, ReleaseReservation.
The chain. Each forward step's Catch routes to its compensation, which then chains to the previous step's compensation.

Question. Design the saga so a failure at any forward step unwinds all prior steps in reverse order.

Input.

Forward step	Compensation	Catch target
Reserve	ReleaseReservation	on later failure
Charge	RefundCharge → ReleaseReservation	ChargeFailed
Book	RefundCharge → ReleaseReservation	BookFailed

Code.

{
  "StartAt": "Reserve",
  "States": {
    "Reserve": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "reserve", "Payload.$": "$" },
      "ResultPath": "$.reserve",
      "Catch": [ { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "ReserveFailed" } ],
      "Next": "Charge"
    },
    "Charge": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "charge", "Payload.$": "$" },
      "ResultPath": "$.charge",
      "Catch": [ { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "ReleaseReservation" } ],
      "Next": "Book"
    },
    "Book": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "book", "Payload.$": "$" },
      "ResultPath": "$.book",
      "Catch": [ { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "RefundCharge" } ],
      "Next": "Succeeded"
    },
    "RefundCharge": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "refund", "Payload.$": "$" },
      "Next": "ReleaseReservation"
    },
    "ReleaseReservation": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "release", "Payload.$": "$" },
      "Next": "SagaFailed"
    },
    "ReserveFailed": { "Type": "Fail", "Error": "ReserveFailed" },
    "SagaFailed":    { "Type": "Fail", "Error": "SagaRolledBack", "Cause": "Compensations applied" },
    "Succeeded":     { "Type": "Succeed" }
  }
}

Step-by-step explanation.

The forward path is Reserve → Charge → Book. Each step nests its result under a distinct key so compensations can see what to undo.
If Book fails, its Catch routes to RefundCharge (undo the charge), which then chains to ReleaseReservation (undo the reservation), which ends at SagaFailed. The unwind runs in reverse order of the forward mutations.
If Charge fails, there is no charge to refund, so its Catch skips straight to ReleaseReservation and then SagaFailed. The compensation chain starts at the right point for wherever the failure occurred.
If Reserve fails, nothing was mutated downstream, so its Catch goes directly to ReserveFailed — no compensation needed.
Every compensation preserves $.error context and is itself idempotent (refunding an already-refunded charge is a no-op), because Catch-driven compensations can themselves be retried. The saga gives transaction-like "all-or-nothing" semantics across services that share no transaction.

Output.

Failure point	Compensations run	Terminal
Reserve	none	ReserveFailed
Charge	ReleaseReservation	SagaFailed
Book	RefundCharge → ReleaseReservation	SagaFailed
none	—	Succeeded

Rule of thumb. Model multi-service mutations as a saga: one compensating action per forward step, each forward step's Catch pointing at the compensation for the previous step, and every compensation idempotent. Step Functions gives you the durable, ordered control flow to make "undo in reverse" a declarative chain rather than fragile cleanup code.

Data engineering interview question on error handling

A senior interviewer might ask: "Your enrichment pipeline calls a partner API that throttles under load and occasionally times out, then loads results to Redshift. Design the error handling: how you retry the throttling without a retry storm, how you catch the unrecoverable failures without losing the input, and how you guarantee no orphaned staging data when the final load fails."

Solution Using layered retries, jittered backoff, and a compensating catch

{
  "StartAt": "Enrich",
  "States": {
    "Enrich": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "enrich-via-partner", "Payload.$": "$" },
      "TimeoutSeconds": 20,
      "Retry": [
        { "ErrorEquals": ["ApiThrottled"], "IntervalSeconds": 2, "BackoffRate": 2.0,
          "MaxAttempts": 6, "MaxDelaySeconds": 30, "JitterStrategy": "FULL" },
        { "ErrorEquals": ["States.Timeout"], "IntervalSeconds": 1, "BackoffRate": 2.0, "MaxAttempts": 3 }
      ],
      "ResultPath": "$.enrich",
      "Catch": [ { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "AlertAndFail" } ],
      "Next": "Stage"
    },
    "Stage": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "write-staging", "Payload.$": "$" },
      "ResultPath": "$.stage",
      "Next": "LoadRedshift"
    },
    "LoadRedshift": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "load-redshift", "Payload.$": "$" },
      "Retry": [
        { "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 3, "BackoffRate": 2.0, "MaxAttempts": 3 }
      ],
      "Catch": [ { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "CleanupStaging" } ],
      "ResultPath": "$.load",
      "Next": "Succeeded"
    },
    "CleanupStaging": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "delete-staging", "Payload": { "key.$": "$.stage.key", "reason.$": "$.error.Cause" } },
      "Next": "AlertAndFail"
    },
    "AlertAndFail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": { "TopicArn": "arn:aws:sns:us-east-1:111:pipeline-alerts", "Message.$": "$.error" },
      "Next": "FailRun"
    },
    "FailRun": { "Type": "Fail", "Error": "PipelineFailed" },
    "Succeeded": { "Type": "Succeed" }
  }
}

Step-by-step trace.

Step	Mechanism	Effect
Enrich throttled	`ApiThrottled` retrier, 6× jittered backoff	absorbs 429s without a storm
Enrich times out	`States.Timeout` retrier, 3×	retries a hung call, then catches
Enrich unrecoverable	Catch → AlertAndFail	input preserved, alert fired
Stage writes file	`$.stage.key` recorded	staging key available for cleanup
Load fails	Catch → CleanupStaging	staging deleted, then alert + fail

Trace a run where the partner throttles twice, then succeeds, then the Redshift load fails: Enrich's ApiThrottled retrier waits ~2s and ~4s (jittered) and succeeds on the third try, nesting results under $.enrich. Stage writes a staging file and records $.stage.key. LoadRedshift exhausts its 3 retries and throws; its Catch preserves $.error and routes to CleanupStaging, which reads $.stage.key, deletes the orphaned file, then alerts and fails the run. No orphaned staging data survives.

Output:

Scenario	Retries used	Compensation	Terminal
throttle then success	2 on Enrich	none	Succeeded
enrich unrecoverable	6 exhausted	none (nothing staged)	Failed + alert
load fails	3 on Load	staging deleted	Failed + alert
clean run	0	none	Succeeded

Why this works — concept by concept:

Layered retriers — a specific ApiThrottled retrier with generous attempts sits before a tighter States.Timeout retrier; ordered evaluation means each error class gets the treatment it deserves, and only truly unrecoverable errors reach the Catch.
Jittered exponential backoff — BackoffRate plus MaxDelaySeconds plus JitterStrategy: FULL spreads retries across executions, so a fleet recovering from a throttle does not synchronize into a second outage. This is the difference between self-healing and a retry storm.
Context-preserving Catch — every Catch uses ResultPath: "$.error", so recovery states inherit the full input. CleanupStaging can read $.stage.key precisely because the error was merged in, not substituted for, the state.
Compensation before failure — LoadRedshift's Catch runs CleanupStaging before FailRun, so the execution ends Failed but with side effects undone. Marking the run honestly failed and leaving no orphaned data are not in tension.
Cost — retries and compensations are extra state transitions (Standard) — pennies — set against the cost they avoid: corrupted warehouse state, manual staging cleanup, and 429-induced cascading failures. O(retries + compensations) transitions per failed run, and zero on the happy path.

ETL
Topic — etl
ETL resilience and retry-design problems

Practice →

Perf Topic — optimization Optimization problems on backoff and throughput

Practice →

5. Standard vs Express + cost & patterns

`express vs standard` is a durability-versus-volume decision that also sets your bill

The mental model in one line: Step Functions ships two workflow types — Standard is durable, exactly-once, runs up to a year, keeps a full replayable execution history, and is billed per state transition, while Express is high-volume, runs up to five minutes, gives at-least-once (async) or at-most-once (sync) semantics, keeps no durable history by default, and is billed per request plus duration and memory — and picking the wrong one either overpays by orders of magnitude for a high-frequency workload or under-delivers durability for a long-running one. The senior answer frames it as a workload question, not a default, and knows the killer pattern: nesting Express workflows inside a Standard parent to get durable orchestration and cheap high-volume inner steps.

Standard — durable and observable.

Duration & durability. Up to one year per execution; state is durably persisted, so a workflow can pause on .waitForTaskToken for days. Survives everything.
Semantics. Exactly-once execution of the workflow. Each state transition is durably recorded.
History. Full execution history retained and visualized — the debuggability that makes Standard the default for anything you need to audit.
Billing. Per state transition (plus the underlying resource costs). Cheap per run, but a high-frequency workload with many transitions adds up.

Express — high-volume and cheap.

Duration & durability. Up to five minutes; no durable per-state history (logs go to CloudWatch if you enable them). Built for volume, not longevity.
Semantics. Asynchronous Express is at-least-once (may re-run on retry); Synchronous Express is at-most-once and returns the result inline — so your steps must be idempotent.
Throughput. Designed for 100,000+ executions per second — event processing, IoT ingestion, high-frequency APIs.
Billing. Per request + duration + memory. For short, high-frequency runs this is dramatically cheaper than per-transition Standard pricing.

The cost intuition every senior engineer carries.

Standard cost driver. Number of state transitions × executions. A 10-state workflow run a million times a day is ten million transitions — measurable money.
Express cost driver. Requests × duration × memory. A sub-second workflow run millions of times is fractions of a cent each.
The crossover. Long, low-frequency, must-be-durable → Standard. Short, high-frequency, idempotent → Express. When both pull at once, nest.

Design patterns worth naming.

Fan-out/fan-in ETL. A Standard parent orchestrates; a Distributed Map with Express children does the massive per-item work. Durable outer, cheap inner.
Event-driven ingestion. EventBridge → Express workflow for high-frequency, short-lived processing where per-run durability is unnecessary.
Human-in-the-loop. Standard with .waitForTaskToken — only Standard can pause for days.
Nested Express inside Standard. The parent Standard workflow owns durability, retries, and history; each inner Express child handles a burst of cheap work and returns.

Common interview probes on Standard vs Express.

"When Express, when Standard?" — Express for short high-volume idempotent; Standard for long durable auditable.
"What are Express's delivery semantics?" — async at-least-once, sync at-most-once; steps must be idempotent.
"How do you get durability and high volume?" — nest Express inside a Standard parent.
"Why can't Express do a 3-day approval?" — 5-minute cap and no durable pause.

Worked example — pick the type from a workload table

Detailed explanation. The fastest way to answer "Standard or Express" is a small decision table keyed on duration, frequency, durability, and idempotency. Walk three concrete workloads through it and record the pick.

Workload A. Nightly ETL: 15 steps, runs once a day, must be auditable and can pause for a manual gate.
Workload B. Clickstream ingestion: a 3-state enrich-and-store workflow, 50,000 events/second, idempotent.
Workload C. Order saga: 6 steps, thousands/day, must be exactly-once and durable, no long pause.

Question. Assign each workload a workflow type and justify it against the four axes.

Input.

Workload	Duration	Frequency	Durability need	Idempotent?
A. Nightly ETL	minutes–hours (with pause)	1/day	high (audit)	mixed
B. Clickstream	milliseconds	50k/s	low	yes
C. Order saga	seconds	thousands/day	high (exactly-once)	mixed

Code.

Pick-the-type decision
======================

if duration > 5 minutes OR needs durable pause OR must be exactly-once:
    -> STANDARD
elif very high frequency AND short AND idempotent:
    -> EXPRESS (sync if caller needs the result inline; async otherwise)
else:
    -> STANDARD by default (durability + history are usually worth it)

Nest when both pull:
    STANDARD parent (durable orchestration, retries, history)
      └── Distributed Map -> EXPRESS children (cheap per-item burst)

Step-by-step explanation.

Workload A runs long (it can pause on a manual gate) and must be auditable. The 5-minute Express cap alone rules Express out; add the durable-pause and audit needs and it is unambiguously Standard.
Workload B is millisecond-short, runs 50,000/second, and is idempotent. This is the textbook Express case — per-request pricing at that volume is orders of magnitude cheaper than per-transition Standard, and no per-run durability is required.
Workload C is short but must be exactly-once and durable (money is moving). Express's at-least-once/at-most-once semantics are wrong for exactly-once; the saga also benefits from durable history. Pick Standard despite the modest per-run cost.
The default when unsure is Standard: durability and history usually earn their keep, and only a genuinely high-frequency, short, idempotent workload justifies giving them up for Express pricing.
When a workload has a durable outer process and a high-volume inner burst (Workload A's per-file transform, for instance), nest: Standard parent for orchestration, Express children under a Distributed Map for the cheap fan-out.

Output.

Workload	Type	Deciding axis
A. Nightly ETL	Standard	duration + durable pause + audit
B. Clickstream	Express (async)	frequency + short + idempotent
C. Order saga	Standard	exactly-once + durability

Rule of thumb. Route by the four axes in order: if it runs over five minutes, needs a durable pause, or must be exactly-once, it is Standard. Only a short, high-frequency, idempotent workload should be Express — and when a durable process wraps a high-volume burst, nest Express inside Standard rather than compromising on either.

Worked example — the cost comparison that justifies the choice

Detailed explanation. Nothing settles the Standard-vs-Express debate like the arithmetic. Compare the two on a high-frequency workload — a 4-state workflow run 10 million times a day — using the pricing shapes (Standard: per state transition; Express: per request + GB-second). The numbers make the pattern obvious.

Standard shape. ~$25 per million state transitions.
Express shape. ~$1.00 per million requests + a small GB-second duration charge.
Workload. 10M runs/day × 4 transitions = 40M transitions/day (Standard) vs 10M requests/day (Express).

Question. Estimate the daily cost of the workload under each type and state the crossover intuition.

Input.

Parameter	Value
Runs/day	10,000,000
States per run	4
Standard: $/1M transitions	~$25
Express: $/1M requests	~$1
Express duration	~100 ms @ 128 MB

Code.

# Illustrative daily-cost estimate (rates approximate; check current pricing)
runs_per_day      = 10_000_000
states_per_run    = 4

# Standard: billed per state transition
std_transitions   = runs_per_day * states_per_run           # 40,000,000
std_cost          = std_transitions / 1_000_000 * 25.00      # ~$1,000/day

# Express: billed per request + GB-seconds of duration
exp_requests_cost = runs_per_day / 1_000_000 * 1.00          # ~$10/day
gb                = 128 / 1024                                # 0.125 GB
gb_seconds        = runs_per_day * 0.100 * gb                 # 0.1s each
exp_duration_cost = gb_seconds * 0.00001667                  # ~$2.6/day
exp_cost          = exp_requests_cost + exp_duration_cost     # ~$13/day

print(f"Standard ~${std_cost:,.0f}/day; Express ~${exp_cost:,.0f}/day")
# -> Standard ~$1,000/day; Express ~$13/day

Step-by-step explanation.

Standard bills every state transition. At 10M runs × 4 states = 40M transitions/day, and ~$25 per million, that is roughly $1,000/day — before any Lambda cost.
Express bills per request plus GB-seconds. 10M requests/day at ~$1/million is ~$10/day; the duration charge for 100 ms at 128 MB adds only a few dollars. Total roughly $13/day.
The ~75× gap is not a rounding difference — it is the structural reason high-frequency, short workloads belong on Express. Per-transition pricing punishes many-state workflows run millions of times.
The intuition inverts for low-frequency, long, durable workflows: a nightly ETL run once a day has 15 transitions and costs a fraction of a cent, and there Standard's durability and history are free wins. Express would save nothing and cost you durability.
The nested pattern captures both: keep the durable orchestration on Standard (a handful of transitions per run) and push the millions of cheap per-item executions to Express children, so the expensive per-transition pricing never touches the high-frequency layer.

Output.

Type	Daily cost (this workload)	Fit
Standard	~$1,000/day	wrong for high frequency
Express	~$13/day	right for high frequency
Nested (Std+Exp)	~$13/day + a few Std transitions	best of both

Rule of thumb. For short, high-frequency workflows, Express is often 50–100× cheaper than Standard because Standard prices per state transition and those multiply fast. Do the back-of-envelope transition count before committing — and if a durable outer process wraps the high-frequency work, nest Express inside Standard so the per-transition pricing never meets the high-volume layer.

Worked example — nested Express inside Standard

Detailed explanation. The capstone pattern: a Standard parent owns durability, retries, and audit history; a Distributed Map inside it launches Express children for the massive, cheap per-item work. Build the two-tier workflow and explain why it is the default shape for large fan-out ETL.

Parent (Standard). Validates input, runs the Distributed Map, then aggregates and notifies — a handful of durable transitions.
Children (Express). Each processes one batch of items, returns, and is billed per request — millions of them, cheaply.
Why nest. Durable orchestration where you need it; cheap volume where you need that.

Question. Write the Standard parent that fans out to Express children via Distributed Map and aggregates the results.

Input.

Tier	Type	Role
Parent	Standard	validate → map → aggregate → notify
Map children	Express	per-batch processing
Aggregation	Standard Task	reduce child results

Code.

{
  "StartAt": "Validate",
  "States": {
    "Validate": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "validate-input", "Payload.$": "$" },
      "ResultPath": "$.validation", "Next": "FanOut"
    },
    "FanOut": {
      "Type": "Map",
      "ItemReader": { "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": { "Bucket": "raw", "Prefix": "batch/2026-09-05/" } },
      "ItemBatcher": { "MaxItemsPerBatch": 250 },
      "MaxConcurrency": 800,
      "ItemProcessor": {
        "ProcessorConfig": { "Mode": "DISTRIBUTED", "ExecutionType": "EXPRESS" },
        "StartAt": "ProcessBatch",
        "States": {
          "ProcessBatch": {
            "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": { "FunctionName": "process-batch", "Payload.$": "$" },
            "Retry": [ { "ErrorEquals": ["States.TaskFailed"], "MaxAttempts": 3, "IntervalSeconds": 2, "BackoffRate": 2.0 } ],
            "End": true
          }
        }
      },
      "ResultWriter": { "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": { "Bucket": "processed", "Prefix": "batch/2026-09-05/results/" } },
      "ResultPath": "$.mapRun", "Next": "Aggregate"
    },
    "Aggregate": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "aggregate-results", "Payload.$": "$.mapRun" },
      "ResultPath": "$.summary", "Next": "Notify"
    },
    "Notify": {
      "Type": "Task", "Resource": "arn:aws:states:::sns:publish",
      "Parameters": { "TopicArn": "arn:aws:sns:us-east-1:111:batch-done", "Message.$": "$.summary" },
      "End": true
    }
  }
}

Step-by-step explanation.

The parent is a Standard workflow: Validate → FanOut → Aggregate → Notify is only four durable state transitions per run, so per-transition pricing is negligible, and the run gets full history and durability.
FanOut is a Distributed Map whose children run as Express executions (ExecutionType: EXPRESS). Each child processes a 250-item batch and is billed per request — the millions of cheap executions live entirely in this Express tier.
MaxConcurrency: 800 caps the fan-out; per-batch Retry handles transient failures inside the cheap tier without bloating the parent's transition count.
ResultWriter writes child outputs to S3, and Aggregate (a Standard Task) reduces them into a summary — the durable parent owns the reduce step and the notification.
The result is the best of both: durable orchestration, retries, and audit trail from the Standard parent; five-figure concurrency and per-request pricing from the Express children. This two-tier shape is the default for large serverless fan-out ETL.

Output.

Tier	Executions	Billing	Property
Standard parent	1/run	per transition (×4)	durable, audited
Express children	~thousands/run	per request + duration	cheap, high-concurrency
Combined	—	dominated by Express	durable + cheap

Rule of thumb. For large fan-out ETL, make the outer workflow Standard for durability, retries, and history, and make the Distributed Map's children Express for cheap, high-concurrency per-item work. Nesting lets each tier do what it prices best — you never pay Standard's per-transition rate for the millions of inner executions.

Data engineering interview question on Standard vs Express

A senior interviewer might ask: "You are designing a serverless pipeline that must process a nightly batch of half a million files, remain fully auditable end-to-end, retry transient failures, and stay cheap at that volume. Choose the workflow type (or types), justify it on cost and durability, and show the state machine that delivers both."

Solution Using a Standard parent with nested Express Distributed-Map children

{
  "StartAt": "Prepare",
  "States": {
    "Prepare": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "prepare-manifest", "Payload.$": "$" },
      "ResultPath": "$.prep",
      "Retry": [ { "ErrorEquals": ["States.TaskFailed"], "MaxAttempts": 2, "IntervalSeconds": 2, "BackoffRate": 2.0 } ],
      "Next": "ProcessAll"
    },
    "ProcessAll": {
      "Type": "Map",
      "ItemReader": { "Resource": "arn:aws:states:::s3:listObjectsV2",
        "Parameters": { "Bucket": "raw", "Prefix": "nightly/2026-09-05/" } },
      "ItemBatcher": { "MaxItemsPerBatch": 200 },
      "MaxConcurrency": 600,
      "ToleratedFailurePercentage": 1,
      "ItemProcessor": {
        "ProcessorConfig": { "Mode": "DISTRIBUTED", "ExecutionType": "EXPRESS" },
        "StartAt": "TransformBatch",
        "States": {
          "TransformBatch": {
            "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": { "FunctionName": "transform-batch", "Payload.$": "$" },
            "Retry": [ { "ErrorEquals": ["Lambda.TooManyRequestsException", "States.TaskFailed"],
              "MaxAttempts": 4, "IntervalSeconds": 2, "BackoffRate": 2.0, "JitterStrategy": "FULL" } ],
            "End": true
          }
        }
      },
      "ResultWriter": { "Resource": "arn:aws:states:::s3:putObject",
        "Parameters": { "Bucket": "processed", "Prefix": "nightly/2026-09-05/results/" } },
      "ResultPath": "$.run",
      "Catch": [ { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "AlertFail" } ],
      "Next": "Reconcile"
    },
    "Reconcile": {
      "Type": "Task", "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "reconcile", "Payload.$": "$.run" },
      "ResultPath": "$.reconcile", "Next": "Notify"
    },
    "Notify": {
      "Type": "Task", "Resource": "arn:aws:states:::sns:publish",
      "Parameters": { "TopicArn": "arn:aws:sns:us-east-1:111:nightly-done", "Message.$": "$.reconcile" },
      "End": true
    },
    "AlertFail": {
      "Type": "Task", "Resource": "arn:aws:states:::sns:publish",
      "Parameters": { "TopicArn": "arn:aws:sns:us-east-1:111:nightly-alerts", "Message.$": "$.error" },
      "Next": "FailRun"
    },
    "FailRun": { "Type": "Fail", "Error": "NightlyBatchFailed" }
  }
}

Step-by-step trace.

Layer	Config	Result
Parent type	Standard	durable, auditable, ~5 transitions/run
Map mode	Distributed / Express children	500k files without in-parent state
Batching	200 items/batch	~2,500 Express children
Concurrency	600	protects downstream; bounded fan-out
Failure tolerance	1%	corrupt files do not abort the night
Cost profile	Express dominates	pennies at volume, durable outer

Walking the nightly run: the Standard Prepare step builds a manifest (2 transitions, durable). ProcessAll fans ~500,000 files into ~2,500 Express children at 600-way concurrency, each with jittered retry on throttles; up to 1% may fail without aborting. ResultWriter lands per-child results in S3; Reconcile (Standard Task) diffs expected vs actual counts; Notify announces success. Any catastrophic Map failure is caught to AlertFail → FailRun. The parent's handful of transitions cost fractions of a cent; the Express children carry the volume cheaply; the whole run is auditable in the Standard execution history.

Output:

Metric	Value
Files processed	~500,000
Express children	~2,500 (200 each)
Standard transitions/run	~5
Tolerated failures	up to ~5,000 files (1%)
Auditability	full Standard history
Cost	Express-dominated (pennies at volume)

Why this works — concept by concept:

Standard parent for durability — the outer workflow is exactly-once with a full execution history and only ~5 state transitions per run, so it is both auditable and cheap; durability where it matters costs almost nothing at low transition counts.
Express Distributed-Map children — pushing the ~2,500 per-batch executions to Express means the high-volume tier is billed per request, not per state transition, keeping cost flat as file counts grow.
Batching + capped concurrency — 200 items per child and MaxConcurrency: 600 amortize invocation overhead and protect downstream systems, converting a 500k-file burst into a steady, bounded flow.
Failure tolerance + jittered retry — ToleratedFailurePercentage: 1 plus per-batch jittered backoff means transient throttles self-heal and a few corrupt files do not force a manual re-kick of the entire night.
Cost — the run's cost is dominated by the Express children (per request + short duration), with the Standard parent adding only a handful of transitions. Compared to an all-Standard design (40M+ transitions if every item were a state) this is 50–100× cheaper, and compared to an all-Express design it keeps end-to-end durability and audit. O(files/batch) Express executions, O(1) durable-parent overhead.

Perf
Topic — optimization
Optimization problems on cost and concurrency

Practice →

Data
Topic — data-processing
Data-processing pipeline-scaling problems

Practice →

Cheat sheet — AWS Step Functions recipes

State type quick reference. Task (do work / call a service), Choice (branch on input), Wait (pause N seconds / until timestamp), Parallel (fixed concurrent branches → array), Map (same sub-workflow per array item → array), Pass (inject/reshape JSON), Succeed / Fail (terminals). Task and Choice and Map are the load-bearing three; everything else supports them.
I/O processing order. The fixed pipeline is InputPath → Parameters → (resource runs) → ResultSelector → ResultPath → OutputPath. InputPath slices the input, Parameters builds the resource payload (.$ pulls from state), ResultSelector reshapes the raw result, ResultPath nests it into state, OutputPath slices what passes on. Give every Task a distinct ResultPath key so state accumulates instead of clobbering.
Retry block template. "Retry": [ { "ErrorEquals": ["ApiThrottled"], "IntervalSeconds": 2, "BackoffRate": 2.0, "MaxAttempts": 5, "MaxDelaySeconds": 30, "JitterStrategy": "FULL" } ] — a specific retrier for transient errors first, a tighter generic one after, States.ALL never in Retry except as the final entry. Jitter is mandatory when many executions can retry at once.
Catch block template. "Catch": [ { "ErrorEquals": ["States.ALL"], "ResultPath": "$.error", "Next": "Cleanup" } ] — always ResultPath: "$.error" so the recovery state inherits input plus error. Route known errors to specific handlers and keep a States.ALL backstop last.
Map vs Distributed Map decision. Inline Map (Mode: INLINE) for arrays of hundreds to a few thousand items, ~40 concurrency, state held in the execution. Distributed Map (Mode: DISTRIBUTED) with ItemReader for S3-scale (tens of thousands+), up to 10,000 concurrent child executions, ItemBatcher to amortize overhead, ToleratedFailurePercentage to survive bad items, ResultWriter to S3.
Parallel vs Map. Parallel = a fixed set of different branches against one input (multi-target load, independent checks). Map = a variable set of the same sub-workflow over an array (per-file fan-out). Parallel is all-or-nothing at the state boundary — catch inside a branch for partial success.
Standard vs Express decision. Standard: up to 1 year, exactly-once, durable history, per-state-transition billing — pick for long, auditable, exactly-once workflows and anything with a durable pause. Express: up to 5 min, at-least-once (async) / at-most-once (sync), per-request+duration billing — pick for short, high-frequency, idempotent workloads. Default to Standard when unsure.
Direct SDK integration vs Lambda proxy. Prefer arn:aws:states:::<service>:<action> direct integrations (glue:startJobRun.sync, dynamodb:putItem, sns:publish, athena:startQueryExecution.sync) over a proxy Lambda per call — fewer functions to own, less code, native error surfacing. Reach for a Lambda only when you need custom logic the SDK integration cannot express.
.sync vs .waitForTaskToken. Use .sync to block on a long AWS job that has a run-a-job integration (Glue, ECS, Batch, EMR). Use .waitForTaskToken when the completion signal comes from outside — human approval, a partner webhook, a job with no .sync integration — and always pair it with TimeoutSeconds + a Catch on States.Timeout.
Cost-control checklist. Count state transitions before choosing Standard; nest Express inside Standard for high-volume inner work; cap MaxConcurrency on Maps; batch items with ItemBatcher; use Wait (free on Standard) instead of a polling loop; and turn on only the CloudWatch logging you will actually read (Express logging has a cost).
Idempotency + at-least-once contract. Every Task can be retried and, on async Express, re-delivered — so make each Task idempotent (dedupe by a business key, upsert not insert, no-op on already-applied compensations). Idempotency is not optional; it is the contract that makes retries and at-least-once delivery safe.
Observability defaults. Standard gives you the execution graph and event history for free — open the execution, find the red state, read its input and $.error. For Express, enable CloudWatch Logs (ALL or ERROR level) or you get no per-run detail. Emit a correlation id through the state so cross-service logs stitch together.

Frequently asked questions

What is AWS Step Functions in one sentence?

AWS Step Functions is a managed serverless orchestration service that lets you define a workflow as a state machine — a set of named states with transitions, written in the JSON-based Amazon States Language — and then runs it durably, handling the ordering, branching, parallelism, retries, and error handling between your Lambda functions and 220+ other AWS services. Instead of hand-wiring Lambdas to Lambdas through queues and cron (choreography), you get a single named workflow with durable state, declarative retry catch, and a replayable execution history. It is AWS's default answer to "how do I orchestrate serverless work" for event-driven and AWS-native pipelines.

Standard vs Express — when do I pick each?

Pick Standard for workflows that run long (up to a year), must be durable and exactly-once, need a full auditable execution history, or must pause for a callback — nightly ETL, order sagas, human-in-the-loop approvals. Pick Express for short (under 5 minutes), high-frequency, idempotent workloads — clickstream ingestion, IoT events, high-volume APIs — where per-request pricing is dramatically cheaper than Standard's per-state-transition billing (often 50–100× at scale). Express is at-least-once when async and at-most-once when sync, so its steps must be idempotent. When a durable outer process wraps high-volume inner work, nest Express children inside a Standard parent to get durability and cheap volume at once.

Inline Map vs Distributed Map — what is the difference?

Both are the Map state — the same sub-workflow run once per item of an array — but they scale differently. Inline Map (Mode: INLINE) keeps every iteration's state inside the single parent execution and caps concurrency around 40, so it fits arrays of hundreds to a few thousand items. Distributed Map (Mode: DISTRIBUTED) reads items from an ItemReader (an S3 object listing, CSV, or JSON manifest), runs each batch as its own child execution, and scales to 10,000 concurrent executions — the tool for processing tens of thousands to millions of S3 objects. Distributed Map adds ItemBatcher (amortize per-invocation overhead), MaxConcurrency (protect downstream), ToleratedFailurePercentage (survive bad items), and ResultWriter (aggregate results to S3).

How do retries and catch work together?

They are two layers of the same resilience model. A Retry array re-runs the same state with exponential backoff for the error classes you expect to be transient — throttles, timeouts, brief unavailability — controlled by IntervalSeconds, BackoffRate, MaxAttempts, MaxDelaySeconds, and JitterStrategy. A Catch array handles any error that either exhausts its retries or was never retryable, transitioning the workflow to a named recovery state (cleanup, compensation, notify, or Fail). The order is: SFN tries the state, applies matching retriers until they are exhausted, then hands the error to the first matching catcher. Always set ResultPath: "$.error" on a Catch so the recovery state inherits the original input plus the error, and add JitterStrategy: FULL on retries when many executions can retry at once to avoid a retry storm.

Can Step Functions replace Airflow?

Sometimes, but they solve overlapping-not-identical problems. Step Functions wins for serverless, AWS-native, event-driven, and bursty workloads: it is pay-per-use with no idle cluster, integrates natively with 220+ AWS services, and gives durable state plus built-in retry/catch. Airflow (or MWAA) wins for scheduled, Python-heavy DAGs with hundreds of community operators, dynamic task mapping, routine date-range backfills, and cross-cloud or on-prem tasks — and for teams already fluent in it. A large nightly analytics DAG with complex dependencies and frequent backfills is usually Airflow; an event-driven fan-out ETL is usually Step Functions. The pragmatic answer is often a hybrid: Airflow schedules and owns the big DAG, and calls Step Functions for the AWS-native leaf sub-workflows.

How does Step Functions handle exactly-once?

Standard workflows execute exactly-once: each state transition is durably recorded, so the workflow itself does not double-run steps, and you can rely on the ordering and history. Express workflows are different — asynchronous Express is at-least-once (a run may be retried) and synchronous Express is at-most-once — so on Express you must make every Task idempotent (dedupe by a business key, upsert rather than insert, treat repeated compensations as no-ops). Even on Standard, the underlying services a Task calls can be invoked more than once when a Retry fires after a partial success, so idempotency remains the safe default for any Task that mutates external state. Exactly-once at the workflow level is a Standard guarantee; end-to-end exactly-once still depends on idempotent Tasks.

Practice on PipeCode

Drill the ETL practice library → for the orchestration, pipeline-design, and retry/compensation problems senior interviewers love to pose around serverless workflows.
Rehearse on the data-processing practice library → for fan-out/fan-in, Map-style parallelism, and batch-reprocessing scenarios that mirror Distributed Map at scale.
Tune the knobs on the optimization practice library → for the concurrency, backoff, and cost-model trade-offs behind Standard vs Express and MaxConcurrency.
Stack the prerequisites against PipeCode's broader 450+ data-engineering catalogue to anchor the state-machine, Map, retry/catch, and Standard-vs-Express decisions against real graded inputs.

Lock in Step Functions orchestration muscle memory

Docs explain the states. PipeCode drills explain the decision — when a Map should go Distributed, when a Catch needs `ResultPath: "$.error"`, when Express is 50× cheaper than Standard, and when a saga's compensation chain earns its place. Pipecode.ai is Leetcode for Data Engineering — pattern-first practice tuned for the production trade-offs serverless data engineers actually face.

Practice ETL problems →
Practice data-processing problems →

1. Why orchestrate serverless work

Orchestration beats choreography once a pipeline has more than two steps that can fail

Worked example — the choreography-to-orchestration refactor

Worked example — Step Functions vs MWAA (Airflow) on a real decision

Data engineering interview question on serverless orchestration

Solution Using an orchestrated state machine with coordinated compensation

2. States & the Amazon States Language

A state machine is JSON — named states, one active at a time, wired by transitions

Worked example — a minimal Task → Choice → Succeed/Fail machine

Worked example — ResultPath plumbing that preserves accumulating context

Worked example — waitForTaskToken for a human-in-the-loop approval

Data engineering interview question on the Amazon States Language

Solution Using typed states, ResultPath accumulation, and a waitForTaskToken gate

3. Map & Parallel for fan-out ETL

Map fans one workflow across many items; Parallel fans many workflows across one input — both collect results as an array

Worked example — inline Map over a batch of files

Worked example — Distributed Map over an S3 prefix at scale

Worked example — Parallel branches for multi-target load

Data engineering interview question on fan-out ETL

Solution Using Distributed Map with batching, capped concurrency, and failure tolerance

4. Retries, Catch & error handling

Retry self-heals transient failures with backoff; Catch routes the unrecoverable ones to a cleanup branch

Worked example — exponential backoff on a throttled API

Worked example — Catch to a cleanup branch with error context

Worked example — the saga pattern for coordinated rollback

Data engineering interview question on error handling

Solution Using layered retries, jittered backoff, and a compensating catch

5. Standard vs Express + cost & patterns

express vs standard is a durability-versus-volume decision that also sets your bill

Worked example — pick the type from a workload table

Worked example — the cost comparison that justifies the choice

Worked example — nested Express inside Standard

Data engineering interview question on Standard vs Express

Solution Using a Standard parent with nested Express Distributed-Map children

Cheat sheet — AWS Step Functions recipes

Frequently asked questions

What is AWS Step Functions in one sentence?

Standard vs Express — when do I pick each?

Inline Map vs Distributed Map — what is the difference?

How do retries and catch work together?

Can Step Functions replace Airflow?

How does Step Functions handle exactly-once?

Practice on PipeCode

Lock in Step Functions orchestration muscle memory

`Map` fans one workflow across many items; `Parallel` fans many workflows across one input — both collect results as an array

`Retry` self-heals transient failures with backoff; `Catch` routes the unrecoverable ones to a cleanup branch

`express vs standard` is a durability-versus-volume decision that also sets your bill