Level: 300–400 | Reading time: ~10 minutes
Recently, a customer asked me when they should choose Amazon Managed Workflows for Apache Airflow (Amazon MWAA) versus AWS Step Functions. I walked them through their use cases, pointed them to the recently published AWS comparison blog, and mentioned AWS Lambda durable functions as something worth considering for their new data platform.
That made me realise how many teams are asking the same question. The AWS blog covers MWAA and Step Functions well. It references Lambda durable functions at the end, pointing readers to the documentation for more detail. This post expands that comparison by bringing Lambda durable functions into the picture.
In this post, you will learn how Lambda durable functions, Step Functions, and MWAA differ in developer experience and architectural fit. You get five real scenarios, the same workflow implemented three ways, and a decision framework you can apply to your own workloads.
Prerequisites
Familiarity with at least one AWS compute service (Lambda, Amazon ECS, or Amazon EC2). A basic understanding of workflow orchestration concepts such as DAGs, state machines, and checkpoints. The AWS Durable Execution SDK installed for Python, TypeScript, or Java.
Lambda durable functions in 30 seconds
Lambda durable functions use a checkpoint and replay mechanism. Your function runs for up to one year, recovering from interruptions by replaying from the last saved checkpoint. You write orchestration in application code, not in Amazon States Language (ASL) and not in Airflow DAGs.
The mental model is simple. Step Functions lives outside your code. Durable functions live inside it.
AWS positions these three services for different orchestration patterns. Durable functions handle application-level orchestration. Step Functions handles cross-service orchestration. MWAA handles data pipeline orchestration. They complement each other.
Developer experience compared
Lambda durable functions give you code-first orchestration. You write workflow logic in Python, TypeScript, or Java using the Durable Execution SDK. You test locally without cloud dependencies. You debug with familiar stack traces. There is no visual designer. You read code, not diagrams.
AWS Step Functions gives you DSL-driven orchestration. You define workflows in ASL or use Workflow Studio's visual canvas. You get 200+ native service integrations without writing Lambda glue code. Non-technical stakeholders can follow execution visually. The trade-off is that complex ASL gets verbose and Workflow Studio has limits on larger state machines.
Amazon MWAA gives you scheduler-driven orchestration. You write Python DAGs using the Apache Airflow ecosystem. You get scheduling, backfills, and SLA monitoring as first-class features. The operator ecosystem covers AWS and non-AWS systems. The trade-off is environment provisioning time and a baseline cost that only makes sense when you run multiple pipelines.
None of these services is universally better. Each fits a different architectural pattern.
Real scenarios: where each service wins
Scenario 1: Multi-step AI agent workflow
A customer submits a support ticket. You classify intent via Amazon Bedrock, extract entities, route to the right team, generate a draft response, wait for human review, then send the reply.
Durable functions fit naturally here. The logic is sequential, the human-in-the-loop wait costs nothing (no compute during the pause), and the entire workflow reads like a script. Step Functions also works well because of native Bedrock integration and visual tracing. MWAA is awkward for this pattern. There is no scheduling need, no backfill requirement, and you would force Airflow into a request/response pattern for which it was not designed.
Trade-off: Durable functions win on developer experience and testability. Step Functions wins on native integrations and visual tracing.
Scenario 2: Nightly data pipeline with conditional branching
Every night at 02:00, you pull data from three APIs, validate the schema, branch conditionally (trigger an AWS Glue crawler if the schema changed, skip if not), transform, load to Amazon Redshift, notify Slack, and support backfills for missed days.
MWAA wins clearly. Scheduling, backfill, SLA monitoring, Glue operators, and Redshift operators are all first-class. Step Functions works but you bolt on Amazon EventBridge for scheduling and build custom backfill logic. Durable functions are a poor fit here. No native scheduling, no backfill capability.
Winner: MWAA.
Scenario 3: E-commerce order fulfilment with parallel fan-out
An order arrives. You validate payment, reserve inventory, then fan out in parallel: generate an invoice to Amazon S3, notify the warehouse via Amazon SQS, send confirmation via Amazon SES, and update the CRM in Amazon DynamoDB. You wait for all branches to complete, then mark the order confirmed.
Step Functions wins clearly. The Parallel state handles fan-out natively. Each branch uses a different service integration directly without Lambda in between. Durable functions work for the sequential parts but parallel fan-out across multiple services requires additional orchestration. MWAA is overkill for this pattern.
Winner: Step Functions.
Scenario 4: Long-running document processing with human approval
A user uploads a contract PDF. You extract text with Amazon Textract, summarise clauses via Bedrock, flag risky clauses, then wait for legal team approval (which could take days). If approved, store and index. If rejected, notify the uploader with reasons.
Durable functions are strong here. The wait costs nothing. The logic reads top-to-bottom. Testing the approval branch is a unit test. Step Functions is also strong. The .waitForTaskToken pattern handles the human gate, and native Textract and Bedrock integrations avoid Lambda glue. MWAA is a poor fit because sensors polling for approval waste worker resources.
Trade-off: Durable functions win on developer experience. Step Functions wins on native integrations and visual audit trail.
Scenario 5: Multi-account infrastructure provisioning
A new team onboards. You create an AWS account via AWS Organizations, deploy baseline AWS CloudFormation stacks, configure networking, set up AWS Identity and Access Management (IAM) roles, wait for security team approval, enable Amazon GuardDuty and AWS Config, then notify via Amazon SNS.
Step Functions wins clearly. Every step is an AWS service call. Native integrations for Organizations, CloudFormation, and IAM mean zero custom code. The visual audit trail supports compliance requirements. Durable functions add no value here because you would write SDK calls for everything that Step Functions already integrates natively. MWAA is the wrong tool entirely.
Winner: Step Functions.
The scorecard
No single service dominates. That is the point.
- Scenario 1: Durable functions ≈ Step Functions (trade-off)
- Scenario 2: MWAA
- Scenario 3: Step Functions
- Scenario 4: Durable functions ≈ Step Functions (trade-off)
- Scenario 5: Step Functions
Same workflow, three implementations
Taking Scenario 1 (AI agent workflow), here is the same logic implemented three ways.
Code samples in this post are illustrative. Refer to the AWS Durable Execution SDK documentation for the latest API syntax.
Lambda durable functions (Python)
from aws_durable_execution_sdk_python.config import Duration
from aws_durable_execution_sdk_python.context import DurableContext, StepContext, durable_step
from aws_durable_execution_sdk_python.execution import durable_execution
@durable_step
def classify_intent(step_context: StepContext, ticket: dict) -> str:
return call_bedrock_classify(ticket)
@durable_step
def extract_entities(step_context: StepContext, ticket: dict) -> dict:
return call_bedrock_extract(ticket)
@durable_step
def route_to_team(step_context: StepContext, intent: str, entities: dict) -> str:
return determine_team(intent, entities)
@durable_step
def generate_draft(step_context: StepContext, ticket: dict, intent: str) -> str:
return call_bedrock_draft(ticket, intent)
@durable_step
def send_reply(step_context: StepContext, ticket: dict, draft: str) -> None:
dispatch_reply(ticket, draft)
@durable_execution
def lambda_handler(event, context: DurableContext) -> dict:
ticket = event["ticket"]
intent = context.step(classify_intent(ticket))
entities = context.step(extract_entities(ticket))
team = context.step(route_to_team(intent, entities))
draft = context.step(generate_draft(ticket, intent))
# Wait for human review window (24 hours max)
context.wait(Duration.from_hours(24))
context.step(send_reply(ticket, draft))
return {"statusCode": 200, "body": f"Ticket routed to {team}, reply sent"}
Step Functions (ASL excerpt)
{
"StartAt": "ClassifyIntent",
"States": {
"ClassifyIntent": {
"Type": "Task",
"Resource": "arn:aws:states:::bedrock:invokeModel",
"Next": "ExtractEntities"
},
"ExtractEntities": {
"Type": "Task",
"Resource": "arn:aws:states:::bedrock:invokeModel",
"Next": "RouteToTeam"
},
"RouteToTeam": {
"Type": "Task",
"Resource": "arn:aws:lambda:...:route-to-team",
"Next": "GenerateDraft"
},
"GenerateDraft": {
"Type": "Task",
"Resource": "arn:aws:states:::bedrock:invokeModel",
"Next": "WaitForReview"
},
"WaitForReview": {
"Type": "Task",
"Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
"Next": "SendReply"
},
"SendReply": {
"Type": "Task",
"Resource": "arn:aws:states:::ses:sendEmail",
"End": true
}
}
}
Amazon MWAA (Airflow DAG excerpt)
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.sensors.external_task import ExternalTaskSensor
with DAG("ticket_workflow", schedule=None) as dag:
classify = PythonOperator(task_id="classify", python_callable=call_bedrock_classify)
extract = PythonOperator(task_id="extract", python_callable=call_bedrock_extract)
route = PythonOperator(task_id="route", python_callable=route_to_team)
draft = PythonOperator(task_id="draft", python_callable=call_bedrock_draft)
review = ExternalTaskSensor(task_id="wait_review", timeout=86400)
send = PythonOperator(task_id="send", python_callable=send_reply)
classify >> extract >> route >> draft >> review >> send
The durable functions version is more verbose than a pseudocode sketch, but each step is explicit, testable, and replay-safe. The ASL version is 35 lines. The Airflow DAG is 12 lines. But line count is not the full picture. Step Functions gives you a visual execution trace without writing a single log statement. MWAA gives you scheduling and backfill without writing a single cron job. Each trade-off is real.
Gotchas I learned the hard way
Replay non-determinism catches you off guard. Durable functions replay from the last checkpoint on recovery. Any non-deterministic call inside the orchestration logic produces a different result on replay. This includes datetime.now(), random(), and HTTP calls. Perform non-deterministic operations inside durable steps and keep your orchestration logic deterministic.
# This breaks on replay — called inside orchestration logic, not a step
timestamp = datetime.now()
# This works — wrapped in a durable step
@durable_step
def get_timestamp(step_context: StepContext) -> str:
return datetime.now().isoformat()
Checkpoint frequency involves trade-offs. Recovery speed, replay overhead, and persistence cost all pull in different directions. The right balance depends on your workload. Test it rather than assuming a universal rule.
MWAA environment provisioning takes time. Spinning up a new environment typically takes 20 to 30 minutes. Iteration speed is slower compared to durable functions or Step Functions where you deploy in seconds.
Large Step Functions workflows become difficult to manage visually. Workflow Studio works well for smaller state machines. As complexity grows, many teams eventually edit ASL directly. Plan for this from the start if your workflow is complex.
Portability varies. MWAA runs on open-source Apache Airflow. Your DAGs work on any compatible Apache Airflow deployment, though provider-specific operators may still need changes. Durable functions use the AWS Durable Execution SDK, which is AWS-specific, though the checkpoint/replay pattern itself is transferable. Step Functions uses ASL, which has no equivalent outside AWS.
Decision framework
Choose durable functions when your team owns the code and wants orchestration embedded in application logic. When you value local testability and fast iteration. When the workflow is application-centric rather than cross-service infrastructure glue.
Choose Step Functions when you orchestrate multiple AWS services without custom code. When non-technical stakeholders need visual workflow visibility. When you want native integrations (DynamoDB, SQS, Bedrock) without Lambda in between. When you need declarative error handling with retry, catch, and fallback patterns.
Choose MWAA when you run complex data pipelines with scheduling, backfills, and SLA monitoring. When your team already knows Airflow and its operator ecosystem. When you need dependency-aware DAGs across dozens of tasks. When you coordinate across AWS and non-AWS systems.
Choose a combination when the real world demands it. Step Functions for cross-service orchestration with a durable function handling complex logic within a single task. MWAA for scheduling with Step Functions handling individual pipeline steps. Durable functions for core business logic with Step Functions for parallel fan-out.
Conclusion
Lambda durable functions, Step Functions, and MWAA serve different developer preferences and architectural patterns. Durable functions bring orchestration into your application code. Step Functions orchestrates across AWS services visually. MWAA handles complex scheduled data pipelines. They complement each other rather than compete.
Start by identifying your orchestration pattern. Match it to the service that fits. Combine them when a single service does not cover the full workflow.
Further reading:
- AWS Durable Execution SDK Developer Guide
- AWS Step Functions Developer Guide
- Amazon MWAA User Guide
- Choosing the right workflow orchestration service (AWS Big Data Blog)
Questions or a scenario you want me to map? Drop them in the comments.
Top comments (0)