AWS MWAA Serverless: Is It Worth the Switch from Provisioned?

#aws #mwaa #dataengineering

In the world of Data Engineering, building data pipelines is unavoidable. But what’s even more critical is the Orchestrator. Why? Because we can't (and shouldn't) manually trigger every step of a pipeline. Automation is a necessity.

Today, we have a massive variety of orchestrators to choose from:

Open Source: Apache Airflow (the most popular), Dagster, and Prefect.
Managed/Closed Source: Cloud-native platforms like Google Cloud (Cloud Composer), Microsoft Azure, and Databricks.

Then there’s Amazon Web Services (AWS) with its managed Airflow offering: Amazon MWAA.

The Arrival of AWS MWAA Serverless

At the end of 2025, AWS introduced MWAA Serverless. The core concept is simple: allowing Data Engineers to focus on workflows without worrying about infrastructure management.

In the Provisioned version (non-serverless), AWS spins up dedicated infrastructure behind the scenes. This means you incur costs as long as the environment is active, even if no tasks are running. MWAA Serverless aims to change that.

*Orchestration on AWS: MWAA vs. Step Functions
It’s worth noting that MWAA isn’t the only orchestrator on AWS; AWS Step Functions is also a very popular choice. I’ll do a deep dive comparison in a future article, but for now, let's focus on MWAA.

A Quick Airflow Refresher
Before we dive in, remember that in Airflow, we write scripts called DAGs (Directed Acyclic Graphs). A DAG defines the structure of your workflow, and inside each DAG are Tasks—the smallest unit of work.

Amazon MWAA Provisioned vs. Serverless

1. Amazon MWAA Provisioned
This is essentially Apache Airflow deployed on AWS. AWS handles the compute, storage, and database, ensuring scalability, availability, and security. It feels exactly like the Airflow you’d deploy on your own server.

2. Amazon MWAA Serverless
This is the main highlight. With Serverless, you don’t set up infrastructure at all. You just drop your DAG into an S3 bucket and you're ready to go.

The Catch: MWAA Serverless only reads YAML files. If you have existing Python-based DAGs, you must convert them to YAML first. (Don't worry, AWS provides a library for this!). Alternatively, you can build DAGs using a drag-and-drop GUI within Amazon SageMaker.

How to use Amazon MWAA Serverless

There are two primary ways to create DAGs:

Method 1: Writing YAML DAGs manually
You can write your DAG in VS Code (or any editor) and upload it to S3.

Example YAML DAG:

simples3test:
  dag_id: simples3test
  tasks:
    list_objects:
      operator: airflow.providers.amazon.aws.operators.s3.S3ListOperator
      bucket: your-s3-bucket-name
      prefix: ""
      retries: 0
    create_object_list:
      dependencies:
        - list_objects
      operator: airflow.providers.amazon.aws.operators.s3.S3CreateObjectOperator
      data: Hello MWAA serverless
      s3_bucket: mwaa-serverless-test-cdlb
      s3_key: demo_text.txt
      replace: true
  schedule: 0 0 * * *

Once uploaded, you use the AWS CLI to register and execute the workflow:

# Create workflow
aws mwaa-serverless create-workflow \
--name simple_s3_test \
--definition-s3-location '{ "Bucket": "your-s3-bucket-name", "ObjectKey": "path/simple_s3_test.yaml" }' \
--role-arn arn:aws:iam::111122223333:role/mwaa-serverless-access-role \
--region us-east-1

# Execute the workflow
aws mwaa-serverless start-workflow-run \
--workflow-arn arn:aws:airflow-serverless:us-east-2:111122223333:workflow/simple_s3_test-abc1234def \
--region us-east-1

# Update workflow if you have made any changes.
aws mwaa-serverless update-workflow \
--workflow-arn arn:aws:airflow-serverless:us-east-2:111122223333:workflow/simple_s3_test-abc1234def \
--definition-s3-location '{ "Bucket": "your-s3-bucket-name", "ObjectKey": "path/simple_s3_test.yaml" }' \
--role-arn arn:aws:iam::111122223333:role/mwaa-serverless-access-role \
--region us-east-1

Method 2: AWS SageMaker Workflow (GUI)
You can create workflows visually. In the MWAA Serverless console, click "Create workflow," and it will redirect you to the SageMaker Canvas/Workflow interface. You can drag and drop tasks like "Save file to S3" or "Trigger Glue Job." It’s incredibly convenient.

Extra: Converting Python DAGs to YAML
If you want to move your existing dag.py files to MWAA Serverless, AWS provides a converter library:

# AWS Python to Yaml Dag Converter for MWAA Serverless
pip install python-to-yaml-dag-converter-mwaa-serverless

# To convert .py to .yaml
dag-converter convert <python-dag-file>

Same Name, Different Game?

While "Serverless" sounds like an upgrade, it is fundamentally different from the standard Airflow experience.

Key Differences & Limitations:
Limited Operators: MWAA Serverless focuses almost exclusively on AWS Operators (e.g., S3, Bedrock, Batch, Glue). Classic operators like PythonOperator or BashOperator are not supported. This is because AWS wants the heavy lifting (compute) to happen in other services, not within the serverless orchestrator itself.

Missing Parameters: Many standard Airflow parameters are unavailable. If you rely on things like email_on_failure, catchup, or on_success_callback, you’ll need to rethink your logic using other AWS services (like SNS for notifications).

Final Verdict

Go with MWAA Serverless if: You are building a "pure" AWS workflow, you want something easy to set up, and you prefer a GUI or simple YAML definitions.

Stay with MWAA Provisioned if: You are migrating a complex, existing Airflow environment. If your DAGs use dynamic tasks, DAG-trigger-DAG patterns, or custom Python logic within the operators, Serverless will be more of a headache than a help.

MWAA Serverless and Provisioned share a name and a concept, but they serve very different use cases. Choose the one that fits your architecture, not just the "Serverless" buzzword!

REF:https://docs.aws.amazon.com/mwaa/latest/mwaa-serverless-userguide/mwaas-concepts.html
🌐 Note: This article was translated from Thai with the help of AI to share these insights with the global community. You can find the original Thai version on my page, here: Clouddatalabor