Fernando Azevedo

Posted on Sep 1 Originally published at fernando.moretes.com

MWAA with Airflow 3.3.1: state, language, and operational discipline

#aiagents #aws #mwaa #airflow33

The September 1, 2026 announcement bringing Apache Airflow 3.3.1 to Amazon MWAA looks small if read as a version update. I do not read it that way. For financial environments, regulated data platforms, and operations with hundreds of DAGs, the central point is that Airflow is formally acknowledging three realities architects already solved outside the platform: tasks need to remember progress, not every domain implementation should become Python, and retries need to understand why a failure happened. MWAA reduces part of the operational burden of running schedulers, workers, webservers, and the metadata database, but it does not remove the obligation to design idempotency, isolation, cost control, and observability. My assessment: it is worth studying and adopting selectively, especially for incremental workloads and long-running jobs; it is not a reason to turn every DAG into an experimental SDK lab.

Numbers that shape the decision

3.3.1 — Airflow version now supported on MWAA. Apache Airflow 3.3.1 was released on August 12, 2026; MWAA support was announced on September 1, 2026.
25 — Workers per environment as the default quota. The documented MWAA quota lists 25 workers per environment, 5 webservers, and 10 environments per account per Region, all adjustable.
$0.49/h — Public example for a small environment in us-east-1. The pricing page uses $0.49 per hour for a small environment and $0.055 per hour for each additional small worker in a Northern Virginia example.

What actually changed

I split this version into two groups: what changes workflow modeling and what improves daily operations. The first group includes the Task and Asset State Store, expanded asset partitioning, pluggable retry policies, and the experimental Language Task SDK for Java and Go. The second group includes bulk actions for DAG runs and task instances, stability, security, and UI fixes, plus practical improvements for teams that operate many pipelines.

The State Store is the most architectural item. Until now, many teams used XCom, Variables, DynamoDB, S3, or auxiliary tables to keep cursors, external job ids, watermarks, and checkpoints. Some of those solutions were correct; others became technical debt with unclear semantics. Airflow 3.3 creates an explicit place for this state: task state for a task instance and asset state for metadata associated with an asset. That does not remove DynamoDB or external databases when the state is domain-owned, auditable, or shared by systems outside Airflow. But it improves DAG hygiene when the state is operational and belongs to the execution cycle.

On MWAA, this matters because the service already runs schedulers and workers on Fargate and maintains a managed metadata database. The temptation will be to fill the metadatabase with too much state. I would do the opposite: use the store for small, versioned, disposable pointers, and keep business data in S3, DynamoDB, Aurora, or the lakehouse.

Where I see value

External job checkpoints become cleaner: task state can store the job id for EMR Serverless, Glue, Batch, ECS, or SageMaker before polling completes.
Per-asset watermarks reduce coupling: a processed S3 partition or table can carry state without becoming a global Variable.
Pluggable retry policies help separate transient from permanent failures, especially for APIs with throttling, expired credentials, and contract validation.
Java and Go become options for existing domain logic, but I would treat the SDK as experimental until packaging, logs, and operational support are proven.
Bulk actions reduce operational friction in reprocessing, but they also call for stronger change controls and audit trails.

Durable state is not permission to forget idempotency

The main mistake I expect with Airflow 3.3 is turning the state store into an application database. In financial systems, a task that settles files, publishes events, calculates exposure, or reconciles positions cannot rely only on “remembering where it stopped.” It needs idempotency by design. That means natural keys, deduplication, conditional writes, and explicit reprocessing boundaries.

A design I would accept: a DAG receives file-arrival events, materializes metadata in DynamoDB with a partition key such as dataset#business_date and a sort key such as source_file#version, stores raw data in S3 with versioning and SSE-KMS, and uses task state only to keep the identifier of an already submitted Glue or EMR Serverless job. If the worker dies, the next attempt retrieves the job id, queries the job state, and decides whether to follow it, cancel it, or open a new attempt with an idempotency token. Durable state avoids duplicated work; the control table remains the auditable ledger.

I would also apply TTL or short retention to task states whenever possible. For asset cursors, I would use small payloads: last confirmed offset, contract hash, partition window, and producer version. If the value starts looking like a document, event, or snapshot, it probably does not belong in the state store.

How I would design safe State Store usage

The diagram shows my preferred pattern: Airflow keeps minimal operational state; auditable state and data remain in purpose-built services.

🟦 Orquestração MWAA

DAG Python asset-aware (compute)
Task State Store job_id, checkpoint (data)
Asset State Store watermark, partition (data)

🟧 Execução AWS

AWS Glue / EMR long-running job (compute)
Amazon S3 raw + curated data (storage)
DynamoDB control ledger (data)

🛡️ Governança e Operação

IAM + KMS least privilege (security)
CloudWatch logs, metrics, alarms (edge)
Runbook replay window (ci)

Flows

dag -> glue: submits job with idempotency token
dag -> taskstate: persists job_id before polling
glue -> s3: writes partitioned outputs
glue -> ddb: updates control ledger
dag -> assetstate: advances watermark after commit
iam -> dag: assumes restricted role
dag -> cw: emits duration, retry, and lag
cw -> runbook: triggers controlled replay

Java/Go SDKs: useful integration, maturity still under test

I like the direction of the Language Task SDK because it recognizes that enterprise data platforms are rarely 100% Python. There are pricing libraries in Java, contract validators in Go, internal clients with already reviewed security policies, and binaries that platform teams would rather not rewrite. The documented model keeps the DAG in Python and declares stub tasks with @task.stub(queue=...); the worker delegates execution to a coordinator, such as JavaCoordinator for JVM or ExecutableCoordinator for self-contained binaries such as Go.

The opportunity is to reduce fragile wrappers. Instead of calling a script through BashOperator and losing task semantics, retries, pools, and XCom behavior, the non-Python task participates in the graph. That improves readability and ownership: the data team keeps orchestration, the domain team keeps implementation.

But I would be conservative. The documentation itself marks this capability as experimental, and that matters. On MWAA, I would first validate whether binary or JAR packaging fits the plugin, requirements, managed image, and worker-directory flow. I would also test cold start, artifact size, remote logs, secrets through Connections, memory limits of the selected environment class, and rollback behavior. For regulated production, I would start with low-risk tasks, without irreversible side effects, and keep pure Python as the recovery path.

The hidden risk is the metadata database: The Airflow metadata database already carries DAG runs, task instances, XComs, indirect logs, serialization, and scheduler state. When adding the state store, I would monitor growth, cleanup, and cardinality from day one. On MWAA, pricing also includes metadata database storage in GB-months; in a platform with many dynamic mappings, per-partition cursors, and frequent reruns, the small cost may matter less than the operational impact of a bloated database.

MWAA operations: managed version is not outsourced operations

MWAA remains a pragmatic choice when an organization wants Airflow without managing Kubernetes, Celery, the metadata database, and image patching. The documentation confirms that schedulers and workers run on Fargate, with a managed Aurora PostgreSQL metadata database and integrations with CloudWatch, S3, SQS, and KMS. That removes undifferentiated work from the team, but it does not turn Airflow into an invisible service.

I would start any 3.3.1 upgrade with a parallel environment, not a direct change to the critical environment. Since the announcement allows a new 3.3.1 environment and upgrades from 3.2 or later, I would use the parallel path to validate dependencies, constraints, airflow.sdk imports, providers, and DAG serialization. The public MWAA version table also shows the lifecycle discipline: AWS supports at least three minor versions and announces end of support 180 days in advance, but the responsibility to keep environments current remains with the customer.

For capacity, I would not size by DAG count alone. Classes range from mw1.micro, with 3 default concurrent tasks and no autoscaling, to mw1.2xlarge, with 80 default concurrent tasks and much larger resources. Schedulers for Airflow v3 accept 2 to 5 in environments above micro, and workers have default 10, minimum 1, and maximum 25. Those numbers require load testing with real DAGs, not an optimistic spreadsheet.

Where to place state in the design

Criterion	Correct use	Avoid	Practical decision
Task State Store	External job id, retry checkpoint, small progress marker inside the task instance.	Business result, large payload, auditable event, or data contract.	Use it to resume execution; do not use it as the source of truth.
Asset State Store	Watermark, last confirmed partition, schema hash, and operational metadata for the asset.	Enterprise catalog, full lineage, quality rules, or authorization.	Good for DAG coordination; complement it with Glue Data Catalog, Lake Formation, or a lineage tool.
DynamoDB / Aurora	Auditable control, cross-system idempotency, locks, approval, SLA, and reconciliation.	Ephemeral state that only one task needs during a retry.	Use it when auditability, external query, or formal retention are requirements.

Pluggable retries change the failure contract

Retry is one of the places where data platforms lose money silently. A fixed retries=3 with a fixed delay looks harmless, but in transactional APIs it can triple cost, create duplicates, and mask permanent errors. In market, risk, fraud, or regulatory data workloads, I want the DAG to know the difference between ThrottlingException, network failure, AccessDenied, schema validation, and a semantic contract break.

Airflow 3.3 retry policies allow rules by exception type and actions such as retry, fail, or default behavior. That brings orchestration closer to an explicit operational policy. I would apply this around operators that call external APIs, managed jobs, and rate-limited integrations. For AWS SDK calls, I would still let low-level botocore retries handle short transient failures; the Airflow retry should be used for workflow decisions, with a delay compatible with the SLA and processing windows.

A healthy pattern: permission failure should fail fast and trigger a runbook; throttling should use backoff and alarm only after N minutes of backlog; schema error should open a contract incident and block downstream; external job failure should check remote state before resubmission. This detail reduces cost and improves trust in reprocessing.

How I would adopt it in production

Create a parallel 3.3.1 environment — I would clone DAGs, requirements, plugins, and configuration while keeping the current environment intact. I would validate constraints, imports, IAM permissions, Connections, pools, and serialization before any real traffic.
Classify DAGs by risk — I would separate read-only DAGs, idempotent DAGs, DAGs with side effects, and regulatory DAGs. The state store would enter first in incremental workflows with simple rollback.
Define the state contract — Each key needs an owner, JSON format, scope, retention, cleanup policy, and relationship to audit. I would reject free-form keys created inside tasks without convention.
Instrument before migrating — I would track task duration, queue time, scheduler lag, active workers, retries by cause, metadata database size, XCom growth, and state count.
Cut over by data window — Instead of migrating all DAGs, I would move one window or domain, freeze competing reprocessing, and keep a clear runbook for replay and rollback.

Anti-patterns I would avoid

Moving to 3.3.1 just to be on the newest version, without an inventory of DAGs, providers, and transitive dependencies.
Storing business payloads in the Task State Store and later trying to reconstruct audit from an orchestration metadata database.
Using the Java/Go SDK to bypass image governance, secret scanning, dependency review, and observability.
Increasing workers up to the default quota of 25 without checking whether the real bottleneck is scheduler, database, external API, pool, or downstream.
Allowing broad bulk clear or rerun without approval, change annotation, and an explicit reprocessing window.

Well-Architected reading

security: I would keep MWAA on private access when the audience is internal, use IAM scoped by environment, KMS for DAG/log/data buckets, and S3 conditional policies such as aws:SecureTransport and domain-specific prefixes. Connections should reference managed secrets, not credentials embedded in DAGs or bundles.
reliability: State Store improves recovery, but reliability comes from idempotency, controlled replay, pools, cause-aware retries, and domain isolation. For critical workloads, I would separate environments by criticality or domain to prevent experimental DAGs from pressuring shared schedulers and workers.
performance: MWAA classes should be selected by measurements: parse time, scheduler lag, task duration, CPU/memory use, and effective concurrency. Webserver scaling reacts to CPU above 70 or ActiveConnectionCount above 15, but that solves UI/API pressure, not processing bottlenecks.

My technical verdict before the final verdict

This version does not change my basic rule for Airflow: it should orchestrate, not do heavy processing inside the worker. On MWAA, I would still call Glue, EMR Serverless, Batch, ECS, EKS, Lambda, or specialized services for substantial work, leaving the DAG to coordinate dependencies, windows, contracts, execution state, and retry decisions. The State Store makes that coordination more honest, especially when a worker can die after submitting a remote job and before recording the result.

I also see a broader trend in recent AWS signals: managed services receiving finer operational controls. CloudWatch added alarm warm-up, DocumentDB received direct major-version upgrade, Redshift strengthened IAM Identity Center authentication with VPC routing, and now MWAA follows Airflow’s evolution in state and multi-language execution. The direction is clear: less undifferentiated operation, more responsibility for policy, governance, and change design.

For engineering leadership, the right conversation is not “should we enable Airflow 3.3.1?” The question is: which incidents, rework, and manual controls does this version remove without increasing risk? If the answer points to incremental reprocessing, long-running jobs, smarter retries, and clean separation of Python orchestration from Java/Go domain logic, there is concrete value. If the answer is only novelty, I would wait.

Curator note: I would adopt MWAA with Airflow 3.3.1 first in a data domain with frequent reprocessing and controlled impact, not in the company’s most sensitive flow. My practical lesson is that orchestrators fail less because of missing features and more because of poorly defined state, lazy retries, and diffuse ownership. The State Store is useful precisely because it forces the team to name the state; without discipline, it only moves the debt elsewhere. For Java and Go, I would wait for the team to prove packaging, logs, and rollback before calling it a standard.

Verified references

Verdict

My recommendation is selective adoption: strong for incremental pipelines, long-running jobs, recovery after worker failure, asset partitioning, and more explicit retry policies; cautious for Java/Go SDKs until the organization proves packaging, security, observability, and support. On an enterprise architecture scale, I would rate MWAA with Airflow 3.3.1 at 8/10 as a managed orchestration platform, provided the team keeps operational state small, auditable data outside Airflow, and an upgrade plan tested. I would not use this version as an excuse to put heavy logic inside workers or to replace data governance with DAG metadata.

Rating: 8/10

Originally published at fernando.moretes.com. By Fernando F. Azevedo — Senior Solutions Architect.

DEV Community