Amazon Managed Workflows for Apache Airflow (MWAA) sends Airflow metrics to Amazon CloudWatch without any setup. Airflow v2 and v3 environments are preconfigured to emit StatsD metrics, which appear in the AmazonMWAA namespace. A second namespace, AWS/MWAA, holds infrastructure metrics from the Airflow containers and from the queue and database behind them.
The data is available from the first day. The difficulty is in reading it. CloudWatch graphs work for a single metric, but a scheduler health view needs heartbeat, parse time, queue depth, and worker memory on one screen. Amazon Managed Grafana provides that screen and reads CloudWatch directly.
There is also a development problem. The Compose file in the aws/amazon-mwaa-docker-images repository starts a local Airflow cluster with Postgres and a local SQS substitute. It contains two TODO comments that describe the missing piece: no local CloudWatch endpoint exists, so a laptop environment shows no metrics at all. A change to a DAG or to a configuration option cannot be checked against a dashboard before it reaches a real environment.
This article builds a deployment metrics viewer in two parts. The first part adds a StatsD exporter, Prometheus, and Grafana to the local Compose stack. The second part creates an Amazon Managed Grafana workspace and connects it to the CloudWatch metrics of a deployed MWAA environment.
What MWAA publishes
Metrics fall into two groups. The Airflow application metrics come from StatsD, while the infrastructure metrics come from the services that MWAA runs on behalf of the customer.
| Namespace | Source | Dimensions | Example metrics |
|---|---|---|---|
AmazonMWAA |
Airflow StatsD | DAG, DAG Filename, Function, Job, Operator, Pool, Task, HostName |
SchedulerHeartbeat, QueuedTasks, RunningTasks, TotalParseTime, ImportErrors
|
AWS/MWAA |
Containers, SQS queue, Aurora database | Cluster, Queue, Database |
CPUUtilization, MemoryUtilization, ApproximateAgeOfOldestTask, DatabaseConnections
|
The AmazonMWAA namespace lists a long set of metrics, and the set depends on the Airflow version. For example, TaskCpuUsage and TaskMemoryUsage need v2.10.1 or later. The AWS/MWAA namespace is the same for every supported version.
A large metric set raises the CloudWatch bill. For that reason, MWAA accepts two Airflow configuration options, metrics.metrics_allow_list and metrics.metrics_block_list, which take comma-separated prefixes. If both options are set, the block list is ignored. Both options apply to Airflow v2.6.3 and later.
Run MWAA locally
The amazon-mwaa-docker-images repository supports Airflow 2.9.2 and later, with one directory per version under images/airflow. This article uses 2.11.2. Python 3.11 or later and Docker with Docker Compose are required.
git clone https://github.com/aws/amazon-mwaa-docker-images.git
cd amazon-mwaa-docker-images
python3 create_venvs.py --target development --version 2.11.2
cd images/airflow/2.11.2
./run.sh
The script builds the image, creates a Fernet key, and ends with compose up. After startup, the Airflow web server answers at http://localhost:8080. No AWS account is needed for this step, because the script uses placeholder credentials and a local SQS substitute named ElasticMQ.
Place DAG files in the dags folder of the version directory. The folder is mounted into the containers, so changes appear without a rebuild.
Add a metrics stack
Docker Compose merges a file named docker-compose.override.yaml into docker-compose.yaml automatically. Because run.sh calls compose up without a file argument, an override file adds services without any change to the script.
Airflow sends StatsD packets over UDP. A StatsD exporter receives them and exposes the values in the Prometheus format, Prometheus stores them, and Grafana draws them. Four files are needed in the images/airflow/2.11.2 directory.
The first file is the override.
x-statsd-config: &statsd-config
MWAA__CORE__CUSTOM_AIRFLOW_CONFIGS: >-
{"AIRFLOW__METRICS__STATSD_ON": "True",
"AIRFLOW__METRICS__STATSD_HOST": "statsd-exporter",
"AIRFLOW__METRICS__STATSD_PORT": "9125",
"AIRFLOW__METRICS__STATSD_PREFIX": "airflow"}
services:
statsd-exporter:
image: prom/statsd-exporter:latest
command:
- --statsd.mapping-config=/etc/statsd/mapping.yaml
- --statsd.listen-udp=:9125
- --web.listen-address=:9102
volumes:
- ./statsd-mapping.yaml:/etc/statsd/mapping.yaml:ro
ports:
- 9102:9102
prometheus:
image: prom/prometheus:latest
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
ports:
- 9090:9090
depends_on:
- statsd-exporter
grafana:
image: grafana/grafana:latest
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning:ro
ports:
- 3000:3000
depends_on:
- prometheus
scheduler:
environment: *statsd-config
depends_on:
- statsd-exporter
worker:
environment: *statsd-config
depends_on:
- statsd-exporter
webserver:
environment: *statsd-config
The local image reads MWAA__CORE__CUSTOM_AIRFLOW_CONFIGS as a JSON object of Airflow environment variables, and the base Compose file sets it to an empty object. The override replaces that value for the scheduler, worker, and web server. The StatsD host is the service name statsd-exporter, which Compose resolves on its internal network.
This setup applies to the local runner only. In a deployed MWAA environment, the service forces StatsD on, sends packets to a local sidecar on port 8125, and does not allow the customer to change the host or port. The two options that remain under customer control are the allow list and the block list.
The second file maps selected StatsD names to Prometheus metrics with labels. Names that have no mapping keep a Prometheus name derived from the StatsD name with dots replaced by underscores, for example airflow_scheduler_heartbeat.
mappings:
- match: "airflow.dagrun.duration.success.*"
name: "airflow_dagrun_duration_success_seconds"
labels:
dag_id: "$1"
- match: "airflow.dagrun.duration.failed.*"
name: "airflow_dagrun_duration_failed_seconds"
labels:
dag_id: "$1"
- match: "airflow.ti.finish.*.*.*"
name: "airflow_ti_finish_total"
labels:
dag_id: "$1"
task_id: "$2"
state: "$3"
- match: "airflow.pool.open_slots.*"
name: "airflow_pool_open_slots"
labels:
pool: "$1"
- match: "airflow.dag_processing.last_duration.*"
name: "airflow_dag_processing_last_duration_seconds"
labels:
dag_file: "$1"
The third and fourth files are short. Save the first as prometheus.yml and the second as grafana/provisioning/datasources/prometheus.yaml.
global:
scrape_interval: 15s
scrape_configs:
- job_name: airflow-statsd
static_configs:
- targets: ["statsd-exporter:9102"]
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
Run ./run.sh again. Once the scheduler has started, the exporter publishes values at http://localhost:9102/metrics. A quick check confirms that Airflow is sending data.
curl -s localhost:9102/metrics | grep airflow_scheduler_heartbeat
Grafana runs at http://localhost:3000, and the sign-in is admin with the password admin. These values suit a laptop and nothing else.
Docker Desktop lists the original MWAA containers together with the three added services under one Compose project.
Build the first local panels
Three queries cover the most common questions about a scheduler. Add each as a time series panel on the Prometheus data source.
| Question | PromQL |
|---|---|
| Is the scheduler alive? | increase(airflow_scheduler_heartbeat[5m]) |
| Is work waiting for a worker? | airflow_executor_queued_tasks |
| Which DAGs fail? | sum by (dag_id, state) (increase(airflow_ti_finish_total[1h])) |
Trigger a DAG that fails on purpose and look at the third panel. A DAG change that raises the failure count or delays the heartbeat is visible before the change leaves the laptop. That is the main value of this half of the setup.
The local stack has one limit. The AWS/MWAA metrics come from Amazon ECS and Amazon SQS and from the Aurora database, so they have no local equivalent. Worker memory and queue age can be observed only in a deployed environment.
Create an Amazon Managed Grafana workspace
Amazon Managed Grafana authenticates users through AWS IAM Identity Center, SAML 2.0, or both. The permission type of the workspace determines how the CloudWatch data source is configured, and the console and the API behave differently here.
The console supports service-managed permissions. With this option, AWS creates the IAM role and policies that the workspace needs to read CloudWatch in the account or in an organizational unit. The documentation for the AWS data source configuration states that service-managed permissions are required for that route.
Calls made through the API accept only customer-managed permissions, and the CLI and CloudFormation use the same API. In that case the caller must supply an IAM role that the workspace assumes. The trust policy names the Grafana service as the principal.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": "grafana.amazonaws.com" },
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": { "aws:SourceAccount": "111122223333" }
}
}
]
}
The permissions policy must allow read access to CloudWatch metrics.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:DescribeAlarmsForMetric",
"cloudwatch:GetMetricData",
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics"
],
"Resource": "*"
}
]
}
With the role in place, the workspace can be created from the command line.
aws grafana create-workspace \
--workspace-name mwaa-observability \
--account-access-type CURRENT_ACCOUNT \
--authentication-providers AWS_SSO \
--permission-type CUSTOMER_MANAGED \
--workspace-role-arn arn:aws:iam::111122223333:role/GrafanaMwaaReadRole
The response contains the workspace identifier, its endpoint, and a status of CREATING. IAM Identity Center users must be assigned to the workspace before they can sign in.
Connect CloudWatch
With a service-managed workspace, the connection takes a few console steps. Open the workspace in the Amazon Managed Grafana console, choose the Data sources tab, select Amazon CloudWatch, and choose Actions and then Enable service-managed policy. In the same tab, choose Configure in Grafana, sign in through IAM Identity Center, and open AWS services and then CloudWatch from the AWS icon. Select the Region where the MWAA environment runs and the accounts to include, and then choose Add data source.
With a customer-managed workspace, add the CloudWatch data source manually in Grafana and select the workspace IAM role as the authentication method. The query editor works the same way in both cases.
Build the CloudWatch dashboard
The metric names differ from the local names. Airflow emits scheduler.heartbeat over StatsD, and CloudWatch stores the same value as SchedulerHeartbeat. A panel for each question from the local setup exists in CloudWatch, and the infrastructure panels have no local counterpart.
| Question | Namespace | Metric | Statistic |
|---|---|---|---|
| Is the scheduler alive? | AmazonMWAA |
SchedulerHeartbeat |
Sum |
| Is work waiting for a worker? | AmazonMWAA |
QueuedTasks |
Maximum |
| Are DAG files slow to parse? | AmazonMWAA |
TotalParseTime |
Average |
| Do DAG files contain errors? | AmazonMWAA |
ImportErrors |
Maximum |
| How old is the oldest queued task? | AWS/MWAA |
ApproximateAgeOfOldestTask |
Maximum |
| Are workers short of memory? | AWS/MWAA |
MemoryUtilization |
Average |
| Is the metadata database busy? | AWS/MWAA |
DatabaseConnections |
Maximum |
SchedulerHeartbeat is a counter, so a flat line at zero for several minutes indicates a scheduler that has stopped. ApproximateAgeOfOldestTask grows whenever tasks wait in the queue, which makes it a direct signal of a worker shortage. Both panels suit an alert rule.
The AWS/MWAA page also describes how to count additional workers. Graph CPUUtilization for the AdditionalWorker component of the Cluster dimension, set the period to one minute, and use the Sample Count statistic. The result equals the number of running tasks.
The finished dashboard resembles the following illustration. The values are sample data, so a real environment will show different numbers.
After the panels are saved, the dashboard exists only in the workspace. Export its JSON into the same repository as the DAGs, so that a dashboard change goes through the same review as a code change.
Keep the two views aligned
The local dashboard and the CloudWatch dashboard answer the same questions with different metric names. A short table in the repository that lists each pair avoids confusion later. Two pairs from this article are airflow_scheduler_heartbeat with SchedulerHeartbeat, and airflow_executor_queued_tasks with QueuedTasks.
Care is needed with the allow list. A prefix that is blocked in MWAA also disappears from the CloudWatch dashboard, while the local stack keeps receiving it. A panel that works locally and shows no data in CloudWatch usually points to a block list entry.
Where this leaves you
The local stack gives fast feedback on scheduler behavior, and Amazon Managed Grafana gives the same questions a permanent home next to the infrastructure metrics. Neither half needs a change to the DAGs. The local half needs one override file and three small configuration files, and the cloud half needs a workspace with a read-only role.
Top comments (0)