DEV Community

Ricardo Sueiras for AWS

Posted on

Using Terraform to configure and deploy Managed Workflows for Apache Airflow (MWAA) environments

Terraform is an open-source infrastructure as code software tool created by HashiCorp. Even though I am very familiar with it and have featured some of the really cool modules and content within my blog, I have never really used Terraform before. When I heard that the AWS-IA team had put together a new Managed Workflows for Apache Airflow (MWAA) module for Terraform, I knew the time had finally come to give this a go. I spent some time last week playing around with this, testing it and adding some tweaks to the docs. I thought I would use that time to take some notes and then put together this blog post.

So read on if you want to see how you can use Terraform to automate the configuration and deployment of your MWAA environments.

architecture overview of mwaa

Installing Terraform

First of all, need to install it.

HashiCorp's documentation is great, and I was able to install Terraform via this page.

brew tap hashicorp/tap
brew install hashicorp/tap/terraform
Enter fullscreen mode Exit fullscreen mode

which on my mac, produced output that looked like this

==> Tapping hashicorp/tap
Cloning into '/usr/local/Homebrew/Library/Taps/hashicorp/homebrew-tap'...
remote: Enumerating objects: 2210, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (34/34), done.
remote: Total 2210 (delta 64), reused 69 (delta 60), pack-reused 2116
Receiving objects: 100% (2210/2210), 386.46 KiB | 752.00 KiB/s, done.
Resolving deltas: 100% (1373/1373), done.
Tapped 1 cask and 18 formulae (51 files, 540.4KB).
(base)  @094459  ~  brew tap hashicorp/tapbrew install hashicorp/tap/terraform
(base)  ✘ @094459  ~  brew install hashicorp/tap/terraform
==> Downloading
######################################################################## 100.0%
==> Installing terraform from hashicorp/tap
🍺  /usr/local/Cellar/terraform/1.2.3: 3 files, 67.4MB, built in 7 seconds
==> Running `brew cleanup terraform`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

terraform -version

Terraform v1.2.3
on darwin_amd64
Enter fullscreen mode Exit fullscreen mode

As I tend to use Visual Studio Code, I also installed the extension that HashiCorp provide to work with Terraform files within Code.

Creating our MWAA Terraform configuration

The MWAA Terraform module can be found at Clone the repo into your local workspace.

git clone
Enter fullscreen mode Exit fullscreen mode

From the examples/basic folder, we have a simple MWAA stack that we can use in order to test that everything is working.

├── dags
│   └──
├── mwaa
│   └── requirements.txt
Enter fullscreen mode Exit fullscreen mode

The contains a quickstart on how to deploy your first environment. This post will walk you through the steps, but I still suggest you check that out. There are a number of key files that you will need to understand before you deploying your first MWAA environment.

mwaa and dags folders

You will notice that we have a sample Apache Airflow DAG ( that we want to deploy as we build our MWAA environment. We also have a requirements.txt file that we will want to upload (it is currently empty).

This example shows you how you can do this as we will see a bit later on. For the moment, all you need to be aware of is that these are resources that you want to deploy as you build out your MWAA environment.

This file contains configuration options that you can alter to change your MWAA environment - the name of the environment, the AWS region and default tags. For this demo, these are the values I am using.

variable "name" {
  description = "Name of MWAA Environment"
  default     = "terraform-mwaa"
  type        = string

variable "region" {
  description = "region"
  type        = string
  default     = "eu-central-1"

variable "tags" {
  description = "Default tags"
  default     = {"env": "test", "dept": "AWS Developer Relations"}
  type        = map(string)

variable "vpc_cidr" {
  description = "VPC CIDR for MWAA"
  type        = string
  default     = ""
Enter fullscreen mode Exit fullscreen mode

The contains the main Terraform configuration file that will deploy resources, using values that are contained in the Lets take a look at this ( in more detail.

At the top of the file we have the following, the important value here is "bucket_name" which will configure a unique S3 bucket which will be used by your MWAA environment. This is important as the subsequent uploading of the sample DAGs, the requirements.txt as well as all the IAM policy documents all use this value.

locals {
  azs         = slice(data.aws_availability_zones.available.names, 0, 2)
  bucket_name = format("%s-%s", "aws-ia-mwaa", data.aws_caller_identity.current.account_id)
Enter fullscreen mode Exit fullscreen mode

Next in the file, we have the section that creates and uploads the sample DAG and requirements files

# Create an S3 bucket and upload sample DAG
#tfsec:ignore:AWS017 tfsec:ignore:AWS002 tfsec:ignore:AWS077
resource "aws_s3_bucket" "this" {
  bucket = local.bucket_name
  tags   = var.tags

resource "aws_s3_bucket_acl" "this" {
  bucket =
  acl    = "private"

resource "aws_s3_bucket_versioning" "this" {
  bucket =
  versioning_configuration {
    status = "Enabled"
resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
  bucket =

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"

resource "aws_s3_bucket_public_access_block" "this" {
  bucket                  =
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true

# Upload DAGS
resource "aws_s3_object" "object1" {
  for_each = fileset("dags/", "*")
  bucket   =
  key      = "dags/${each.value}"
  source   = "dags/${each.value}"
  etag     = filemd5("dags/${each.value}")

# Upload plugins/requirements.txt
resource "aws_s3_object" "reqs" {
  for_each = fileset("mwaa/", "*")
  bucket   =
  key      = each.value
  source   = "mwaa/${each.value}"
  etag     = filemd5("mwaa/${each.value}")
Enter fullscreen mode Exit fullscreen mode

In this section, we define the name of the MWAA environment we want to use, the version of Apache Airflow (1.12, 2.0.2, or 2.2.2), and the size of the MWAA Worker nodes (mw1.small, mw1.medium, or mw1.large). We then define the name of the dags folder which Apache Airflow will use as the "dags folder" to search for DAGs to run. Finally, you can optionally set a and file and location, but these are not set by default.

  name                 = "basic-mwaa"
  airflow_version      = "2.2.2"
  environment_class    = "mw1.medium"
  dag_s3_path          = "dags"
  #plugins_s3_path      = ""
  #requirements_s3_path = "requirements.txt"
Enter fullscreen mode Exit fullscreen mode

If you want to set the plugins_s3_path or requirements_s3_path you will need to set these here and then configure/deploy the and requirements.txt separately.

The next section configures the logging for the various MWAA services. This section allows us to define the logging verbosity for the different MWAA services. Bear in mind that there is a cost associated with this, so understand this before configuring these. The values you can use are CRITICAL, ERROR, WARNING, INFO, or DEBUG.

  logging_configuration = {
    dag_processing_logs = {
      enabled   = true
      log_level = "INFO"

    scheduler_logs = {
      enabled   = true
      log_level = "WARNING"

    task_logs = {
      enabled   = true
      log_level = "DEBUG"

    webserver_logs = {
      enabled   = true
      log_level = "INFO"

    worker_logs = {
      enabled   = true
      log_level = "INFO"
Enter fullscreen mode Exit fullscreen mode

In the next section you can define some custom Apache Airflow configuration parameters if you need to do so. You can check out the MWAA documentation to find out more, but you might use these to tweak performance settings or enable AWS integration of things like AWS Secrets Manager.

  airflow_configuration_options = {
    "core.load_default_connections" = "false"
    "core.load_examples" = "false"
    "webserver.dag_default_view" = "tree"
    "webserver.dag_orientation" = "TB"
Enter fullscreen mode Exit fullscreen mode

Provides scaling settings for the Apache Airflow worker nodes, and then provides details of the VPC networking. You should probably not need to change these (network) settings.

  min_workers        = 1
  max_workers        = 25
  vpc_id             = module.vpc.vpc_id
  private_subnet_ids = module.vpc.private_subnets

  webserver_access_mode = "PUBLIC_ONLY"
  source_cidr           = [""] 
Enter fullscreen mode Exit fullscreen mode

Finally, these options allows you to define and bring your own AWS security groups, execution roles or S3 buckets to use within MWAA. If you do create your own, make sure that these meet the minimum requirements by checking the MWAA documentation. Also, you will need to comment out the sections above so that Terraform will configure the correct resources.

  # create_security_group = 
  # source_bucket_arn = 
  # execution_role_arn = 
Enter fullscreen mode Exit fullscreen mode

Now that we have gone over these configuration, we are ready to try and deploy this. From my Visual Code IDE, I run the following command to initiate.

terraform init
Enter fullscreen mode Exit fullscreen mode

which generates the following output

 Initializing modules...
- mwaa in ../..
Downloading 3.14.2 for vpc...
- vpc in .terraform/modules/vpc

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching ">= 3.63.0, ~> 4.20.0"...
- Installing hashicorp/aws v4.20.1...
- Installed hashicorp/aws v4.20.1 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Enter fullscreen mode Exit fullscreen mode
terraform plan
Enter fullscreen mode Exit fullscreen mode

Displays what AWS resources Terraform will create. I will not display all the output as it will be different to yours but you can review it and you will see all the different resources that are about to be configured and deployed.

You are now ready to deploy your MWAA environment.

Deploying the MWAA environment

To deploy, you can now run "terraform apply", and having checked the output, answering "yes" when prompted if it looks all good.

terraform apply
Enter fullscreen mode Exit fullscreen mode

The deployment will then begin. This is what my output looked like:

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.vpc.aws_vpc.this[0]: Creating...
module.vpc.aws_eip.nat[0]: Creating...
module.mwaa.aws_iam_role.mwaa[0]: Creating...
module.mwaa.aws_s3_bucket.mwaa[0]: Creating...
module.vpc.aws_eip.nat[0]: Creation complete after 1s [id=eipalloc-0b7a488bea6bfddfc]
module.mwaa.aws_iam_role.mwaa[0]: Creation complete after 2s [id=mwaa-executor20220628091608972100000001]
module.mwaa.aws_s3_bucket.mwaa[0]: Creation complete after 3s [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_bucket_versioning.mwaa[0]: Creating...
module.mwaa.aws_s3_bucket_acl.mwaa[0]: Creating...
module.mwaa.aws_s3_bucket_server_side_encryption_configuration.mwaa[0]: Creating...
module.mwaa.aws_s3_bucket_public_access_block.mwaa[0]: Creating...
module.mwaa.aws_s3_object.python_requirements[0]: Creating...
module.mwaa.aws_s3_object.plugins[0]: Creating... Reading... Read complete after 0s [id=3112667403]
module.mwaa.aws_iam_role_policy.mwaa[0]: Creating...
module.mwaa.aws_iam_role_policy.mwaa[0]: Creation complete after 0s [id=mwaa-executor20220628091608972100000001:mwaa-executor20220628091611718600000003]
module.mwaa.aws_s3_bucket_acl.mwaa[0]: Creation complete after 1s [id=mwaa-704533066374-20220628091608973700000002,private]
module.mwaa.aws_s3_bucket_public_access_block.mwaa[0]: Creation complete after 1s [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_bucket_server_side_encryption_configuration.mwaa[0]: Creation complete after 1s [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_object.plugins[0]: Creation complete after 1s []
module.mwaa.aws_s3_object.python_requirements[0]: Creation complete after 1s [id=requirements.txt]
module.mwaa.aws_s3_bucket_versioning.mwaa[0]: Creation complete after 2s [id=mwaa-704533066374-20220628091608973700000002]
module.vpc.aws_vpc.this[0]: Still creating... [10s elapsed]
module.vpc.aws_vpc.this[0]: Creation complete after 12s [id=vpc-08c5b33e125c3e2df]
module.vpc.aws_subnet.public[1]: Creating...
module.vpc.aws_subnet.private[0]: Creating...
module.vpc.aws_subnet.private[1]: Creating...
module.vpc.aws_subnet.public[0]: Creating...
module.vpc.aws_route_table.public[0]: Creating...
module.mwaa.aws_security_group.mwaa[0]: Creating...
module.vpc.aws_internet_gateway.this[0]: Creating...
module.vpc.aws_route_table.private[0]: Creating...
module.vpc.aws_route_table.public[0]: Creation complete after 1s [id=rtb-026cb73aa7c373b6d]
module.vpc.aws_subnet.private[1]: Creation complete after 1s [id=subnet-02c22ce11ef5d1f24]
module.vpc.aws_route_table.private[0]: Creation complete after 1s [id=rtb-06a02df385b093898]
module.vpc.aws_internet_gateway.this[0]: Creation complete after 1s [id=igw-058e9b992d5def525]
module.vpc.aws_route.public_internet_gateway[0]: Creating...
module.vpc.aws_subnet.private[0]: Creation complete after 1s [id=subnet-014e17effc050d3f8]
module.vpc.aws_route_table_association.private[0]: Creating...
module.vpc.aws_route_table_association.private[1]: Creating...
module.vpc.aws_route_table_association.private[0]: Creation complete after 1s [id=rtbassoc-063cb70e81c051767]
module.vpc.aws_route_table_association.private[1]: Creation complete after 1s [id=rtbassoc-00ced4c22a1f470c0]
module.vpc.aws_route.public_internet_gateway[0]: Creation complete after 1s [id=r-rtb-026cb73aa7c373b6d1080289494]
module.mwaa.aws_security_group.mwaa[0]: Creation complete after 2s [id=sg-0fc61902a4cbbae48]
module.mwaa.aws_security_group_rule.mwaa_sg_outbound[0]: Creating...
module.mwaa.aws_security_group_rule.mwaa_sg_inbound[0]: Creating...
module.mwaa.aws_security_group_rule.mwaa_sg_inbound_vpn[0]: Creating...
module.mwaa.aws_mwaa_environment.mwaa: Creating...
module.mwaa.aws_security_group_rule.mwaa_sg_inbound[0]: Creation complete after 1s [id=sgrule-1253679849]
module.mwaa.aws_security_group_rule.mwaa_sg_outbound[0]: Creation complete after 1s [id=sgrule-2481519482]
module.mwaa.aws_security_group_rule.mwaa_sg_inbound_vpn[0]: Creation complete after 2s [id=sgrule-2489225287]
module.vpc.aws_subnet.public[0]: Still creating... [10s elapsed]
module.vpc.aws_subnet.public[1]: Still creating... [10s elapsed]
module.vpc.aws_subnet.public[0]: Creation complete after 11s [id=subnet-02c0205901abdb075]
module.vpc.aws_subnet.public[1]: Creation complete after 11s [id=subnet-06ad25fc78a2cf59f]
module.vpc.aws_route_table_association.public[1]: Creating...
module.vpc.aws_route_table_association.public[0]: Creating...
module.vpc.aws_nat_gateway.this[0]: Creating...
module.vpc.aws_route_table_association.public[1]: Creation complete after 1s [id=rtbassoc-0440eeba24fee7e0b]
module.vpc.aws_route_table_association.public[0]: Creation complete after 1s [id=rtbassoc-02c12d5499fc36282]
module.mwaa.aws_mwaa_environment.mwaa: Still creating... [10s elapsed]
module.vpc.aws_nat_gateway.this[0]: Still creating... [10s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still creating... [20s elapsed]
Enter fullscreen mode Exit fullscreen mode

After about 20-25 minutes, you should get the following output to show that the deployment has been completed

Apply complete! Resources: 30 added, 0 changed, 0 destroyed.


mwaa_arn = "arn:aws:airflow:eu-central-1:704533066374:environment/basic-mwaa"
mwaa_role_arn = "arn:aws:iam::704533066374:role/mwaa-executor20220628091608972100000001"
mwaa_security_group_id = "sg-0fc61902a4cbbae48"
mwaa_service_role_arn = "arn:aws:iam::704533066374:role/aws-service-role/"
mwaa_status = "AVAILABLE"
mwaa_webserver_url = ""
Enter fullscreen mode Exit fullscreen mode

We can now copy the "mwaa_webserver_url" into a browser, and then login using our AWS credentials to access our new MWAA environment.

Apache Airflow UI in MWAA

As you can see from the screenshot, our sample DAG has also been uploaded into the environment and we can use this to test that our environment is working as expected.

Deleting our MWAA environment

So we have covered how to configure and deploy MWAA environments, now I will cover how you can clean up and remove them. The clean up process takes around 20 minutes to complete. There are some things to be aware of when deleting your MWAA resources.

First, the destroy process will clean up all resources, including the S3 bucket that contains your DAGs. If you need to keep these safe, make sure that you copy/move them to another location before cleaning up your MWAA environment.

Second, the CloudWatch log groups that are created when the MWAA environment is configured will also not be deleted. If you want to completely clean up your environment, remember to go to CloudWatch and search under log groups and delete as needed.

With that out of the way, to remove this new environment we created and clean up all the resources, we issue the "terraform destroy" command, and when prompted respond appropriately.

terraform destroy

Enter fullscreen mode Exit fullscreen mode

this will generate a lot of output, displaying what resources will be removed. Here is a small portion of what my output looked like Reading... Reading... Reading...
data.aws_availability_zones.available: Reading... Reading... Read complete after 0s [id=aws]
module.vpc.aws_vpc.this[0]: Refreshing state... [id=vpc-08c5b33e125c3e2df] Read complete after 0s [id=eu-central-1] Read complete after 0s [id=2236429369]
module.mwaa.aws_iam_role.mwaa[0]: Refreshing state... [id=mwaa-executor20220628091608972100000001]
data.aws_availability_zones.available: Read complete after 0s [id=eu-central-1]
module.vpc.aws_eip.nat[0]: Refreshing state... [id=eipalloc-0b7a488bea6bfddfc] Read complete after 1s [id=704533066374]
module.mwaa.aws_s3_bucket.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_security_group.mwaa[0]: Refreshing state... [id=sg-0fc61902a4cbbae48]
module.vpc.aws_route_table.private[0]: Refreshing state... [id=rtb-06a02df385b093898]
module.vpc.aws_route_table.public[0]: Refreshing state... [id=rtb-026cb73aa7c373b6d]
module.vpc.aws_subnet.private[0]: Refreshing state... [id=subnet-014e17effc050d3f8]
module.vpc.aws_internet_gateway.this[0]: Refreshing state... [id=igw-058e9b992d5def525]
module.vpc.aws_subnet.private[1]: Refreshing state... [id=subnet-02c22ce11ef5d1f24]
module.vpc.aws_subnet.public[0]: Refreshing state... [id=subnet-02c0205901abdb075]
module.vpc.aws_subnet.public[1]: Refreshing state... [id=subnet-06ad25fc78a2cf59f]
module.mwaa.aws_security_group_rule.mwaa_sg_outbound[0]: Refreshing state... [id=sgrule-2481519482]
module.mwaa.aws_security_group_rule.mwaa_sg_inbound[0]: Refreshing state... [id=sgrule-1253679849]
module.mwaa.aws_security_group_rule.mwaa_sg_inbound_vpn[0]: Refreshing state... [id=sgrule-2489225287]
module.vpc.aws_route.public_internet_gateway[0]: Refreshing state... [id=r-rtb-026cb73aa7c373b6d1080289494]
module.vpc.aws_route_table_association.private[0]: Refreshing state... [id=rtbassoc-063cb70e81c051767]
module.vpc.aws_route_table_association.private[1]: Refreshing state... [id=rtbassoc-00ced4c22a1f470c0]
module.vpc.aws_nat_gateway.this[0]: Refreshing state... [id=nat-0d73ce39bbe46fdd9]
module.vpc.aws_route_table_association.public[0]: Refreshing state... [id=rtbassoc-02c12d5499fc36282]
module.vpc.aws_route_table_association.public[1]: Refreshing state... [id=rtbassoc-0440eeba24fee7e0b]
module.vpc.aws_route.private_nat_gateway[0]: Refreshing state... [id=r-rtb-06a02df385b0938981080289494]
module.mwaa.aws_s3_bucket_public_access_block.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_bucket_versioning.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_object.plugins[0]: Refreshing state... []
module.mwaa.aws_s3_object.python_requirements[0]: Refreshing state... [id=requirements.txt]
module.mwaa.aws_s3_bucket_server_side_encryption_configuration.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002]
module.mwaa.aws_s3_bucket_acl.mwaa[0]: Refreshing state... [id=mwaa-704533066374-20220628091608973700000002,private]
module.mwaa.aws_mwaa_environment.mwaa: Refreshing state... [id=basic-mwaa] Reading... Read complete after 0s [id=3112667403]
module.mwaa.aws_iam_role_policy.mwaa[0]: Refreshing state... [id=mwaa-executor20220628091608972100000001:mwaa-executor20220628091611718600000003]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # module.mwaa.aws_iam_role.mwaa[0] will be destroyed
  - resource "aws_iam_role" "mwaa" {
      - arn                   = "arn:aws:iam::704533066374:role/mwaa-executor20220628091608972100000001" -> null
      - assume_role_policy    = jsonencode(

Enter fullscreen mode Exit fullscreen mode

You will be prompted to enter "yes" to confirm you want to delete the MWAA environment. It will then start cleaning up the resources, and then display the following:

module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 2m40s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 2m50s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 3m0s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 3m10s elapsed]
module.mwaa.aws_mwaa_environment.mwaa: Still destroying... [id=terraform-mwaa, 3m20s elapsed]
Enter fullscreen mode Exit fullscreen mode

as it cleans up the MWAA environment. This will take approx. 20 minutes.


I plan to build upon this blog post and revisit and provide Terraform build files for some of the other MWAA related articles that I have done that show MWAA with other AWS resources such as Amazon EMR, Amazon Athena, Amazon RedShift and more.

All the resources you need are available at the Terraform module page, and make sure you check out the examples which is what I have used in this post.

The MWAA Terraform module is also available in the Terraform registry.

The AWS that have put this new Terraform module together would love your feedback. Does this work as you expect it? What examples would you like included? Did you try this and find any errors or quirks? Please let us know, either directly by raising an issue, or via comments below.

Many thanks.

Top comments (1)