DEV Community

Ricardo Sueiras for AWS

Posted on • Originally published at

Getting mwaa-local-runner up on AWS Cloud9

Here is a quick recipe if you are looking to get mwaa-local-runner up and running on your Cloud9 developer setup. This might not be the most optimised way, so I am very happy to received suggestions on how to improve this. What I will cover here is how to deploy mwaa-local-runner onto a standard Cloud9 IDE, deployed in a default VPC.

Updating my AWS Cloud9 environment

The first thing I needed to do was to increase the size of my local disk as Cloud9 only provides 10gb of storage. This is fine for typical use cases, but we are going to be building container images, so need to set this higher.

pip3 install --user --upgrade boto3
export instance_id=$(curl -s
python -c "import boto3
import os
from botocore.exceptions import ClientError 
ec2 = boto3.client('ec2')
volume_info = ec2.describe_volumes(
            'Name': 'attachment.instance-id',
            'Values': [
volume_id = volume_info['Volumes'][0]['VolumeId']
    resize = ec2.modify_volume(    
except ClientError as e:
    if e.response['Error']['Code'] == 'InvalidParameterValue':
        print('ERROR MESSAGE: {}'.format(e))"
if [ $? -eq 0 ]; then
    sudo reboot
Enter fullscreen mode Exit fullscreen mode

Installing some missing tools

The next thing I need to do is install "docker-compose" as whilst Docker is installed, docker-compose is not. This is the script I use

wget$(uname -s)-$(uname -m)
sudo mv docker-compose-$(uname -s)-$(uname -m) /usr/bin/docker-compose
chmod +x /usr/bin/docker-compose
Enter fullscreen mode Exit fullscreen mode

And if this is successful, you should now have it available via the command line

ops:~/environment $ docker-compose version
Docker Compose version v2.17.2
Enter fullscreen mode Exit fullscreen mode

Grabbing my AWS Cloud9 public IP address

This should be everything you need to get started. However, we do need one piece of information before we proceed, and that is the public IP address for your Cloud9 instance. We need this as we are going to update the URL configuration parameter in mwaa-local-runner. (If you do not do this, you will get 400 Gateway errors when accessing Airflow via the UI). Run the following command to find yours

Enter fullscreen mode Exit fullscreen mode

Configuring and building mwaa-local-runner

You can now check out the mwaa-local-runner project into your Cloud9 environment

git clone
Enter fullscreen mode Exit fullscreen mode

We now need to change a couple of things.

The first is to update the airflow.cfg (docker/config/airflow.cfg) so that we can update the base_url parameter (line 406) with the public IP of your Cloud9 instance.

#base_url = http://localhost:8080
base_url =
Enter fullscreen mode Exit fullscreen mode

The next change, still in the airflow.cfg, to update how quickly it picks up DAGs in the DAGs folder. Line 757 is changed

#dag_dir_list_interval = 300
dag_dir_list_interval = 5

Enter fullscreen mode Exit fullscreen mode

You can now save this file, and you are ready to build your mwaa-local-runner image. You do this using the following command (and this works against any version of Apache Airflow that mwaa supports)

mwaa-local-env build-image
Enter fullscreen mode Exit fullscreen mode

This will now take around 10 minutes to complete the build process. Go grab your favourite drink and then come back. Hopefully, you should see something like:

[+] Building 543.8s (27/27) FINISHED                                                                                                                                                                           
 => [internal] load build definition from Dockerfile                                                                                                                                                      0.0s
 => => transferring dockerfile: 43B                                                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                                                         0.0s
 => => transferring context: 2B                                                                                                                                                                           0.0s
 => [internal] load metadata for                                                                                                                                          1.6s
 => [auth] library/amazonlinux:pull token for                                                                                                                                        0.0s
 => CACHED [ 1/21] FROM                                                                           0.0s
 => [internal] load build context                                                                                                                                                                         0.0s
 => => transferring context: 5.46kB                                                                                                                                                                       0.0s
 => [ 2/21] COPY script/ /                                                                                                                                                        0.1s
 => [ 3/21] COPY script/ /                                                                                                                                                      0.0s
 => [ 4/21] COPY script/ /                                                                                                                                                  0.0s
 => [ 5/21] COPY script/ /                                                                                                                                                    0.0s
 => [ 6/21] COPY script/ /                                                                                                                                    0.0s
 => [ 7/21] COPY script/ /                                                                                                                                                  0.0s
 => [ 8/21] COPY config/constraints.txt /constraints.txt                                                                                                                                                  0.0s
 => [ 9/21] COPY config/mwaa-base-providers-requirements.txt /mwaa-base-providers-requirements.txt                                                                                                        0.0s
 => [10/21] RUN chmod u+x / && /                                                                                                                                              133.1s
 => [11/21] RUN chmod u+x / && /                                                                                                                                                343.3s 
 => [12/21] RUN chmod u+x / && /                                                                                                                                            0.6s 
 => [13/21] RUN chmod u+x /                                                                                                                                                                 0.3s 
 => [14/21] RUN chmod u+x /                                                                                                                                                         0.3s 
 => [15/21] RUN chmod u+x /                                                                                                                                                                0.5s 
 => [16/21] COPY script/ /                                                                                                                                                      0.0s 
 => [17/21] COPY config/airflow.cfg /usr/local/airflow/airflow.cfg                                                                                                                                        0.0s 
 => [18/21] COPY config/ /usr/local/airflow/                                                                                                                        0.0s
 => [19/21] RUN chown -R airflow: /usr/local/airflow                                                                                                                                                     31.4s
 => [20/21] RUN chmod +x /                                                                                                                                                                   0.4s
 => [21/21] WORKDIR /usr/local/airflow                                                                                                                                                                    0.1s
 => exporting to image                                                                                                                                                                                   31.7s
 => => exporting layers                                                                                                                                                                                  31.7s
 => => writing image sha256:54d466e5db131a71b694018f94ab69653794548551c37b05d172f5070f4deda8                                                                                                              0.0s
 => => naming to                                                                                                                                                          0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Enter fullscreen mode Exit fullscreen mode

Congratulations, the hard part has been done.

Note If you see the following error:

16 188.4 + sudo -u airflow pip3 install --no-use-pep517 --constraint /constraints.txt poetry
16 188.8 
16 188.8 Usage:   
16 188.8   pip3 install [options] <requirement specifier> [package-index-options] ...
16 188.8   pip3 install [options] -r <requirements file> [package-index-options] ...
16 188.8   pip3 install [options] [-e] <vcs project url> ...
16 188.8   pip3 install [options] [-e] <local project path> ...
16 188.8   pip3 install [options] <archive url/path> ...
16 188.8 
16 188.8 --no-use-pep517 error: It is not possible to use --no-use-pep517
16 188.8 without setuptools and wheel installed.

executor failed running [/bin/sh -c chmod u+x / && /]: exit code: 2

You will need to update the "" script, and update line 43 so that it looks like:

pip3 install $PIP_OPTION --upgrade 'pip<23'

This is to resolve an issue with pip version 23.1

To start mwaa-local-runner, all you need to do now is:

mwaa-local-env start
Enter fullscreen mode Exit fullscreen mode

And Apache Airflow will start to boot. It will take 2-3 minutes, but once ready you will see something like

aws-mwaa-local-runner-2_4-local-runner-1  | [2023-04-17 10:09:29 +0000] [202] [INFO] Starting gunicorn 20.1.0
aws-mwaa-local-runner-2_4-local-runner-1  | [2023-04-17 10:09:29 +0000] [202] [INFO] Listening at: (202)
Enter fullscreen mode Exit fullscreen mode

And you can now open a browser, pointing to your public Ip address and access Apache Airflow on port 8080.


This short post showed you how you can get mwaa-local-runner up and running on your AWS Cloud9 instances. I have been experimenting using this, and really like that I can decouple this from my local setup. I am still working on tweaking the setup so that I can locadown public access to the Cloud9 instance to just my own IP address, something I would recommend you do to.

If you have found this blog post helpful, please give me some feedback by completing this very short survey here

Top comments (0)