DEV Community

Mahmudul Haque
Mahmudul Haque

Posted on

Apache Airflow on EC2 with RDS 

sudo apt update && sudo apt upgrade -y
Enter fullscreen mode Exit fullscreen mode 
sudo apt install python3-pip python3-psycopg2 default-libmysqlclient-dev
Enter fullscreen mode Exit fullscreen mode 
python3 -m pip install --upgrade pip
Enter fullscreen mode Exit fullscreen mode 
pip install --user apache-airflow[amazon] apache-airflow[postgres] apache-airflow[celery] psycopg2-binary apache-airflow-providers-mysql Redis
Enter fullscreen mode Exit fullscreen mode 
echo "export AIRFLOW_HOME=${HOME}/airflow" >> .zshrc
Enter fullscreen mode Exit fullscreen mode 
airflow db init
Enter fullscreen mode Exit fullscreen mode

airflow.cfg

[core]
executor = CeleryExecutor
load_examples = False

[database]
sql_alchemy_conn = postgresql://myuser:mypassword@my-rds-instance.abc123.us-west-2.rds.amazonaws.com:5432/mydatabase`

[celery]
broker_url = redis://localhost:6379/0
result_backend = redis://localhost:6379/1
Enter fullscreen mode Exit fullscreen mode

Celery Executor: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery.html

airflow db init
Enter fullscreen mode Exit fullscreen mode 
airflow users create --username admin --firstname FirstName --lastname LastName --role Admin --email email@example.com
Enter fullscreen mode Exit fullscreen mode

/etc/systemd/system

sudo systemctl enable airflow-webserver.service

sudo systemctl enable airflow-scheduler.service

sudo systemctl enable airflow-celery-flower.service

sudo systemctl enable airflow-celery-worker.service

sudo systemctl start airflow-webserver

sudo systemctl start airflow-scheduler

sudo systemctl start airflow-celery-flower

sudo systemctl start airflow-celery-worker
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

orlidev profile image

** Construyendo una API REST: Una Aventura LEGO🚀 **

Orli Dun -

rishi_karanam profile image

Exploring API Architectures: A Beginner's Journey

Rishi karanam -

mugultum profile image

Mastering Data Exploration with Tidyverse: A Beginner-Friendly Guide

Silvester -

nagwan profile image

What is JavaScript?!!

nagwan sami -