DEV Community

Mahmudul Haque
Mahmudul Haque

Posted on

Apache Airflow on EC2 with RDS 

sudo apt update && sudo apt upgrade -y
Enter fullscreen mode Exit fullscreen mode 
sudo apt install python3-pip python3-psycopg2 default-libmysqlclient-dev
Enter fullscreen mode Exit fullscreen mode 
python3 -m pip install --upgrade pip
Enter fullscreen mode Exit fullscreen mode 
pip install --user apache-airflow[amazon] apache-airflow[postgres] apache-airflow[celery] psycopg2-binary apache-airflow-providers-mysql Redis
Enter fullscreen mode Exit fullscreen mode 
echo "export AIRFLOW_HOME=${HOME}/airflow" >> .zshrc
Enter fullscreen mode Exit fullscreen mode 
airflow db init
Enter fullscreen mode Exit fullscreen mode

airflow.cfg

[core]
executor = CeleryExecutor
load_examples = False

[database]
sql_alchemy_conn = postgresql://myuser:mypassword@my-rds-instance.abc123.us-west-2.rds.amazonaws.com:5432/mydatabase`

[celery]
broker_url = redis://localhost:6379/0
result_backend = redis://localhost:6379/1
Enter fullscreen mode Exit fullscreen mode

Celery Executor: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/celery.html

airflow db init
Enter fullscreen mode Exit fullscreen mode 
airflow users create --username admin --firstname FirstName --lastname LastName --role Admin --email email@example.com
Enter fullscreen mode Exit fullscreen mode

/etc/systemd/system

sudo systemctl enable airflow-webserver.service

sudo systemctl enable airflow-scheduler.service

sudo systemctl enable airflow-celery-flower.service

sudo systemctl enable airflow-celery-worker.service

sudo systemctl start airflow-webserver

sudo systemctl start airflow-scheduler

sudo systemctl start airflow-celery-flower

sudo systemctl start airflow-celery-worker
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Read next

olalekan_johnayansola_4a profile image

A technical blog on Git and Github, how to set up git, creating a repository, making commits, pushing,pulling

OLALEKAN john Ayansola -

xxzeroxx profile image

PHP Design Patterns: Front Controller

Antonio Silva -

manashjb profile image

Stop Trying to Learn Everything -Focus on These 5 Key Skills Every Developer Needs

Manash Jyoti Baruah -

beansharon1 profile image

A Handbook to Grasping Contemporary Mobile Device Technology

Kara Zhao -