DEV Community

CitizenK
CitizenK

Posted on

1

Apache Airflow Installation - mysql+celery

pre requisites: python 3.7 or greater

Execute the below ones one after another

pip install apache-airflow==1.10.6
pip install 'apache-airflow[celery]'
pip install 'apache-airflow[mysql]'
brew install rabbitmq

start the rabbitmq-server in the background
rabbitmq-server -detached

make the mysql config changes
sudo chown -R _mysql:mysql /usr/local/var/mysql

start the mysql server
sudo mysql.server start

mysql setup

mysql -uroot

set the password for the root user with the below

mysql>ALTER USER 'root'@'localhost' IDENTIFIED BY 'new password';

create the DB, the user and grant the required privileges

mysql>CREATE DATABASE airflow CHARACTER SET utf8 COLLATE utf8_unicode_ci;
mysql>create user ‘airflow’@’localhost’ identified by ‘airflow’;
mysql>grant all privileges on * . * to 'airflow'@'localhost';
mysql>flush privileges;
mysql>quit

airflow initialization

airflow initdb

airflow config file update

Update the airflow.cfg file (should be available in ~/airflow/ directory.

sql connection as:
sql_alchemy_conn = mysql://root:airflow@localhost/airflow
there should be a sql alchemy connection string and you can comment it and add the above

executor as:
CeleryExecutor

Save the file and exit.

run the below on their separate windows

airflow webserver
airflow scheduler
airflow worker

airflow webserver window will show you the url for airflow UI.

note: airflow initialization (airflow initdb) should load the example dags. You can turn them off in your airflow.cfg file, if you don't want to. command: load_examples = False

Heroku

Build apps, not infrastructure.

Dealing with servers, hardware, and infrastructure can take up your valuable time. Discover the benefits of Heroku, the PaaS of choice for developers since 2007.

Visit Site

Top comments (0)

Billboard image

Create up to 10 Postgres Databases on Neon's free plan.

If you're starting a new project, Neon has got your databases covered. No credit cards. No trials. No getting in your way.

Try Neon for Free →

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay