DEV Community

Cover image for Installing Apache Airflow on Ubuntu 24.04
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya Originally published at docs.vultr.com

Installing Apache Airflow on Ubuntu 24.04

#airflow #python #postgres #devops

Apache Airflow is an open-source workflow management platform that manages data pipelines and automates workflows such as Extract, Transform, and Load (ETL) processes. It uses Python-based Directed Acyclic Graphs (DAGs) to schedule and execute tasks while streamlining the management of all required dependencies for error-free execution. This guide installs Apache Airflow on Ubuntu 24.04, configures a secure environment behind Nginx with Let's Encrypt, and tests the deployment with a sample DAG. By the end, you'll have a working Apache Airflow instance served securely over HTTPS.

Prerequisites

  • An Ubuntu 24.04 server with at least 4 GB RAM.
  • A domain A record with your DNS provider pointing to the server's IP address.

Install Apache Airflow

Apache Airflow is available as a Python package installed with a package manager such as Pip.

1. Update the server's package index:

$ sudo apt update
Enter fullscreen mode Exit fullscreen mode

2. View the available Python version:

$ python3 --version
Enter fullscreen mode Exit fullscreen mode

Your output should be similar to:

Python 3.12.3
Enter fullscreen mode Exit fullscreen mode

Install Python if it's not available on your server:

$ sudo apt install python3
Enter fullscreen mode Exit fullscreen mode

3. Install the python3-venv virtual environment module and the PostgreSQL development library:

$ sudo apt install python3-venv libpq-dev -y
Enter fullscreen mode Exit fullscreen mode

4. Create a new virtual environment, for example airflow_env:

$ python3 -m venv airflow_env
Enter fullscreen mode Exit fullscreen mode

5. Activate the airflow_env virtual environment:

$ source ~/airflow_env/bin/activate
Enter fullscreen mode Exit fullscreen mode

Verify that your shell prompt changes to the airflow_env virtual environment:

(airflow_env) linuxuser@example:~$
Enter fullscreen mode Exit fullscreen mode

6. Use Pip to install Apache Airflow with PostgreSQL support:

$ pip install apache-airflow[postgres] psycopg2
Enter fullscreen mode Exit fullscreen mode

7. Install PostgreSQL:

$ sudo apt install postgresql postgresql-contrib
Enter fullscreen mode Exit fullscreen mode

8. Start the PostgreSQL service:

$ sudo systemctl start postgresql
Enter fullscreen mode Exit fullscreen mode

9. Access the PostgreSQL console using the postgres user:

$ sudo -u postgres psql
Enter fullscreen mode Exit fullscreen mode

Your output should be similar to:

psql (16.6 (Ubuntu 16.6-0ubuntu0.24.04.1))
Type "help" for help.

postgres=#
Enter fullscreen mode Exit fullscreen mode

10. Create a new airflow PostgreSQL user with a strong password. Replace YourStrongPassword with your desired password:

postgres=# CREATE USER airflow PASSWORD 'YourStrongPassword';
Enter fullscreen mode Exit fullscreen mode

11. Create a new database, for example airflowdb:

postgres=# CREATE DATABASE airflowdb;
Enter fullscreen mode Exit fullscreen mode

12. Grant the airflow user full privileges to all tables in the public schema:

postgres=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO airflow;
Enter fullscreen mode Exit fullscreen mode

13. Grant the airflow user ownership privileges to the airflowdb database:

postgres=# ALTER DATABASE airflowdb OWNER TO airflow;
Enter fullscreen mode Exit fullscreen mode

14. Grant the airflow user all privileges on the public schema:

postgres=# GRANT ALL ON SCHEMA public TO airflow;
Enter fullscreen mode Exit fullscreen mode

15. Exit the PostgreSQL console:

postgres=# exit;
Enter fullscreen mode Exit fullscreen mode

16. Open the airflow.cfg file in your Airflow installation directory:

$ nano ~/airflow/airflow.cfg
Enter fullscreen mode Exit fullscreen mode

Temporarily initialize the database and start the Airflow scheduler to create the necessary directories if the airflow directory is missing:

$ airflow db init; airflow scheduler
Enter fullscreen mode Exit fullscreen mode

Press Ctrl+C to stop the scheduler.

17. Replace the default executor and sql_alchemy_conn values with the following configuration to enable parallel execution and set PostgreSQL as the metadata database:

executor = LocalExecutor
sql_alchemy_conn = postgresql+psycopg2://airflow:YourStrongPassword@localhost/airflowdb
Enter fullscreen mode Exit fullscreen mode

Save and close the file.

18. Initialize the Airflow metadata database to apply the changes:

$ airflow db init
Enter fullscreen mode Exit fullscreen mode

Your output should be similar to:

DB: postgresql+psycopg2://airflow:***@localhost/airflow
[2025-01-05T23:58:36.808+0000] {migration.py:207} INFO - Context impl PostgresqlImpl.
[2025-01-05T23:58:36.809+0000] {migration.py:210} INFO - Will assume transactional DDL.
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.
INFO  [alembic.runtime.migration] Running stamp_revision  -> 5f2621c13b39
WARNI [airflow.models.crypto] empty cryptography key - values will not be stored encrypted.
Initialization done
Enter fullscreen mode Exit fullscreen mode

19. Create a new administrative user to use with Apache Airflow. Replace admin with your desired username:

$ airflow users create \
   --username admin \
   --password yourSuperSecretPassword \
   --firstname Admin \
   --lastname User \
   --role Admin \
   --email admin@example.com
Enter fullscreen mode Exit fullscreen mode

20. Start the Airflow web server on port 8080 as a background process and redirect logs to webserver.log:

$ nohup airflow webserver -p 8080 > webserver.log 2>&1 &
Enter fullscreen mode Exit fullscreen mode

21. Start the Airflow scheduler and redirect logs to scheduler.log:

$ nohup airflow scheduler > scheduler.log 2>&1 &
Enter fullscreen mode Exit fullscreen mode

Configure Nginx as a Reverse Proxy to Expose Apache Airflow

Apache Airflow listens for connections on the default port 8080. Use Nginx to front that port and serve requests over HTTP and HTTPS.

1. Install Nginx:

$ sudo apt install -y nginx
Enter fullscreen mode Exit fullscreen mode

2. Create a new airflow Nginx virtual host configuration file:

$ sudo nano /etc/nginx/sites-available/airflow
Enter fullscreen mode Exit fullscreen mode

3. Add the following configuration to the file. Replace airflow.example.com with your actual domain:

server {
    listen 80;
    server_name airflow.example.com;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
Enter fullscreen mode Exit fullscreen mode

Save and close the file. This configuration listens for connections on your airflow.example.com domain and forwards them to the Apache Airflow port 8080.

4. Link the configuration to the Nginx sites-enabled directory to enable it:

$ sudo ln -s /etc/nginx/sites-available/airflow /etc/nginx/sites-enabled/
Enter fullscreen mode Exit fullscreen mode

5. Test the Nginx configuration for errors:

$ sudo nginx -t
Enter fullscreen mode Exit fullscreen mode

Your output should be similar to:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
Enter fullscreen mode Exit fullscreen mode

6. Reload Nginx to apply the changes:

$ sudo systemctl reload nginx
Enter fullscreen mode Exit fullscreen mode

7. Allow connections to the HTTP port 80 through the firewall:

$ sudo ufw allow 80/tcp
Enter fullscreen mode Exit fullscreen mode

8. Reload UFW to apply the firewall configuration changes:

$ sudo ufw reload
Enter fullscreen mode Exit fullscreen mode

9. Access your airflow.example.com domain in a web browser and verify that the Airflow login page displays.

http://airflow.example.com
Enter fullscreen mode Exit fullscreen mode

Generate Trusted SSL Certificates to Secure Apache Airflow

SSL certificates encrypt the connection between a client and the Apache Airflow server. Use Certbot to generate Let's Encrypt SSL certificates.

1. Install the Certbot Let's Encrypt client:

$ sudo snap install --classic certbot
Enter fullscreen mode Exit fullscreen mode

Install Snap first if it's not available on your workstation:

$ sudo apt install snapd -y
Enter fullscreen mode Exit fullscreen mode

2. Move the Certbot binary to /usr/bin to enable it as a system-wide command:

$ sudo ln -s /snap/bin/certbot /usr/bin/certbot
Enter fullscreen mode Exit fullscreen mode

3. Request a new Let's Encrypt SSL certificate using the Nginx plugin and your domain. Replace airflow.example.com with your actual domain and admin@example.com with your active email address:

$ sudo certbot --nginx --redirect -d airflow.example.com -m admin@example.com --agree-tos
Enter fullscreen mode Exit fullscreen mode

Your output should be similar to the following when the certificate request succeeds:

...
Account registered.
Requesting a certificate for airflow.example.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/airflow.example.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/airflow.example.com/privkey.pem
This certificate expires on 2025-04-21.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

Deploying certificate
Successfully deployed certificate for airflow.example.com to /etc/nginx/sites-enabled/airflow
Congratulations! You have successfully enabled HTTPS on https://airflow.example.com
...
Enter fullscreen mode Exit fullscreen mode

4. Verify that Certbot auto-renews the SSL certificate before it expires:

$ sudo certbot renew --dry-run
Enter fullscreen mode Exit fullscreen mode

5. Restart Nginx to apply the SSL configuration changes:

$ sudo systemctl restart nginx
Enter fullscreen mode Exit fullscreen mode

Access Apache Airflow

1. Access the Apache Airflow web interface using your domain:

https://airflow.example.com
Enter fullscreen mode Exit fullscreen mode

Enter the credentials you set earlier to log in:

  • Username: admin
  • Password: yourSuperSecretPassword

Create and Run DAGs Using Apache Airflow

1. Create the dags directory in the Airflow installation directory:

$ mkdir ~/airflow/dags
Enter fullscreen mode Exit fullscreen mode

2. Create a new my_first_dag.py Python application file in the dags directory:

$ nano ~/airflow/dags/my_first_dag.py
Enter fullscreen mode Exit fullscreen mode

3. Add the following code to define a new DAG:

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta

with DAG(
    'my_first_dag',
    start_date=datetime(2024, 1, 1),
    schedule_interval=timedelta(days=1),
    catchup=False
) as dag:

    def print_hello():
        print('Hello from Airflow!')

    hello_task = PythonOperator(
        task_id='hello_task',
        python_callable=print_hello
    )
Enter fullscreen mode Exit fullscreen mode

Save and close the file. This creates a my_first_dag sample DAG that runs daily and prints Hello from Airflow!.

4. In the Apache Airflow interface, navigate to the DAGs list, find the DAG, and enable it to trigger it manually.

5. Use the Graph View and Event Log to monitor the DAG.

Next Steps

Apache Airflow is running behind Nginx with HTTPS and executing your first DAG. From here you can:

  • Add more DAGs to automate real ETL and data pipeline workflows
  • Configure Airflow connections and variables for external systems (databases, APIs, object storage)
  • Set up monitoring and alerting for task failures and SLA misses
  • Review the Certbot renewal timer and Airflow's authentication/RBAC options before going to production

For the full guide with additional tips, visit the original article on Vultr Docs.

Top comments (0)