DEV Community

Cover image for 5 Docker Compose Mistakes That Broke My Airflow 3 Setup 𖣘
De' Clerke
De' Clerke

Posted on

5 Docker Compose Mistakes That Broke My Airflow 3 Setup 𖣘

#apacheairflow #dataengineering #orchestration #airflow3

I've run Airflow 3 in Docker Compose across 8 data pipeline projects: a call center analytics pipeline, a loan risk model, an ecommerce pipeline, and more. Each time I set up a fresh environment, I hit at least one of these five mistakes. Some crashed loudly. Others failed silently for hours before I found the cause.

This isn't a getting-started guide. The official docs cover that. This is the list of things the docs assume you already know, plus the exact config that fixes each one.

Mistake 1: Running pip install as root in your Dockerfile

This is the most common setup mistake, and also the hardest to debug because it doesn't fail at build time. Your image builds cleanly. Containers start. Then your DAGs crash with ModuleNotFoundError even though the package is right there in requirements.txt.

Here's what happened:

# WRONG: installs to /root/.local/lib/ (root-only path)
FROM apache/airflow:3.0.0
RUN pip install --no-cache-dir pandas==2.2.3
Enter fullscreen mode Exit fullscreen mode

The Airflow image runs as the airflow user at runtime. When you install as root, packages land in /root/.local/lib/, which the airflow user can't reach. The build succeeds, the packages exist, but they're invisible to the process that actually runs your tasks.

# CORRECT: USER airflow must come before pip install
FROM apache/airflow:3.0.0

USER airflow
COPY requirements.txt /tmp/requirements.txt
RUN pip install --no-cache-dir -r /tmp/requirements.txt
Enter fullscreen mode Exit fullscreen mode

Same problem with uv pip install --system inside the Airflow image. It installs to a system path that the airflow user's Python doesn't see. Stick to plain pip install as USER airflow.

Mistake 2: Using airflow db init or airflow db upgrade

If you've used Airflow 2, these are pure muscle memory:

airflow db init     # Airflow 2
airflow db upgrade  # Airflow 2
Enter fullscreen mode Exit fullscreen mode

Both commands are gone in Airflow 3. The replacement is airflow db migrate, which handles both initial setup and upgrades in one shot:

# In your docker-compose.yml airflow-init service
airflow-init:
  <<: *airflow-common
  command: >
    bash -c "airflow db migrate &&
             airflow users create
               --username admin
               --firstname Admin
               --lastname User
               --role Admin
               --email your@email.com
               --password ${AIRFLOW_ADMIN_PASSWORD}"
  restart: "no"
Enter fullscreen mode Exit fullscreen mode

Run this as a one-shot init service with restart: "no". Run it again every time you bump the Airflow image tag.

Two other things that catch Airflow 2 migrants off guard: Airflow 3 requires SQLAlchemy 2.x (not 1.4) and Python 3.9+ (3.11 recommended). If you're copying a requirements.txt from an older project, update those pins before you build.

Mistake 3: Leaving out the triggerer service

In Airflow 2, the triggerer was optional. You only needed it for deferrable operators. In Airflow 3, it is required, even when you think you're not using any deferred tasks.

Leave it out and tasks that use sensors or any deferrable operator will enter a deferred state and sit there indefinitely. No error message. No log output. The task just never moves.

Add it to your docker-compose.yml:

airflow-triggerer:
  <<: *airflow-common
  command: triggerer
  restart: unless-stopped
Enter fullscreen mode Exit fullscreen mode

It uses the same x-airflow-common anchor as your other services. The triggerer is lightweight. It runs an async event loop, not heavy task execution. There's no reason to leave it out.

Mistake 4: Writing passwords.json as a list

Airflow 3 replaces Flask-AppBuilder (FAB) with SimpleAuthManager as the default authentication backend. Instead of a database-backed user table, credentials live in a JSON file you manage yourself.

The catch: the file must be a JSON object (dict), not an array. I wrote it as a list the first time because it felt natural for a user record:

// WRONG: Airflow crashes on startup with a cryptic auth error
[
  {"username": "admin", "password": "your_secure_password"}
]
Enter fullscreen mode Exit fullscreen mode 
// CORRECT: username as key, password as value
{
  "admin": "your_secure_password"
}
Enter fullscreen mode Exit fullscreen mode

When the format is wrong, the startup error doesn't say "wrong format." It surfaces as an auth manager initialization failure deep in the stack. You can easily spend 30 minutes looking in the wrong place.

Enable SimpleAuthManager and mount the file in your docker-compose.yml:

environment:
  AIRFLOW__CORE__AUTH_MANAGER: airflow.auth.managers.simple.simple_auth_manager.SimpleAuthManager
  AIRFLOW__SIMPLE_AUTH_MANAGER__PASSWORDS_FILE: /opt/airflow/config/passwords.json

volumes:
  - ./config/passwords.json:/opt/airflow/config/passwords.json:ro
Enter fullscreen mode Exit fullscreen mode

One more thing: the _AIRFLOW_WWW_USER_CREATE environment variable that created admin users in Airflow 2 is silently ignored in Airflow 3 with SimpleAuthManager. If you copied a Compose file from an Airflow 2 project, that env var does nothing. Your user won't be created and login will fail even when the credentials are correct.

Mistake 5: Using localhost for service connections inside Docker

When all your services run inside the same Docker Compose network, localhost doesn't work as a hostname. Each container is its own isolated network namespace. Inside the airflow-scheduler container, localhost resolves to that container itself, not to Postgres or any other service.

Use the service name from docker-compose.yml as the hostname instead:

# WRONG: localhost doesn't resolve to Postgres from inside the container
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow_user:${POSTGRES_PASSWORD}@localhost/airflow_db

# CORRECT: use the service name as the host
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow_user:${POSTGRES_PASSWORD}@postgres/airflow_db
Enter fullscreen mode Exit fullscreen mode

The same rule applies to every connection your tasks make: SQLAlchemy URLs, Airflow connections, any API endpoint pointing to another service in the stack. If the target is defined in docker-compose.yml, use its name as the host.

This also applies to Airflow 3's Task Execution API. Airflow 3 introduced AIP-72, which routes task-to-scheduler communication through an authenticated HTTP API instead of direct database access. If tasks are silently failing with auth errors across multiple services, check whether AIRFLOW__CORE__EXECUTION_API_SERVER_URL is still pointing at localhost:

# In your x-airflow-common environment block
AIRFLOW__CORE__EXECUTION_API_SERVER_URL: http://airflow-webserver:8080/execution/
Enter fullscreen mode Exit fullscreen mode

The Full docker-compose.yml

Here's the complete config that has worked reliably across all 8 of my projects:

x-airflow-common: &airflow-common
  build: .
  env_file: .env
  environment:
    AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow_user:${POSTGRES_PASSWORD}@postgres/airflow_db
    AIRFLOW__CORE__EXECUTOR: LocalExecutor
    AIRFLOW__CORE__LOAD_EXAMPLES: "false"
    AIRFLOW__WEBSERVER__SECRET_KEY: ${AIRFLOW_SECRET_KEY}
  volumes:
    - ./dags:/opt/airflow/dags
    - airflow_logs:/opt/airflow/logs
    - airflow_plugins:/opt/airflow/plugins
  depends_on:
    postgres:
      condition: service_healthy
  networks:
    - pipeline_net

services:

  postgres:
    image: postgres:15
    restart: unless-stopped
    environment:
      POSTGRES_USER: airflow_user
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      POSTGRES_DB: airflow_db
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U airflow_user -d airflow_db"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
    networks:
      - pipeline_net

  airflow-init:
    <<: *airflow-common
    command: >
      bash -c "airflow db migrate &&
               airflow users create --username admin --firstname Admin
                 --lastname User --role Admin --email your@email.com
                 --password ${AIRFLOW_ADMIN_PASSWORD}"
    restart: "no"

  airflow-webserver:
    <<: *airflow-common
    command: webserver
    ports:
      - "8080:8080"
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  airflow-scheduler:
    <<: *airflow-common
    command: scheduler
    restart: unless-stopped

  airflow-triggerer:
    <<: *airflow-common
    command: triggerer
    restart: unless-stopped

volumes:
  postgres_data:
  airflow_logs:
  airflow_plugins:

networks:
  pipeline_net:
    driver: bridge
Enter fullscreen mode Exit fullscreen mode

Your .env (never commit this):

POSTGRES_PASSWORD=yourpassword
AIRFLOW_ADMIN_PASSWORD=yourpassword
AIRFLOW_SECRET_KEY=your-long-random-secret
DATABASE_URL=postgresql+psycopg2://airflow_user:yourpassword@postgres/airflow_db
Enter fullscreen mode Exit fullscreen mode

Quick Reference

Mistake What breaks Fix
pip install as root ModuleNotFoundError at runtime USER airflow before install in Dockerfile
airflow db init / db upgrade Command not found Use airflow db migrate
No triggerer service Deferred tasks hang silently Add triggerer to docker-compose
passwords.json as list Cryptic auth startup crash Use dict: {"user": "pass"}
localhost for service URLs Connection refused in tasks Use Docker service names as hostnames

Starting a fresh Airflow 3 project? This config skips every one of these. Migrating from Airflow 2? The Dockerfile user order and db migrate rename will catch you first. The rest are additive changes specific to Airflow 3.

Hit a different gotcha not on this list? Drop it in the comments.

Follow me on dev.to for more data engineering content, or check out the full pipeline code at github.com/declerke.

Top comments (0)