Building a Simple ETL Pipeline with Docker, MinIO, and Airflow

Tran Huynh — Sun, 16 Aug 2026 08:57:07 +0000

1. How my idea started

Initially, I wanted to practice working with AWS S3, but I worried about unexpected cloud costs if I couldn't control the billing properly. In the past, I created a free tier account, but unfortunately, it got locked ambiguously. Since credit cards can only be used for one-time free credits, I needed another solution. Luckily, I discovered MinIO. I found this guide - it's very easy to set up using Docker Compose, and all the steps are clear to follow:"
Read the full guide on Medium by Abhishek Jain

In the past, whenever I wanted to build a project using Docker, it would take me several days to resolve library and version issues. And I wasn't always lucky, sometimes my efforts failed completely, which made me want to give up many times. But today, with the help of AI, fixing these configuration issues is much easier. I just let the AI read the error messages and wait for it to help me fix them until everything works. :D

After setting these things, my components look like this:

MinIO holding 3 Parquet files migrated from CSV:

Connected to PostgreSQL using DBeaver to verify data ingestion into the RDBMS:

2. Then airflow was added

To demonstrate an ETL process, I decided to add Airflow by including the Airflow component in my docker-compose.yml file (I will provide the full file at the end of this post). I also added the DAGs and the etl-manager files, all written in Python with suport from Copilot. Now, whenever I append new data, I don't have to start from the main function manually, I can trigger it right from Airflow with a manual start or set it on a schedule.

The generated code met about 90,99% of my requirements, but I had to tweak a few quirks along the way. Don't trust AI-generated code completely.

DAG graph displayed on the Airflow web server (commonly accessible on port 8080):

3. Adding More Dynamic Input

CSV files are good for a demo, but commonly our input data will come from dynamic sources like a live database or an API.
For the scope of my personal storage project, I decided to add an API data source using a free API from Stack Overflow - Question's data. I wrote another DAG file to handle this case.

Fetching data from the API:

Triggering the run through the Airflow:

4. Add a very simple dashboard

I chose Metabase to demonstrate dashboard connectivity and adapt end-to-end functionality, just loading the data from Stack Exchange.
Maybe in the future, I will design a proper data model and metrics to make the dashboard more meaningful.

Metabase looks like it will be more than a static table once we can build a data model with several tables.

To be continue...

File docker-compose.yml for architecture demonstration purposes

services:
  postgres:
    image: postgres:17
    container_name: postgres_lakehouse
    mem_limit: 1g
    environment:
      POSTGRES_USER: ${POSTGRES_USER:-lakehouse_user}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change_me_now}
      POSTGRES_DB: ${POSTGRES_DB:-lakehouse_db}
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
      - ./init-scripts:/docker-entrypoint-initdb.d
    restart: unless-stopped
    networks:
      - lakehouse_network

  minio:
    image: minio/minio:latest
    container_name: minio_datalake
    mem_limit: 512m
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minioadmin}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-change_me_now}
    ports:
      - "9000:9000"   # API endpoint
      - "9001:9001"   # Web Console
    volumes:
      - minio_data:/data
    command: server /data --console-address ":9001"
    restart: unless-stopped
    networks:
      - lakehouse_network
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  # MinIO Client for easy data management
  mc:
    image: minio/mc:latest
    container_name: minio_client
    mem_limit: 128m
    depends_on:
      - minio
    networks:
      - lakehouse_network
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minioadmin}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-change_me_now}
    entrypoint: >
      /bin/sh -c "
      sleep 10;
      mc alias set myminio http://minio:9000 "$${MINIO_ROOT_USER}" "$${MINIO_ROOT_PASSWORD}";
      mc mb myminio/lakehouse-data --ignore-existing;
      mc mb myminio/raw-data --ignore-existing;
      mc mb myminio/processed-data --ignore-existing;
      echo 'MinIO buckets created successfully';
      tail -f /dev/null;
      "

  # Python service for ETL operations
  python-etl:
    build:
      context: ./python-etl
      dockerfile: Dockerfile
    container_name: python_etl
    mem_limit: 256m
    depends_on:
      - postgres
      - minio
    networks:
      - lakehouse_network
    volumes:
      - ./python-etl:/app
      - ./sample-data:/sample-data
    environment:
      POSTGRES_HOST: ${POSTGRES_HOST:-postgres}
      POSTGRES_DB: ${POSTGRES_DB:-lakehouse_db}
      POSTGRES_USER: ${POSTGRES_USER:-lakehouse_user}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change_me_now}
      MINIO_ENDPOINT: ${MINIO_ENDPOINT:-minio:9000}
      MINIO_ACCESS_KEY: ${MINIO_ROOT_USER:-minioadmin}
      MINIO_SECRET_KEY: ${MINIO_ROOT_PASSWORD:-change_me_now}
    command: tail -f /dev/null

  # 5. Airflow Webserver (UI at localhost:8080)
  airflow-webserver:
    build:
      context: .
      dockerfile: Dockerfile.airflow
    container_name: airflow_webserver
    mem_limit: 768m
    restart: unless-stopped
    user: airflow
    depends_on:
      - postgres
      - minio
    environment:
      PYTHONPATH: "/opt/airflow:/opt/airflow/python-etl"
      AIRFLOW__CORE__LOAD_EXAMPLES: "False"
      AIRFLOW__CORE__EXECUTOR: "SequentialExecutor"
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: "postgresql+psycopg2://${POSTGRES_USER:-lakehouse_user}:${POSTGRES_PASSWORD:-change_me_now}@postgres:5432/${POSTGRES_DB:-lakehouse_db}"
      AIRFLOW__CORE__FERNET_KEY: ${AIRFLOW__CORE__FERNET_KEY:-"MRh***SPChyoonjeonghanub4j4WLUCwOQ="}
      AIRFLOW__WEBSERVER__SECRET_KEY: ${AIRFLOW__WEBSERVER__SECRET_KEY:-"ICPR***tjT5cqxyoonjeonghanyg4P_VUMV_nI"}
      AIRFLOW__CORE__TIMEZONE: "Asia/Ho_Chi_Minh"
      AIRFLOW__WEBSERVER__SESSION_BACKEND: "securecookie"
      GUNICORN_CMD_ARGS: "--workers 2 --timeout 180 --keep-alive 30"
    volumes:
      - ./dags:/opt/airflow/dags
      - ./python-etl:/opt/airflow/python-etl
      - /var/run/docker.sock:/var/run/docker.sock
      - ./sample-data:/sample-data
    ports:
      - "8081:8080"
    command: bash -lc "airflow db init && airflow webserver"
    networks:
      - lakehouse_network

  # 6. Airflow Scheduler (Runs tasks / DockerOperator)
  airflow-scheduler:
    build:
      context: .
      dockerfile: Dockerfile.airflow
    container_name: airflow_scheduler
    mem_limit: 512m
    restart: unless-stopped
    user: airflow
    depends_on:
      - airflow-webserver
    environment:
      PYTHONPATH: "/opt/airflow:/opt/airflow/python-etl"
      AIRFLOW__CORE__LOAD_EXAMPLES: "False"
      AIRFLOW__CORE__EXECUTOR: "SequentialExecutor"
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: "postgresql+psycopg2://${POSTGRES_USER:-lakehouse_user}:${POSTGRES_PASSWORD:-change_me_now}@postgres:5432/${POSTGRES_DB:-lakehouse_db}"
      AIRFLOW__CORE__FERNET_KEY: ${AIRFLOW__CORE__FERNET_KEY:-"MRhqlS********OQ="}
      AIRFLOW__WEBSERVER__SECRET_KEY: ${AIRFLOW__WEBSERVER__SECRET_KEY:-"ICPR-JvttjT5cq**********_VUMV_nI"}
      AIRFLOW__CORE__TIMEZONE: "Asia/Ho_Chi_Minh"
      AIRFLOW__WEBSERVER__SESSION_BACKEND: "securecookie"
    volumes:
      - ./dags:/opt/airflow/dags
      - ./python-etl:/opt/airflow/python-etl
      - /var/run/docker.sock:/var/run/docker.sock
      - ./sample-data:/sample-data
    networks:
      - lakehouse_network
    command: bash -lc "airflow db init && airflow scheduler"

  metabase:
    image: metabase/metabase:latest
    container_name: metabase_app
    mem_limit: 2g <---------------------[[[[ eating ram like a bulldozer]]]]
    restart: unless-stopped
    ports:
      - "3000:3000"
    environment:
      MB_DB_TYPE: postgres
      MB_DB_DBNAME: ${POSTGRES_DB:-lakehouse_db}
      MB_DB_PORT: 5432
      MB_DB_USER: ${POSTGRES_USER:-lakehouse_user}
      MB_DB_PASS: ${POSTGRES_PASSWORD:-change_me_now}
      MB_DB_HOST: postgres
      MB_JVM_OPTIONS: "-Xms768m -Xmx1g"
    networks:
      - lakehouse_network
    depends_on:
      - postgres

networks:
  lakehouse_network:
    driver: bridge

volumes:
  postgres_data:
  minio_data: