1. How my idea started
Initially, I wanted to practice working with AWS S3, but I worried about unexpected cloud costs if I couldn't control the billing properly. In the past, I created a free tier account, but unfortunately, it got locked ambiguously. Since credit cards can only be used for one-time free credits, I needed another solution. Luckily, I discovered MinIO. I found this guide - it's very easy to set up using Docker Compose, and all the steps are clear to follow:"
Read the full guide on Medium by Abhishek Jain
In the past, whenever I wanted to build a project using Docker, it would take me several days to resolve library and version issues. And I wasn't always lucky, sometimes my efforts failed completely, which made me want to give up many times. But today, with the help of AI, fixing these configuration issues is much easier. I just let the AI read the error messages and wait for it to help me fix them until everything works. :D
After setting these things, my components look like this:
MinIO holding 3 Parquet files migrated from CSV:
Connected to PostgreSQL using DBeaver to verify data ingestion into the RDBMS:
2. Then airflow was added
To demonstrate an ETL process, I decided to add Airflow by including the Airflow component in my docker-compose.yml file (I will provide the full file at the end of this post). I also added the DAGs and the etl-manager files, all written in Python with suport from Copilot. Now, whenever I append new data, I don't have to start from the main function manually, I can trigger it right from Airflow with a manual start or set it on a schedule.
The generated code met about 90,99% of my requirements, but I had to tweak a few quirks along the way. Don't trust AI-generated code completely.
DAG graph displayed on the Airflow web server (commonly accessible on port 8080):
3. Adding More Dynamic Input
CSV files are good for a demo, but commonly our input data will come from dynamic sources like a live database or an API.
For the scope of my personal storage project, I decided to add an API data source using a free API from Stack Overflow - Question's data. I wrote another DAG file to handle this case.
Fetching data from the API:
Triggering the run through the Airflow:
4. Add a very simple dashboard
I chose Metabase to demonstrate dashboard connectivity and adapt end-to-end functionality, just loading the data from Stack Exchange.
Maybe in the future, I will design a proper data model and metrics to make the dashboard more meaningful.
Metabase looks like it will be more than a static table once we can build a data model with several tables.
To be continue...
File docker-compose.yml for architecture demonstration purposes
services:
postgres:
image: postgres:17
container_name: postgres_lakehouse
mem_limit: 1g
environment:
POSTGRES_USER: ${POSTGRES_USER:-lakehouse_user}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change_me_now}
POSTGRES_DB: ${POSTGRES_DB:-lakehouse_db}
ports:
- "5432:5432"
volumes:
- postgres_data:/var/lib/postgresql/data
- ./init-scripts:/docker-entrypoint-initdb.d
restart: unless-stopped
networks:
- lakehouse_network
minio:
image: minio/minio:latest
container_name: minio_datalake
mem_limit: 512m
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minioadmin}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-change_me_now}
ports:
- "9000:9000" # API endpoint
- "9001:9001" # Web Console
volumes:
- minio_data:/data
command: server /data --console-address ":9001"
restart: unless-stopped
networks:
- lakehouse_network
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
# MinIO Client for easy data management
mc:
image: minio/mc:latest
container_name: minio_client
mem_limit: 128m
depends_on:
- minio
networks:
- lakehouse_network
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER:-minioadmin}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD:-change_me_now}
entrypoint: >
/bin/sh -c "
sleep 10;
mc alias set myminio http://minio:9000 "$${MINIO_ROOT_USER}" "$${MINIO_ROOT_PASSWORD}";
mc mb myminio/lakehouse-data --ignore-existing;
mc mb myminio/raw-data --ignore-existing;
mc mb myminio/processed-data --ignore-existing;
echo 'MinIO buckets created successfully';
tail -f /dev/null;
"
# Python service for ETL operations
python-etl:
build:
context: ./python-etl
dockerfile: Dockerfile
container_name: python_etl
mem_limit: 256m
depends_on:
- postgres
- minio
networks:
- lakehouse_network
volumes:
- ./python-etl:/app
- ./sample-data:/sample-data
environment:
POSTGRES_HOST: ${POSTGRES_HOST:-postgres}
POSTGRES_DB: ${POSTGRES_DB:-lakehouse_db}
POSTGRES_USER: ${POSTGRES_USER:-lakehouse_user}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-change_me_now}
MINIO_ENDPOINT: ${MINIO_ENDPOINT:-minio:9000}
MINIO_ACCESS_KEY: ${MINIO_ROOT_USER:-minioadmin}
MINIO_SECRET_KEY: ${MINIO_ROOT_PASSWORD:-change_me_now}
command: tail -f /dev/null
# 5. Airflow Webserver (UI at localhost:8080)
airflow-webserver:
build:
context: .
dockerfile: Dockerfile.airflow
container_name: airflow_webserver
mem_limit: 768m
restart: unless-stopped
user: airflow
depends_on:
- postgres
- minio
environment:
PYTHONPATH: "/opt/airflow:/opt/airflow/python-etl"
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
AIRFLOW__CORE__EXECUTOR: "SequentialExecutor"
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: "postgresql+psycopg2://${POSTGRES_USER:-lakehouse_user}:${POSTGRES_PASSWORD:-change_me_now}@postgres:5432/${POSTGRES_DB:-lakehouse_db}"
AIRFLOW__CORE__FERNET_KEY: ${AIRFLOW__CORE__FERNET_KEY:-"MRh***SPChyoonjeonghanub4j4WLUCwOQ="}
AIRFLOW__WEBSERVER__SECRET_KEY: ${AIRFLOW__WEBSERVER__SECRET_KEY:-"ICPR***tjT5cqxyoonjeonghanyg4P_VUMV_nI"}
AIRFLOW__CORE__TIMEZONE: "Asia/Ho_Chi_Minh"
AIRFLOW__WEBSERVER__SESSION_BACKEND: "securecookie"
GUNICORN_CMD_ARGS: "--workers 2 --timeout 180 --keep-alive 30"
volumes:
- ./dags:/opt/airflow/dags
- ./python-etl:/opt/airflow/python-etl
- /var/run/docker.sock:/var/run/docker.sock
- ./sample-data:/sample-data
ports:
- "8081:8080"
command: bash -lc "airflow db init && airflow webserver"
networks:
- lakehouse_network
# 6. Airflow Scheduler (Runs tasks / DockerOperator)
airflow-scheduler:
build:
context: .
dockerfile: Dockerfile.airflow
container_name: airflow_scheduler
mem_limit: 512m
restart: unless-stopped
user: airflow
depends_on:
- airflow-webserver
environment:
PYTHONPATH: "/opt/airflow:/opt/airflow/python-etl"
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
AIRFLOW__CORE__EXECUTOR: "SequentialExecutor"
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: "postgresql+psycopg2://${POSTGRES_USER:-lakehouse_user}:${POSTGRES_PASSWORD:-change_me_now}@postgres:5432/${POSTGRES_DB:-lakehouse_db}"
AIRFLOW__CORE__FERNET_KEY: ${AIRFLOW__CORE__FERNET_KEY:-"MRhqlS********OQ="}
AIRFLOW__WEBSERVER__SECRET_KEY: ${AIRFLOW__WEBSERVER__SECRET_KEY:-"ICPR-JvttjT5cq**********_VUMV_nI"}
AIRFLOW__CORE__TIMEZONE: "Asia/Ho_Chi_Minh"
AIRFLOW__WEBSERVER__SESSION_BACKEND: "securecookie"
volumes:
- ./dags:/opt/airflow/dags
- ./python-etl:/opt/airflow/python-etl
- /var/run/docker.sock:/var/run/docker.sock
- ./sample-data:/sample-data
networks:
- lakehouse_network
command: bash -lc "airflow db init && airflow scheduler"
metabase:
image: metabase/metabase:latest
container_name: metabase_app
mem_limit: 2g <---------------------[[[[ eating ram like a bulldozer]]]]
restart: unless-stopped
ports:
- "3000:3000"
environment:
MB_DB_TYPE: postgres
MB_DB_DBNAME: ${POSTGRES_DB:-lakehouse_db}
MB_DB_PORT: 5432
MB_DB_USER: ${POSTGRES_USER:-lakehouse_user}
MB_DB_PASS: ${POSTGRES_PASSWORD:-change_me_now}
MB_DB_HOST: postgres
MB_JVM_OPTIONS: "-Xms768m -Xmx1g"
networks:
- lakehouse_network
depends_on:
- postgres
networks:
lakehouse_network:
driver: bridge
volumes:
postgres_data:
minio_data:
Top comments (0)