Part 13 - Local Development and Docker Compose 🐳

This part continues from the Superset automation and explains how the repository is meant to run locally.

The local stack

The local environment is defined in docker-compose.yml. It brings up:

• PostgreSQL,
• Airflow webserver,
• Airflow scheduler,
• Airflow initialization,
• Superset initialization,
• and the Superset web server.

That gives the project a full end-to-end development stack without needing AWS.

Why Docker Compose is useful here

Docker Compose makes the project easier to understand because every service is declared in one place. A reader can see immediately how Airflow connects to PostgreSQL and how Superset depends on the warehouse.

Makefile workflow

The Makefile provides short commands for common actions such as:

• make install,
• make local-up,
• make local-init,
• make generate-egypt-stations,
• make demo-run.

That is good developer ergonomics even if some documentation still needs cleanup.

What to notice in the local setup

The project makes a few decisions that are worth learning from:

• the same code works locally and in cloud mode,
• the local container mounts the DAGs, scripts, and spark jobs,
• the environment file drives runtime values,
• and the demo run can be triggered after the stack is ready.

Continue

The final part explains the Terraform deployment, the EC2 bootstrap flow, and the lessons learned from building the project as a Zoomcamp final project.

Continue to Part 14: Cloud Deployment and Lessons Learned.

Tag: #dataengineeringzoomcamp