Amos Augo

Posted on Oct 13

Containerization for Data Engineering: A Practical Guide with Docker and Docker Compose

#docker

1. Introduction

Data engineers today face numerous challenges: environment inconsistencies between development and production, dependency conflicts when different projects require different library versions, and scaling issues as data volumes grow. Containerization solves these problems by packaging applications and their dependencies into isolated, portable units. In this guide, you'll learn how to use Docker and Docker Compose to build reproducible data engineering environments that run consistently anywhere. This practical guide is designed for data engineers, analysts, and developers who want to automate and scale their data pipelines efficiently.

2. Understanding Containerization in Data Engineering

Containerization is a lightweight virtualization technology that packages an application with all its dependencies, libraries, system tools, code, and runtime, into a single, self-contained unit called a container. Unlike virtual machines, which require a full operating system for each instance, containers share the host system's kernel, making them faster to start and more resource-efficient.

For data engineering, this technology offers significant benefits:

Reproducibility: Ensure pipelines run identically across different environments
Portability: Move containers seamlessly between local machines, cloud platforms, and on-premises servers
Scalability: Quickly scale services up or down based on workload demands
Collaboration: Share standardized environments across teams

Typical components in a data engineering stack that benefit from containerization include ETL scripts, databases (PostgreSQL, MySQL), schedulers (Airflow), processing engines (Spark), and visualization tools (Grafana).

3. Key Docker Concepts You Need to Know

Docker Images & Containers: An image is a read-only template with instructions for creating a container, while a container is a runnable instance of an image.
Dockerfile: A text document containing all commands to build a Docker image.
Docker Hub: A registry of Docker images where you can pull base images for common applications.
Volumes & Networks: Volumes persist data beyond container lifecycles, while networks enable secure communication between containers.
Docker CLI Basics: Essential commands include docker build (create images), docker run (start containers), docker ps (list running containers), and docker stop (halt containers).

4. Setting Up a Data Engineering Environment

Let's build a simple pipeline environment with Airflow for orchestration, PostgreSQL for data storage, and Python scripts for data transformation.

Folder Structure:

├── docker-compose.yml
├── airflow/
│   ├── Dockerfile
│   └── dags/
├── postgres/
│   └── init.sql
├── scripts/
│   └── data_transformation.py
└── requirements.txt

Step-by-Step Setup:

Containerize Airflow: Create a custom Dockerfile to extend the official Airflow image and install additional Python packages.
Configure PostgreSQL: Use the official PostgreSQL image with initialization scripts for database schema.
Package Transformation Scripts: Build a custom image for data processing tasks with required dependencies.
Define Dependencies: List Python packages in requirements.txt for consistent installation.

5. Using Docker Compose to Orchestrate Services

Docker Compose simplifies multi-container setups by allowing you to define and manage them in a single YAML file. Instead of starting each container manually with complex docker run commands, you can start all services with one command.

Example docker-compose.yml:

version: '3.8'
services:
  postgres:
    image: postgres:13
    environment:
      POSTGRES_DB: airflow
      POSTGRES_USER: airflow
      POSTGRES_PASSWORD: airflow
    volumes:
      - postgres_data:/var/lib/postgresql/data

  airflow:
    build: ./airflow
    depends_on:
      - postgres
    environment:
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    ports:
      - "8080:8080"
    volumes:
      - ./airflow/dags:/opt/airflow/dags

volumes:
  postgres_data:

Essential Commands:

docker compose up -d: Start all services in detached mode
docker compose ps: Check status of running services
docker compose down: Stop and remove all containers

6. Practical Example: Containerizing a Mini Data Pipeline

Let's implement a complete pipeline that ingests, transforms, stores, and visualizes data.

Objective: Ingest → Transform → Store → Visualize

Step 1: Extract - A Python script pulls mock data from an API or generates synthetic data.
Step 2: Transform - Use PySpark or pandas within a container to clean and process the data.
Step 3: Load - Load transformed data into PostgreSQL using appropriate connectors.
Step 4: Orchestrate - Schedule the pipeline tasks using Airflow DAGs.
Step 5: Monitor - Connect Grafana to PostgreSQL to create dashboards for data visualization.

Docker Compose Integration:

services:
  # ... previous services
  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    depends_on:
      - postgres

7. Common Pitfalls & Best Practices

Pitfalls to Avoid:

Storing credentials directly in Dockerfiles or compose files (use .env files instead)
Forgetting volume mounts, leading to data loss when containers restart
Ignoring container logs and resource usage, making debugging difficult

Best Practices:

Use lightweight base images (python:3.9-slim instead of python:3.9) to reduce image size
Version your images (e.g., myapp:v1.2) for better traceability
Document your Compose services with comments in the YAML file
Follow the single-responsibility principle, each container should have one specific purpose

8. Scaling & Extending the Setup

As your data needs grow, Docker Compose makes scaling straightforward:

Scale services: docker compose up --scale spark-worker=3 to add multiple Spark workers
Production deployment: Consider Kubernetes or AWS ECS for orchestration at scale
CI/CD integration: Automate image building and testing in your deployment pipeline
Enhanced monitoring: Integrate Prometheus for metrics collection and Loki for log aggregation

9. Conclusion

Containerization fundamentally simplifies data engineering by providing consistent, reproducible environments that eliminate "it works on my machine" problems. Docker and Docker Compose offer practical tools to build, share, and scale data pipelines efficiently. Your next step should be to containerize a simple ETL pipeline from your own work. For a production-ready implementation, check out our follow-up article "Containerizing Airflow for Production" on our blog.

10. Appendix

Resources:

Full Working Example:

# docker-compose.yml
version: '3.8'
services:
  spark:
    image: bitnami/spark:3.5.0
    environment:
      SPARK_MODE: master
    ports:
      - "8080:8080"
      - "7077:7077"

  postgres:
    image: postgres:13
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
      - db_data:/var/lib/postgresql/data

  data_ingestion:
    build: ./data_ingestion_service
    depends_on:
      - postgres
    environment:
      DATABASE_URL: postgres://user:password@postgres:5432/mydatabase

volumes:
  db_data:

DEV Community