DEV Community

Cover image for Dockerized Spark and Kafka: YouTube Data Pipeline Implementation
Lagat Josiah
Lagat Josiah

Posted on

Dockerized Spark and Kafka: YouTube Data Pipeline Implementation

#dataengineering #architecture #docker #tutorial

Dockerized Spark and Kafka: YouTube Data Pipeline Implementation

Executive Summary

This presentation demonstrates a production-grade, containerized data engineering pipeline that extracts, processes, and analyzes YouTube data using Apache Airflow for orchestration, Apache Kafka for real-time streaming, and Apache Spark for distributed data processing—all deployed using Docker containerization.

Project Architecture Overview

Present the high-level architecture showcasing:

  • Docker containers hosting Airflow, Kafka, Spark, and Jupyter components
  • Data flow from YouTube API through Kafka topics to Spark processing
  • PostgreSQL database for metadata storage
  • End-to-end pipeline orchestration

Step-by-Step Implementation Process

1. Project Structure and Organization

Objective: Establish a well-organized, maintainable project structure

youtube-pipeline/
├── docker-compose.yml          # Multi-container orchestration
├── .env                        # Environment variables and API keys
├── airflow/
│   ├── Dockerfile             # Airflow container configuration
│   ├── dags/
│   │   └── youtube_pipeline.py    # Workflow definitions
│   └── requirements.txt       # Airflow dependencies
├── spark/
│   ├── Dockerfile             # Spark container configuration
│   └── scripts/
│       └── process_youtube_data.py  # Data processing logic
├── jupyter/
│   ├── Dockerfile             # Jupyter container configuration
│   └── notebooks/
│       └── youtube_analysis.ipynb   # Analysis and visualization
├── youtube_extractor/
│   ├── Dockerfile             # Extractor container configuration
│   ├── extractor.py           # YouTube API integration
│   └── requirements.txt       # Extractor dependencies
└── certificates/
    └── ca.pem                 # SSL/TLS certificates
Enter fullscreen mode Exit fullscreen mode

  • Root Level:

    • docker-compose.yml - Orchestrates all services and their dependencies
    • .env - Centralized configuration for API keys, credentials, and environment variables

  • Airflow Module:

    • Contains workflow orchestration logic and DAG definitions
    • Isolated dependencies via dedicated requirements.txt

  • Spark Module:

    • Houses distributed data processing scripts
    • Separate container for scalability and resource management

  • Jupyter Module:

    • Provides interactive analysis environment
    • Notebooks for data exploration and visualization

  • YouTube Extractor Module:

    • Standalone service for data extraction from YouTube API
    • Independent scaling and deployment

  • Certificates:

    • Secure communication certificates for production deployment

2. Environment Configuration

Objective: Configure all necessary environment variables and dependencies

  • Configuration Files Setup:

    • Environment variables for API keys and credentials
    • Service-specific configurations (Airflow, Kafka, Spark)
    • Network and port configurations
    • Volume mounting for data persistence

  • Dependencies Management:

    • requirements.txt files for Python dependencies
    • ca.pem certificate configuration for secure connections
    • .env file structure and security considerations

3. Docker Containerization Strategy

Objective: Design and implement isolated, scalable containers

  • Dockerfile Architecture:

    • Base images selection and justification
    • Multi-stage builds for optimization
    • Dependency installation and layer caching
    • Custom plugin integration

  • Docker Compose Orchestration:

    • Service definitions for each component
    • Network configuration and inter-service communication
    • Volume mounts for code and data persistence
    • Health checks and restart policies
    • Port mappings and external access

4. YouTube Data Extraction Module

Objective: Implement robust data extraction from YouTube API

  • Extractor Service (youtube_extractor/extractor.py):
    • YouTube Data API v3 integration
    • Channel statistics retrieval (views, subscribers, video count)
    • Rate limiting and quota management
    • Error handling and retry logic 
# Example: YouTube API call structure
url = "https://www.googleapis.com/youtube/v3/channels"
params = {
    'part': 'snippet,statistics',
    'id': channel_id,
    'key': api_key
}
response = requests.get(url, params=params)
Enter fullscreen mode Exit fullscreen mode
  • Data Schema:
    • Channel ID, title, and description
    • Statistics: total views, subscriber count, video count
    • Timestamps for tracking data freshness

5. Apache Airflow Orchestration

Objective: Automate and schedule the complete data pipeline

  • DAG Development (airflow/dags/youtube_pipeline.py):
    • Pipeline definition with task dependencies
    • Schedule interval: Every 6 hours for fresh data
    • Retry logic and failure handling 
default_args = {
    'owner': 'airflow',
    'start_date': datetime(2024, 1, 1),
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

with DAG(
    'youtube_data_pipeline',
    default_args=default_args,
    schedule_interval=timedelta(hours=6),
    catchup=False,
    tags=['youtube'],
) as dag:
    create_tables >> extract_data
Enter fullscreen mode Exit fullscreen mode

  • Task Flow:

    1. create_tables: Initialize PostgreSQL database schema
    2. extract_youtube_data: Fetch data from YouTube API and load into database

  • Database Schema Creation:

CREATE TABLE IF NOT EXISTS channel_stats (
    channel_id VARCHAR(255) PRIMARY KEY,
    channel_title TEXT,
    channel_description TEXT,
    channel_created_at TIMESTAMP,
    total_views BIGINT,
    subscriber_count BIGINT,
    video_count BIGINT,
    processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
Enter fullscreen mode Exit fullscreen mode
  • Data Loading Strategy:
    • PostgresHook for database connections
    • UPSERT operations (INSERT ... ON CONFLICT) for idempotency
    • Batch processing for multiple channels
    • Transaction management and error recovery

6. PostgreSQL Data Storage

Objective: Persist and manage extracted YouTube data

  • Database Configuration:
    • PostgreSQL 13 containerization
    • Connection string: postgresql+psycopg2://postgres:password@postgres:5432/youtube_db
    • Volume mounting for data persistence across container restarts
    • Health checks for service availability 
postgres:
  image: postgres:13
  environment:
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: password
    POSTGRES_DB: youtube_db
  ports:
    - "5433:5432"
  volumes:
    - ./postgres_data:/var/lib/postgresql/data
Enter fullscreen mode Exit fullscreen mode
  • Data Models:
    • channel_stats: Channel-level metrics and metadata
    • video_stats: Video-level analytics (prepared for future enhancement)
    • Timestamp tracking for temporal analysis
    • Primary key constraints for data integrity

7. Analysis and Visualization with Jupyter

Objective: Provide interactive data analysis and insights visualization

  • Jupyter Notebook Environment:
    • Jupyter DataScience Notebook with pre-installed analytics libraries
    • Port 8888 with token-based authentication
    • Volume mounting for notebook persistence 
docker run -d --name jupyter -p 8888:8888 \
  -v $(pwd)/jupyter/notebooks:/home/jovyan/work \
  -e JUPYTER_TOKEN=password \
  jupyter/datascience-notebook:latest
Enter fullscreen mode Exit fullscreen mode

  • Analysis Capabilities (jupyter/notebooks/youtube_analysis.ipynb):

    • Database connectivity using psycopg2 or SQLAlchemy
    • Exploratory Data Analysis (EDA) of channel metrics
    • Time-series analysis of subscriber growth
    • Visualization using matplotlib, seaborn, and plotly

  • Example Analysis Workflow:

import pandas as pd
import psycopg2
import matplotlib.pyplot as plt

# Connect to PostgreSQL
conn = psycopg2.connect(
    host="postgres",
    port=5432,
    database="youtube_db",
    user="postgres",
    password="password"
)

# Load channel statistics
df = pd.read_sql_query("""
    SELECT channel_title, subscriber_count, total_views, video_count
    FROM channel_stats
    ORDER BY subscriber_count DESC
""", conn)

# Visualize subscriber distribution
df.plot(x='channel_title', y='subscriber_count', kind='bar', 
        title='YouTube Channels by Subscriber Count')
plt.show()

# Calculate engagement metrics
df['avg_views_per_video'] = df['total_views'] / df['video_count']
df['engagement_ratio'] = df['total_views'] / df['subscriber_count']
Enter fullscreen mode Exit fullscreen mode
  • Interactive Features:
    • Real-time data queries from PostgreSQL
    • Custom metrics calculation and visualization
    • Export analysis results to CSV/Excel
    • Dashboard creation for monitoring channel performance

8. Deployment and Automation

Objective: Ensure smooth deployment and operational efficiency

  • Automated Deployment Script (start.sh):
    • Directory structure initialization
    • Environment configuration validation
    • Docker Compose orchestration
    • Service health checks and startup verification 
#!/bin/bash
# Create necessary directories
mkdir -p airflow/dags airflow/logs airflow/plugins jupyter/notebooks

# Start core services
docker-compose up -d

# Start Jupyter
docker run -d --name jupyter -p 8888:8888 \
  -v $(pwd)/jupyter/notebooks:/home/jovyan/work \
  -e JUPYTER_TOKEN=password \
  jupyter/datascience-notebook:latest
Enter fullscreen mode Exit fullscreen mode
  • Configuration Management:
    • .env file for sensitive credentials (YouTube API key)
    • Airflow Variables for runtime configuration
    • Environment variable injection into containers 
# Set YouTube API key in Airflow
docker-compose exec airflow airflow variables set YOUTUBE_API_KEY 'your_actual_key'

# Trigger pipeline manually
docker-compose exec airflow airflow dags trigger youtube_data_pipeline
Enter fullscreen mode Exit fullscreen mode
  • Service Initialization Sequence:
    1. PostgreSQL starts with health checks
    2. Airflow waits for database availability
    3. Database schema initialization
    4. Admin user creation (username: admin, password: admin)
    5. Webserver and scheduler startup
    6. Jupyter notebook service launch

9. Testing and Validation

Objective: Verify system functionality and data integrity

  • Component Testing:

    • Database connectivity verification
    • YouTube API authentication testing
    • Airflow DAG validation and syntax checking
    • Jupyter notebook execution testing

  • Integration Testing:

    • End-to-end pipeline execution
    • Data flow validation: API → Airflow → PostgreSQL
    • Query results verification in database 
-- Verify data loaded successfully
SELECT channel_title, subscriber_count, processed_at
FROM channel_stats
ORDER BY subscriber_count DESC;
Enter fullscreen mode Exit fullscreen mode
  • Service Health Verification: 
# Check PostgreSQL
docker-compose exec postgres pg_isready -U postgres

# Check Airflow status
docker-compose exec airflow airflow dags list

# View pipeline logs
docker-compose logs airflow
Enter fullscreen mode Exit fullscreen mode
  • Data Quality Checks:
    • Null value validation in critical fields
    • Timestamp accuracy verification
    • Duplicate record detection (enforced by PRIMARY KEY)
    • API response error handling validation

10. Live Demonstration

Objective: Showcase the working pipeline in action

  • System Startup: 
# Execute automated startup script
./start.sh
Enter fullscreen mode Exit fullscreen mode
  • Watch service initialization sequence
  • Display container health checks passing

  • Show successful database connection

    • Access Services:

  • Jupyter Notebook: http://localhost:8888 (token: password)

  • Airflow Web UI: http://localhost:8080 (admin/admin)

  • PostgreSQL: localhost:5433 (postgres/password)

    • Configure API Key: 
docker-compose exec airflow airflow variables set YOUTUBE_API_KEY 'AIza...'
Enter fullscreen mode Exit fullscreen mode
  • Trigger Data Pipeline: 
# Manual trigger
docker-compose exec airflow airflow dags trigger youtube_data_pipeline

# View DAG execution in Airflow UI
# Navigate to: http://localhost:8080 → DAGs → youtube_data_pipeline
Enter fullscreen mode Exit fullscreen mode

  • Monitor Pipeline Execution:

    • Show Airflow DAG graph view with task dependencies
    • Display task logs showing YouTube API calls
    • Demonstrate successful data extraction messages: 
    ✓ Extracted: Google for Developers - 1.2M subscribers
✓ Extracted: YouTube Creators - 3.5M subscribers
✓ Extracted: Marques Brownlee - 18M subscribers
✅ Successfully loaded 3 channels into database

  • Verify Data in PostgreSQL:

docker-compose exec postgres psql -U postgres -d youtube_db
Enter fullscreen mode Exit fullscreen mode 
SELECT channel_title, subscriber_count, total_views 
FROM channel_stats 
ORDER BY subscriber_count DESC;
Enter fullscreen mode Exit fullscreen mode

  • Jupyter Analysis Demonstration:

    • Open youtube_analysis.ipynb in browser
    • Connect to PostgreSQL database
    • Execute data analysis cells showing:
    • Subscriber count bar chart
    • Views vs. subscribers scatter plot
    • Engagement metrics calculation
    • Top performing channels table

  • Real-time Updates:

    • Show scheduled execution (runs every 6 hours)
    • Demonstrate UPSERT logic by re-triggering DAG
    • Display updated timestamps in database

Technical Stack Summary

Component Technology Purpose Port Credentials
Orchestration Apache Airflow 2.8.1 Workflow automation & scheduling 8080 admin/admin
Database PostgreSQL 13 Data persistence 5433 postgres/password
Analysis Jupyter DataScience Notebook Interactive data exploration 8888 token: password
API Integration YouTube Data API v3 Channel statistics extraction - API Key required
Container Orchestration Docker Compose 3.8 Multi-service management - -

Key Achievements and Learning Outcomes

Docker Mastery:

  • Multi-container orchestration with Docker Compose
  • Service dependency management and health checks
  • Volume management for data persistence
  • Network configuration and inter-container communication
  • Environment variable management for security

Data Engineering Pipeline:

  • ETL workflow design and implementation
  • Scheduled data extraction with retry logic
  • Database schema design and optimization
  • UPSERT operations for idempotent data loading
  • API integration with rate limiting considerations

Workflow Orchestration:

  • Airflow DAG creation with task dependencies
  • PostgreSQL operator for DDL operations
  • Python operators for custom logic
  • Connection and variable management
  • Monitoring and logging configuration

Data Analysis:

  • Jupyter notebook integration with containerized databases
  • SQL queries for data retrieval and analysis
  • Data visualization using Python libraries
  • Interactive dashboard development
  • Metric calculation and KPI tracking

Challenges and Solutions

Challenge 1: Airflow Initialization Timing

Solution: Implemented health checks and database readiness verification in docker-compose.yml before starting Airflow services

Challenge 2: API Key Management

Solution: Used Airflow Variables instead of environment variables for secure, runtime-configurable API key storage

Challenge 3: Data Persistence Across Container Restarts

Solution: Configured volume mounts for PostgreSQL data directory (./postgres_data:/var/lib/postgresql/data)

Challenge 4: Service Accessibility

Solution: Mapped distinct ports (8080 for Airflow, 8888 for Jupyter, 5433 for PostgreSQL) to avoid conflicts

Future Enhancements

  1. Scalability Enhancements:

    • Implement Apache Spark for large-scale data processing
    • Add Kafka for real-time streaming architecture
    • Kubernetes migration for production deployment
    • Horizontal scaling of Airflow workers

  2. Feature Expansion:

    • Video-level analytics extraction
    • Sentiment analysis on comments
    • Trend detection and forecasting
    • Real-time alerting for subscriber milestones
    • REST API for external data access

  3. Advanced Analytics:

    • Machine learning models for growth prediction
    • Comparative channel analysis
    • Engagement rate optimization insights
    • Content performance correlation analysis

  4. Performance Optimization:

    • Connection pooling for database operations
    • Caching layer for frequently accessed data
    • Batch processing optimization
    • Query performance tuning with indexes

Conclusion

This project successfully demonstrates the integration of industry-standard technologies (Docker, Airflow, PostgreSQL, and Jupyter) to build a robust, automated data pipeline. The containerized approach ensures portability, reproducibility, and ease of deployment across different environments.

The implementation showcases practical skills in:

  • Cloud-native application development using containerization
  • ETL pipeline design with scheduled execution and error handling
  • API integration with external data sources (YouTube Data API)
  • Data persistence and schema design with relational databases
  • Interactive data analysis using Jupyter notebooks and Python
  • DevOps practices including automation, monitoring, and configuration management

Key metrics achieved:

  • 3 YouTube channels monitored continuously
  • Automatic updates every 6 hours with retry logic
  • Zero-downtime deployment with health checks
  • Interactive analysis capabilities through Jupyter
  • Scalable architecture ready for additional data sources

The complete source code, documentation, and deployment scripts demonstrate production-ready practices suitable for real-world data engineering scenarios.

Top comments (0)