Big Data Fundamentals: data pipeline example

Building a Production-Grade CDC Pipeline with Apache Iceberg and Spark

Introduction

The need for near real-time analytics on transactional data is a constant pressure in modern data platforms. We recently faced a challenge at scale: ingesting changes from a high-volume, multi-terabyte PostgreSQL database into a data lake for downstream fraud detection and customer behavior analysis. Traditional batch ETL processes couldn’t meet the latency requirements (target: < 5 minutes end-to-end). Furthermore, the schema of the source database was evolving rapidly, necessitating a robust schema evolution strategy. This led us to design a Change Data Capture (CDC) pipeline leveraging Debezium, Apache Kafka, Apache Spark, and Apache Iceberg. The pipeline handles approximately 500 million changes per day, with a peak throughput of 10,000 events/second, while maintaining sub-second query latency on the resulting Iceberg tables. Cost efficiency is paramount, so we focused on optimizing Spark resource allocation and storage tiering.

What is a CDC Pipeline in Big Data Systems?

A CDC pipeline, in this context, is a data integration pattern that captures and propagates data changes from a source database to a target data store. It’s fundamentally different from traditional ETL, which operates on full snapshots. CDC focuses on incremental updates, minimizing data transfer and processing overhead. Our implementation utilizes a publish-subscribe model. Debezium, acting as a change data connector, captures database changes in real-time and publishes them as events to Kafka topics. Spark Structured Streaming consumes these events, transforms them as needed, and writes them to Apache Iceberg tables.

Iceberg is crucial here. Unlike traditional Hive tables, Iceberg provides ACID transactions, schema evolution, and time travel capabilities, essential for handling the dynamic nature of our source data. We use Avro as the serialization format for Kafka messages and Iceberg data files, providing schema evolution support and efficient compression. Protocol-level behavior involves Kafka’s offset management for exactly-once processing semantics and Iceberg’s metadata management for consistent reads and writes.

Real-World Use Cases

1. Fraud Detection: Near real-time detection of fraudulent transactions based on changes in account activity.
2. Customer 360: Building a unified customer view by integrating transactional data with other data sources (e.g., web logs, marketing data).
3. Inventory Management: Tracking inventory levels and movements in real-time for optimized supply chain management.
4. Auditing & Compliance: Maintaining a complete audit trail of data changes for regulatory compliance.
5. Personalized Recommendations: Updating user profiles and recommendation models based on recent activity.

System Design & Architecture

graph LR
    A[PostgreSQL Database] --> B(Debezium Connector);
    B --> C{Apache Kafka};
    C --> D[Spark Structured Streaming];
    D --> E[Apache Iceberg Tables (S3)];
    E --> F[Presto/Trino];
    F --> G[BI Tools/Downstream Applications];

    subgraph Cloud Infrastructure (AWS)
        E
    end

    style A fill:#f9f,stroke:#333,stroke-width:2px
    style B fill:#ccf,stroke:#333,stroke-width:2px
    style C fill:#ccf,stroke:#333,stroke-width:2px
    style D fill:#ccf,stroke:#333,stroke-width:2px
    style E fill:#9cf,stroke:#333,stroke-width:2px
    style F fill:#9cf,stroke:#333,stroke-width:2px
    style G fill:#9cf,stroke:#333,stroke-width:2px

This architecture leverages AWS services. Debezium runs as a self-managed connector within the database network. Kafka is deployed as a managed service (MSK). Spark runs on EMR with transient clusters launched by a workflow orchestrator (Airflow). Iceberg tables are stored in S3, partitioned by event time and a hash of the primary key for even data distribution. Presto/Trino provides a SQL interface for querying the Iceberg tables.

Performance Tuning & Resource Management

Performance is critical. We’ve focused on several key areas:

• Spark Parallelism: spark.sql.shuffle.partitions is set to 400 to maximize parallelism during joins and aggregations. We monitor shuffle read/write sizes in the Spark UI and adjust this value accordingly.
• Kafka Consumer Configuration: fs.s3a.connection.maximum is set to 100 to handle concurrent S3 requests from Spark. kafka.consumer.fetch.min.bytes is tuned to 1MB to reduce the number of fetch requests.
• Iceberg Compaction: We schedule regular compaction jobs to optimize Iceberg metadata and reduce the number of small files. We use the OPTIMIZE command with a target file size of 256MB.
• Memory Management: We carefully configure Spark executor memory (spark.executor.memory) and driver memory (spark.driver.memory) based on the size of the data being processed. We monitor garbage collection activity to identify potential memory leaks.
• File Size: We aim for Parquet file sizes of 128MB-256MB in Iceberg. Smaller files lead to increased metadata overhead and slower query performance.

A recent configuration change, increasing spark.executor.cores from 4 to 8, resulted in a 15% throughput improvement.

Failure Modes & Debugging

Common failure modes include:

• Data Skew: Uneven distribution of data across Spark partitions, leading to performance bottlenecks. We address this using salting techniques or repartitioning based on a more evenly distributed key.
• Out-of-Memory Errors: Insufficient memory allocated to Spark executors or the driver. We increase memory allocation or optimize data processing logic.
• Job Retries: Transient errors (e.g., network issues) can cause Spark jobs to fail and retry. We configure appropriate retry policies in Airflow.
• DAG Crashes: Errors in the Spark application code can cause the entire DAG to crash. We use thorough unit testing and integration testing to prevent this.

Debugging tools include:

• Spark UI: Provides detailed information about Spark job execution, including task durations, shuffle sizes, and memory usage.
• Flink Dashboard (Kafka): Monitors Kafka consumer lag and throughput.
• Datadog: Alerts on key metrics, such as Kafka consumer lag, Spark job failure rate, and S3 storage usage.
• CloudWatch Logs: Provides access to logs from all components of the pipeline.

Data Governance & Schema Management

We use the AWS Glue Data Catalog as our central metadata repository. Debezium automatically captures schema changes from the PostgreSQL database and publishes them to a schema registry. Spark Structured Streaming uses this schema registry to validate incoming data and evolve the Iceberg table schema accordingly. Iceberg’s schema evolution capabilities allow us to add, remove, or rename columns without breaking downstream applications. We enforce schema validation at the Kafka consumer level to reject invalid data.

Security and Access Control

Data is encrypted at rest in S3 using KMS. Access to Kafka topics and Iceberg tables is controlled using IAM policies. We leverage AWS Lake Formation to manage fine-grained access control to Iceberg tables, including row-level security based on user attributes. Audit logging is enabled for all components of the pipeline.

Testing & CI/CD Integration

We use Great Expectations for data quality testing. We define expectations for data types, ranges, and completeness. These expectations are validated as part of our CI/CD pipeline. We also use DBT for data transformation testing. Our CI/CD pipeline includes linting, unit testing, integration testing, and automated regression tests. We deploy changes to a staging environment before promoting them to production.

Common Pitfalls & Operational Misconceptions

1. Ignoring Schema Evolution: Assuming the source schema will remain static. Mitigation: Implement a robust schema registry and leverage Iceberg’s schema evolution capabilities.
2. Insufficient Kafka Partitioning: Not enough Kafka partitions to handle the throughput. Symptom: High Kafka consumer lag. Mitigation: Increase the number of Kafka partitions.
3. Incorrect Spark Configuration: Suboptimal Spark configuration leading to performance bottlenecks. Symptom: Long job durations, high shuffle read/write sizes. Mitigation: Tune Spark configuration based on monitoring metrics.
4. Lack of Monitoring: Not monitoring key metrics, making it difficult to identify and resolve issues. Symptom: Unexpected failures, performance degradation. Mitigation: Implement comprehensive monitoring and alerting.
5. Overlooking Data Skew: Uneven data distribution leading to performance bottlenecks. Symptom: Long task durations for specific partitions. Mitigation: Use salting or repartitioning techniques.

Enterprise Patterns & Best Practices

• Data Lakehouse: Embrace the data lakehouse architecture, combining the benefits of data lakes and data warehouses.
• Batch vs. Streaming: Choose the appropriate processing paradigm based on latency requirements. For near real-time analytics, streaming is essential.
• File Format: Parquet and ORC are excellent choices for columnar storage and efficient compression.
• Storage Tiering: Use S3 lifecycle policies to move infrequently accessed data to cheaper storage tiers.
• Workflow Orchestration: Use a workflow orchestrator (e.g., Airflow, Dagster) to manage the complexity of the pipeline.

Conclusion

Building a production-grade CDC pipeline requires careful consideration of architecture, performance, scalability, and operational reliability. By leveraging technologies like Debezium, Kafka, Spark, and Iceberg, we’ve successfully built a pipeline that meets our stringent latency and data quality requirements. Next steps include benchmarking new Spark configurations, introducing schema enforcement at the source database level, and migrating to a more cost-effective storage tier for historical data. Continuous monitoring and optimization are crucial for maintaining a reliable and scalable data platform.