DEV Community

# dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
The Offline Data Engineer: Building Resilient API Pipelines that Work on an Airplane

The Offline Data Engineer: Building Resilient API Pipelines that Work on an Airplane

4
Comments
5 min read
Understanding Kafka Architecture, Schema Registry, ksqlDB, PostgreSQL, Couchbase, and Microservices

Understanding Kafka Architecture, Schema Registry, ksqlDB, PostgreSQL, Couchbase, and Microservices

2
Comments
3 min read
The "Shift-Left" Imperative: Implementing Data Contracts in CI/CD Pipeline

The "Shift-Left" Imperative: Implementing Data Contracts in CI/CD Pipeline

Comments
4 min read
Building a 75,000-Product Image Feature Dataset for the Amazon ML Challenge 2025

Building a 75,000-Product Image Feature Dataset for the Amazon ML Challenge 2025

1
Comments
4 min read
An Exploration of the Commercial Iceberg Catalog Ecosystem

An Exploration of the Commercial Iceberg Catalog Ecosystem

Comments
14 min read
đź§  ClickHouse LEFT JOINs: Why join_use_nulls Matters

đź§  ClickHouse LEFT JOINs: Why join_use_nulls Matters

5
Comments
2 min read
Getting Started Building a Data Platform

Getting Started Building a Data Platform

Comments
3 min read
Building a Universal Lakehouse Catalog: Beyond Iceberg Tables

Building a Universal Lakehouse Catalog: Beyond Iceberg Tables

Comments
10 min read
Real-time Data Analytics at Scale: Integrating Apache Flink and Apache Doris with Flink Doris Connector and Flink CDC

Real-time Data Analytics at Scale: Integrating Apache Flink and Apache Doris with Flink Doris Connector and Flink CDC

Comments
10 min read
Optimizing Kafka Performance: Best Practices for High Throughput and Low Latency

Optimizing Kafka Performance: Best Practices for High Throughput and Low Latency

Comments
7 min read
Fixing Type Hints for Callable Objects with Custom Signatures in Dagster

Fixing Type Hints for Callable Objects with Custom Signatures in Dagster

1
Comments
3 min read
Apache Spark সহজভাবে জানি

Apache Spark সহজভাবে জানি

1
Comments
1 min read
Building a Test Data Platform After Watching Teams Secretly Use Production for Years

Building a Test Data Platform After Watching Teams Secretly Use Production for Years

1
Comments
3 min read
Chinese DBA's Story: Sui Haifeng - Grasp the two most important five-year periods of your career

Chinese DBA's Story: Sui Haifeng - Grasp the two most important five-year periods of your career

Comments
5 min read
Kafka

Kafka

3
Comments
10 min read
A Modern Data Governance Framework for Google Cloud: Implementing Just-Enough and Just-in-Time Access

A Modern Data Governance Framework for Google Cloud: Implementing Just-Enough and Just-in-Time Access

3
Comments
8 min read
Scaling Customer Analytics: Designing ML Pipelines for Millions of Users

Scaling Customer Analytics: Designing ML Pipelines for Millions of Users

Comments
7 min read
Temperature, Tokens, and Context Windows: The Three Pillars of LLM Control

Temperature, Tokens, and Context Windows: The Three Pillars of LLM Control

2
Comments
13 min read
Building Intelligent, Metadata-Driven Pipelines with Azure Data Factory

Building Intelligent, Metadata-Driven Pipelines with Azure Data Factory

3
Comments
6 min read
Why Your Enterprise Data Platform Is No Longer Just for Analytics

Why Your Enterprise Data Platform Is No Longer Just for Analytics

2
Comments 1
11 min read
The State of Apache Iceberg, Polaris, and Arrow: October–November 2025

The State of Apache Iceberg, Polaris, and Arrow: October–November 2025

2
Comments
7 min read
Realtime Data Streaming Platform: Building a Unified Monitoring Stack

Realtime Data Streaming Platform: Building a Unified Monitoring Stack

4
Comments
8 min read
Real-Time Data Streaming Platform: From 140K to 1 Million Messages/Sec - A Flink Performance Tuning Journey

Real-Time Data Streaming Platform: From 140K to 1 Million Messages/Sec - A Flink Performance Tuning Journey

1
Comments
10 min read
Real-Time Streaming Platform with Pulsar, Flink & ClickHouse

Real-Time Streaming Platform with Pulsar, Flink & ClickHouse

4
Comments
6 min read
Why Parquet Is Everywhere - And What Makes It Actually Fast?

Why Parquet Is Everywhere - And What Makes It Actually Fast?

2
Comments
3 min read
loading...