DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Apache Gravitino 1.0.0 — From Metadata Management to Contextual Engineering

Apache Gravitino 1.0.0 — From Metadata Management to Contextual Engineering

1
Comments
7 min read
Chinese DBA's Story: Hu Zhonghao - The Journey of Becoming a DBA for Domestic Distributed Databases

Chinese DBA's Story: Hu Zhonghao - The Journey of Becoming a DBA for Domestic Distributed Databases

Comments 1
7 min read
Apache Kafka in Data engineering

Apache Kafka in Data engineering

6
Comments 1
1 min read
🧭System Design Roadmap for Data Engineers

🧭System Design Roadmap for Data Engineers

4
Comments
3 min read
Orchestrating and Observing Data Pipelines with Airflow, PostgreSQL, and Polar

Orchestrating and Observing Data Pipelines with Airflow, PostgreSQL, and Polar

2
Comments
3 min read
💥 Polars vs. Pandas: Why Your Next ETL Pipeline Should Run on Rust (Part 1/5)

💥 Polars vs. Pandas: Why Your Next ETL Pipeline Should Run on Rust (Part 1/5)

1
Comments
2 min read
(Ⅱ) A Complete Guide to Core Data Warehouse Design Standards: From Layers, Types to Lifecycle

(Ⅱ) A Complete Guide to Core Data Warehouse Design Standards: From Layers, Types to Lifecycle

Comments
6 min read
Building Distributed Systems with Ray—Just Like Running a Restaurant

Building Distributed Systems with Ray—Just Like Running a Restaurant

1
Comments
7 min read
The State of Apache Iceberg v4 - October 2025 Edition

The State of Apache Iceberg v4 - October 2025 Edition

3
Comments
6 min read
Data Automation: A Deep Dive

Data Automation: A Deep Dive

1
Comments
5 min read
TikTok Data Engineer Full 3-Round Interview

TikTok Data Engineer Full 3-Round Interview

2
Comments
4 min read
Why Data Partitioning Is Harder Than It Looks

Why Data Partitioning Is Harder Than It Looks

1
Comments
2 min read
Part 2: Snowflake's Autonomous Future

Part 2: Snowflake's Autonomous Future

Comments
8 min read
Collecting Africa’s Energy Insights:

Collecting Africa’s Energy Insights:

3
Comments
4 min read
Making JSON Compression Searchable — SEE (Schema-Aware Encoding)

Making JSON Compression Searchable — SEE (Schema-Aware Encoding)

1
Comments
2 min read
Containerization for Data Engineering: A Practical Guide with Docker and Docker Compose

Containerization for Data Engineering: A Practical Guide with Docker and Docker Compose

Comments
5 min read
Apache Iceberg Dev List Digest (Sept 15–19, 2025)

Apache Iceberg Dev List Digest (Sept 15–19, 2025)

Comments
3 min read
Data Engineering with Docker: A Hands-On Guide to Containerization

Data Engineering with Docker: A Hands-On Guide to Containerization

7
Comments 2
3 min read
From Kafka to Clean Tables: Building a Confluent Snowflake Pipeline with Streams & Tasks

From Kafka to Clean Tables: Building a Confluent Snowflake Pipeline with Streams & Tasks

3
Comments 1
10 min read
Understanding the Basics of Linux Operating System

Understanding the Basics of Linux Operating System

Comments
1 min read
Why you need to learn Apache Airflow - right now

Why you need to learn Apache Airflow - right now

Comments
3 min read
Building a True Dual-Destination Analytics Pipeline: Real-Time Streaming with S3 Backup and Recovery

Building a True Dual-Destination Analytics Pipeline: Real-Time Streaming with S3 Backup and Recovery

1
Comments
8 min read
Apache Kafka Deep Dive: Concepts, Applications, and Production

Apache Kafka Deep Dive: Concepts, Applications, and Production

Comments
4 min read
Automating NASA’s Astronomy Picture of the Day with Airflow

Automating NASA’s Astronomy Picture of the Day with Airflow

Comments
6 min read
Building Modern Data Systems: Event-Driven Architecture, Messaging Queues, Batch Processing, ETL & ELT

Building Modern Data Systems: Event-Driven Architecture, Messaging Queues, Batch Processing, ETL & ELT

2
Comments
11 min read
loading...