DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
The Bear Awakens: From Pure Speed to Massive Endurance (640 Million Rows Tested)

The Bear Awakens: From Pure Speed to Massive Endurance (640 Million Rows Tested)

Comments
16 min read
Bulletproof Power Query (Part 2): A Smart, Fuzzy-Match Rename Function

Bulletproof Power Query (Part 2): A Smart, Fuzzy-Match Rename Function

Comments
4 min read
Part 1: Database Concepts & Architecture

Part 1: Database Concepts & Architecture

Comments
14 min read
New release: LightningChart Python 2.1

New release: LightningChart Python 2.1

Comments
1 min read
The Time Our Pipeline Processed the Same Day’s Data 47 Times

The Time Our Pipeline Processed the Same Day’s Data 47 Times

Comments
5 min read
Firehose and Iceberg Tables

Firehose and Iceberg Tables

Comments
4 min read
I Built an ETL Pipeline That Actually Thinks & And Cut Token Costs by 52% (And Here's What I Learned)

I Built an ETL Pipeline That Actually Thinks & And Cut Token Costs by 52% (And Here's What I Learned)

1
Comments
17 min read
Beyond SQL: Solving Data Warehouse Performance Bottlenecks with Smart Algorithms, Not Just Bigger Clusters

Beyond SQL: Solving Data Warehouse Performance Bottlenecks with Smart Algorithms, Not Just Bigger Clusters

5
Comments
13 min read
Day 16: Delta Lake Explained - How Spark Finally Became Reliable for Production ETL

Day 16: Delta Lake Explained - How Spark Finally Became Reliable for Production ETL

Comments
2 min read
Migrate the legacy Greenplum to Apache Cloudberry with cbcopy

Migrate the legacy Greenplum to Apache Cloudberry with cbcopy

Comments
7 min read
Apache Dev List Digest: Iceberg, Polaris, Arrow & Parquet (Dec 9th - Dec15th, 2025)

Apache Dev List Digest: Iceberg, Polaris, Arrow & Parquet (Dec 9th - Dec15th, 2025)

Comments
7 min read
Building a dbt-UI I Wish Existed

Building a dbt-UI I Wish Existed

Comments
3 min read
Unpacking the Google File System Paper: A Simple Breakdown

Unpacking the Google File System Paper: A Simple Breakdown

Comments
3 min read
Day 15: Running Spark in the Cloud - Dataproc vs Databricks

Day 15: Running Spark in the Cloud - Dataproc vs Databricks

Comments
2 min read
The Myth of Distributed Computing as a Silver Bullet for Big Data

The Myth of Distributed Computing as a Silver Bullet for Big Data

5
Comments
10 min read
Rethinking Stream-Batch Unification: Real-Time Processing with Incremental Materialized Views in Apache Cloudberry

Rethinking Stream-Batch Unification: Real-Time Processing with Incremental Materialized Views in Apache Cloudberry

Comments
5 min read
Interesting links - December 2025

Interesting links - December 2025

Comments
13 min read
Data Engineering Processes: From Raw Data to Cleaned, Processed, Analytics-Ready Data.

Data Engineering Processes: From Raw Data to Cleaned, Processed, Analytics-Ready Data.

Comments
5 min read
Navigating the Future: Top Data Engineering Trends Shaping 2024 and Beyond

Navigating the Future: Top Data Engineering Trends Shaping 2024 and Beyond

Comments
4 min read
Day 14: Building a Real Retail Analytics Pipeline Using Spark Window Functions

Day 14: Building a Real Retail Analytics Pipeline Using Spark Window Functions

Comments
1 min read
Day 13: Window Functions in PySpark

Day 13: Window Functions in PySpark

Comments
2 min read
Why Idempotency Is So Important in Data Engineering

Why Idempotency Is So Important in Data Engineering

Comments
6 min read
REST API Calls for Data Engineers: A Practical Guide with Examples

REST API Calls for Data Engineers: A Practical Guide with Examples

Comments
3 min read
Is CsvPath an easy or hard language?

Is CsvPath an easy or hard language?

Comments
16 min read
Day 17: Building a Real ETL Pipeline in Spark Using Bronze-Silver-Gold Architecture

Day 17: Building a Real ETL Pipeline in Spark Using Bronze-Silver-Gold Architecture

Comments
1 min read
loading...