DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Day 23: Spark Shuffle Optimization

Day 23: Spark Shuffle Optimization

Comments
1 min read
Day 22: Spark Shuffle Deep Dive

Day 22: Spark Shuffle Deep Dive

Comments
1 min read
Day 20: Handling Bad Records & Data Quality in Spark

Day 20: Handling Bad Records & Data Quality in Spark

Comments
1 min read
Data-Architect-Master-Professional-Workbook

Data-Architect-Master-Professional-Workbook

Comments
1 min read
Day 18: Spark Performance Tuning

Day 18: Spark Performance Tuning

Comments
1 min read
Day 19: Spark Broadcasting & Caching

Day 19: Spark Broadcasting & Caching

Comments
1 min read
Designing a YouTube Digest for Signal Over Noise

Designing a YouTube Digest for Signal Over Noise

Comments
4 min read
Day 21: Building a Production-Grade Data Quality Pipeline with Spark & Delta

Day 21: Building a Production-Grade Data Quality Pipeline with Spark & Delta

Comments
1 min read
dbt & Airflow in 2025: Why These Data Powerhouses Are Redefining Engineering

dbt & Airflow in 2025: Why These Data Powerhouses Are Redefining Engineering

Comments
11 min read
Why Most MIS Reporting Systems Break Before Data Processing Starts

Why Most MIS Reporting Systems Break Before Data Processing Starts

Comments
1 min read
The Missing Step in RAG: Why Your Vector DB is Bloated (and how to fix it locally)

The Missing Step in RAG: Why Your Vector DB is Bloated (and how to fix it locally)

1
Comments
3 min read
The Proxy Economy: Residential, Datacenter, and ISP Rotation

The Proxy Economy: Residential, Datacenter, and ISP Rotation

Comments 2
5 min read
Building a CDC Skyscraper: How SeaTunnel Leverages Debezium Under the Hood

Building a CDC Skyscraper: How SeaTunnel Leverages Debezium Under the Hood

Comments
3 min read
Amazon S3 Tables Just Got Smarter: Intelligent-Tiering & Native Replication Explained

Amazon S3 Tables Just Got Smarter: Intelligent-Tiering & Native Replication Explained

Comments
4 min read
Bulletproof Power Query (Part 2): A Smart, Fuzzy-Match Rename Function

Bulletproof Power Query (Part 2): A Smart, Fuzzy-Match Rename Function

Comments
4 min read
System Architecture Analysis: The Data Pipeline Issues of TraderKnows

System Architecture Analysis: The Data Pipeline Issues of TraderKnows

Comments
2 min read
Building an AI-Powered Customer Churn Prediction Pipeline on AWS (Step-by-Step)

Building an AI-Powered Customer Churn Prediction Pipeline on AWS (Step-by-Step)

2
Comments
5 min read
Beyond Tagging: A Blueprint for Real-Time Cost Attribution in Data Platforms

Beyond Tagging: A Blueprint for Real-Time Cost Attribution in Data Platforms

Comments
9 min read
New release: LightningChart Python 2.1

New release: LightningChart Python 2.1

Comments
1 min read
Why Your Model is Failing (Hint: It’s Not the Architecture)

Why Your Model is Failing (Hint: It’s Not the Architecture)

Comments
4 min read
Architecting for the Crash: Why 'Clean Data' is the Only Safety Net in Trading Wind-Down (TWD)

Architecting for the Crash: Why 'Clean Data' is the Only Safety Net in Trading Wind-Down (TWD)

1
Comments
3 min read
How One Can Start Their Journey in Data Engineering

How One Can Start Their Journey in Data Engineering

Comments 2
4 min read
The Time Our Pipeline Processed the Same Day’s Data 47 Times

The Time Our Pipeline Processed the Same Day’s Data 47 Times

Comments
5 min read
Firehose and Iceberg Tables

Firehose and Iceberg Tables

Comments
4 min read
I Built an ETL Pipeline That Actually Thinks & And Cut Token Costs by 52% (And Here's What I Learned)

I Built an ETL Pipeline That Actually Thinks & And Cut Token Costs by 52% (And Here's What I Learned)

1
Comments
17 min read
loading...