Dataengineering

👋 Sign in for the ability to sort posts by relevant, latest, or top.

DataDriven

Jun 16

I Built a Mini Message Broker in Pure Python and Finally Understood How Kafka Moves Millions of Events

#kafka #python #dataengineering #distributedsystems

6 min read

Felipe de Godoy

Jul 20

Pytest - Writing Cleaner Tests with Parametrization and Temporary Files

#dataengineering #python #testing #tutorial

4 min read

Jitendra Devabhaktuni

Jun 16

Creating HIPAA-Safe Synthetic Patient Data for Healthcare App Testing

#ai #datascience #database #dataengineering

4 min read

Kanishga Subramani

Jun 16

Day 19 of 100 Days of ClickHouse®: Managing Users and Roles with RBAC

#clickhouse #analytics #database #dataengineering

4 min read

Alex Merced

Jul 18

Apache Data Lakehouse Weekly: July 11 to July 18, 2026

#news #database #dataengineering #opensource

25 min read

Nate S

Jun 15

Building Confidence Scoring for Email Open Tracking (Engineering Notes)

#email #engineering #dataengineering #sales

2 min read

Bobby Ray

Jul 19

Building a Production Data Pipeline with Incremental Loading and dbt

#dataengineering #python #dbt #airflow

4 min read

Alisson Rosa

Jun 28

Leoflow v0.1.0 — run your Airflow DAGs on a Go control plane (no Airflow in the hot path)

#airflow #go #dataengineering #kubernetes

6 min read

Haji Rufai

Jun 15

I Built a B-Tree in Pure Python and Finally Understood Why Postgres Uses It for Every Index

#python #dataengineering #postgres #computerscience

6 min read

Alex Merced

Jul 18

Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust

#agents #ai #dataengineering #llm

19 min read

Petascale Labs

Jun 14

The Data Engineer Roadmap for 2026 (in an AI-Native World)

#dataengineering #career #roadmap #ai

7 min read

Patrick Lierse

Jun 14

Querying Germany's Company Register via API: Clean JSON and the new eGbR

#api #data #dataengineering #webscraping

1 min read

James Lee

Jun 18

Part 1 — How Do Unstructured Documents Become a Searchable Knowledge Base? Five Key Engineering Decisions in the Ingestion Pipeline

#ai #architecture #dataengineering #rag

9 min read

Kanishga Subramani

Jun 14

Day 17 of #100DaysOfClickHouse: Mastering Data Filtering for Faster ClickHouse Queries

#clickhouse #database #dataengineering #analytics

5 min read

👋 Sign in for the ability to sort posts by relevant, latest, or top.

DEV Community

#dataengineering

Top 12 Spark Interview Problems for Data Engineers, With Answers

I Built a Mini Message Broker in Pure Python and Finally Understood How Kafka Moves Millions of Events

Pytest - Writing Cleaner Tests with Parametrization and Temporary Files

Creating HIPAA-Safe Synthetic Patient Data for Healthcare App Testing

Day 19 of 100 Days of ClickHouse®: Managing Users and Roles with RBAC

Apache Data Lakehouse Weekly: July 11 to July 18, 2026

Building Confidence Scoring for Email Open Tracking (Engineering Notes)

Building a Production Data Pipeline with Incremental Loading and dbt

Leoflow v0.1.0 — run your Airflow DAGs on a Go control plane (no Airflow in the hot path)

I Built a B-Tree in Pure Python and Finally Understood Why Postgres Uses It for Every Index

Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust

The Data Engineer Roadmap for 2026 (in an AI-Native World)

Querying Germany's Company Register via API: Clean JSON and the new eGbR

Part 1 — How Do Unstructured Documents Become a Searchable Knowledge Base? Five Key Engineering Decisions in the Ingestion Pipeline

Day 17 of #100DaysOfClickHouse: Mastering Data Filtering for Faster ClickHouse Queries