DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Interesting links - November 2025

Interesting links - November 2025

Comments
19 min read
dupl

dupl

Comments
1 min read
Apache Dev List Digest: Iceberg, Polaris, Arrow & Parquet (Nov 18–24, 2025)

Apache Dev List Digest: Iceberg, Polaris, Arrow & Parquet (Nov 18–24, 2025)

Comments
5 min read
How to Sync Data from an Oracle Table to Elasticsearch using Kafka Connect

How to Sync Data from an Oracle Table to Elasticsearch using Kafka Connect

1
Comments 1
5 min read
Agent Cost Optimization: A Data Engineer's Guide

Agent Cost Optimization: A Data Engineer's Guide

Comments
13 min read
From Raw to Refined: Data Pipeline Architecture at Scale

From Raw to Refined: Data Pipeline Architecture at Scale

Comments
12 min read
INTRODUCTION TO DBT(Data Build Tool)

INTRODUCTION TO DBT(Data Build Tool)

1
Comments
2 min read
Breaking Into Gaming Analytics: From 1 Billion Mobile Users to 5B Daily Events

Breaking Into Gaming Analytics: From 1 Billion Mobile Users to 5B Daily Events

Comments 1
6 min read
Entrenando a Prize (Parte 2): Creando un Dataset de 5,500 líneas por $0.08 USD

Entrenando a Prize (Parte 2): Creando un Dataset de 5,500 líneas por $0.08 USD

Comments
5 min read
Taming the Data Beast: Build Pipelines That Bend, Not Break by Arvind Sundararajan

Taming the Data Beast: Build Pipelines That Bend, Not Break by Arvind Sundararajan

Comments
2 min read
Azure Data Solutions: Data Factory, Synapse, Data Lake & Databricks Integration

Azure Data Solutions: Data Factory, Synapse, Data Lake & Databricks Integration

1
Comments
4 min read
Data Quality at Scale: Why Your Pipeline Needs More Than Green Checkmarks

Data Quality at Scale: Why Your Pipeline Needs More Than Green Checkmarks

Comments
8 min read
Prompt Engineering Patterns: From Zero-Shot to Chain-of-Thought Reasoning

Prompt Engineering Patterns: From Zero-Shot to Chain-of-Thought Reasoning

1
Comments
14 min read
Why We Need Schema Registry in Kafka

Why We Need Schema Registry in Kafka

2
Comments
17 min read
Introduction to the Confluent REST Proxy

Introduction to the Confluent REST Proxy

2
Comments
4 min read
While We're Measuring Developer Productivity, Won't Someone Think of the Data Engineers?

While We're Measuring Developer Productivity, Won't Someone Think of the Data Engineers?

Comments
9 min read
Analyzing and Optimizing a Parquet ClickHouse Ingestion Pipeline

Analyzing and Optimizing a Parquet ClickHouse Ingestion Pipeline

2
Comments 2
3 min read
Azure Synapse Analytics

Azure Synapse Analytics

Comments
5 min read
Debugging Windows Race Conditions in Dagster

Debugging Windows Race Conditions in Dagster

Comments
3 min read
Design Patterns for Data Engineers: Cleaner ETL with the Builder Pattern.

Design Patterns for Data Engineers: Cleaner ETL with the Builder Pattern.

3
Comments
2 min read
Positional Encodings and Context Window Engineering: Why Token Order Matters

Positional Encodings and Context Window Engineering: Why Token Order Matters

3
Comments
12 min read
Trying Out Dagster for Data Orchestration

Trying Out Dagster for Data Orchestration

4
Comments
9 min read
🔐 Understanding Governance in Microsoft Fabric

🔐 Understanding Governance in Microsoft Fabric

1
Comments
3 min read
 Day 2: Data Engineering vs Data Science vs Data Analytics

 Day 2: Data Engineering vs Data Science vs Data Analytics

Comments
2 min read
6 Different Data Formats Commonly Used in Data Analytics

6 Different Data Formats Commonly Used in Data Analytics

Comments
3 min read
loading...