DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Simulating An Event-Driven Python Shopping App with Kafka on AWS For Real-Time Processing.

Simulating An Event-Driven Python Shopping App with Kafka on AWS For Real-Time Processing.

5
Comments 2
8 min read
Real-Time Data Streaming Platform: How We Built a Self-Hosted Platform with 90% Cost Reduction vs AWS Managed Services

Real-Time Data Streaming Platform: How We Built a Self-Hosted Platform with 90% Cost Reduction vs AWS Managed Services

1
Comments
6 min read
Building a Real-Time Data Platform with Kubernetes (Kind) - A Complete Local Setup Guide

Building a Real-Time Data Platform with Kubernetes (Kind) - A Complete Local Setup Guide

2
Comments
8 min read
Building a Task Manager with Apache NiFi: From Custom Scheduler to Distributed Workflows

Building a Task Manager with Apache NiFi: From Custom Scheduler to Distributed Workflows

Comments
9 min read
Azure Data Factory (ADF) - A Beginner's Guide to Cloud Data Integration

Azure Data Factory (ADF) - A Beginner's Guide to Cloud Data Integration

2
Comments
4 min read
Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Comments
4 min read
Another Data Nerd Guide to re:Invent 2025

Another Data Nerd Guide to re:Invent 2025

2
Comments
3 min read
“𝗘𝗧𝗟 𝗶𝘀 𝗘𝘃𝗼𝗹𝘃𝗶𝗻𝗴 — 𝗔𝗿𝗲 𝗬𝗼𝘂?”

“𝗘𝗧𝗟 𝗶𝘀 𝗘𝘃𝗼𝗹𝘃𝗶𝗻𝗴 — 𝗔𝗿𝗲 𝗬𝗼𝘂?”

5
Comments
1 min read
From OLTP to OLAP: Streaming Databases into MotherDuck with Estuary

From OLTP to OLAP: Streaming Databases into MotherDuck with Estuary

1
Comments
7 min read
Apache Kafka & Amazon MSK: The Beating Heart of Real-Time Data

Apache Kafka & Amazon MSK: The Beating Heart of Real-Time Data

1
Comments
4 min read
Migrating Oracle Fusion Cloud Data to Azure Fabric: A Practical Guide

Migrating Oracle Fusion Cloud Data to Azure Fabric: A Practical Guide

Comments
3 min read
Migrating Oracle Fusion Cloud Data to Azure Fabric: A Practical Guide

Migrating Oracle Fusion Cloud Data to Azure Fabric: A Practical Guide

Comments
3 min read
Handling Distributed Transactions with Orchestrator Pattern (Withdrawal & Deposit Example)

Handling Distributed Transactions with Orchestrator Pattern (Withdrawal & Deposit Example)

Comments
2 min read
Streaming Data Using Apache Kafka

Streaming Data Using Apache Kafka

1
Comments
2 min read
SUPCON Uses SeaTunnel to Build an Efficient Data Collection Framework, Achieving 0 Failures in Core Data Synchronization Tasks!

SUPCON Uses SeaTunnel to Build an Efficient Data Collection Framework, Achieving 0 Failures in Core Data Synchronization Tasks!

Comments
14 min read
Apache Doris 4.0: One Engine for Analytics, Full-Text Search, and Vector Search

Apache Doris 4.0: One Engine for Analytics, Full-Text Search, and Vector Search

5
Comments
7 min read
Building a Streaming Data Pipeline with Kafka and Spark: Real-Time Analytics Implementation Guide

Building a Streaming Data Pipeline with Kafka and Spark: Real-Time Analytics Implementation Guide

1
Comments
10 min read
🚀 How I Learned That Thinking “Small” Can Save Hours of SQL Headaches

🚀 How I Learned That Thinking “Small” Can Save Hours of SQL Headaches

1
Comments
2 min read
Elusion v8.0.0 is the best END-TO-END Data Engineering library writen in RUST

Elusion v8.0.0 is the best END-TO-END Data Engineering library writen in RUST

Comments
2 min read
Hands-On ACID in PostgreSQL : Part-1

Hands-On ACID in PostgreSQL : Part-1

1
Comments
3 min read
GETTING TO KNOW:STAR AND SNOWFLAKE SCHEMA.

GETTING TO KNOW:STAR AND SNOWFLAKE SCHEMA.

1
Comments
3 min read
Hackeando o Data Engineering: Os Padrões que Todo Engenheiro Precisa Conhecer

Hackeando o Data Engineering: Os Padrões que Todo Engenheiro Precisa Conhecer

Comments
1 min read
YouTube Analytics Pipeline Using Delta Lake and PySpark.

YouTube Analytics Pipeline Using Delta Lake and PySpark.

Comments
8 min read
Apache Kafka in Data Engineering

Apache Kafka in Data Engineering

Comments
1 min read
Data Engineering 102: Understanding Transactions, ACID, and Isolation in PostgreSQL

Data Engineering 102: Understanding Transactions, ACID, and Isolation in PostgreSQL

3
Comments
5 min read
loading...