DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
📘 Foundation Phase Completed - Starting Phase 2 of My Journey

📘 Foundation Phase Completed - Starting Phase 2 of My Journey

Comments
3 min read
Building an Automated YouTube Analytics Dashboard with Airflow, PySpark, MinIO, PostgreSQL & Grafana

Building an Automated YouTube Analytics Dashboard with Airflow, PySpark, MinIO, PostgreSQL & Grafana

6
Comments
5 min read
A Beginner’s Guide to Big Data Analytics with Apache Spark and PySpark

A Beginner’s Guide to Big Data Analytics with Apache Spark and PySpark

Comments
4 min read
Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Comments
4 min read
Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Comments
7 min read
Self-Adapting Data Pipelines: The Intelligent Future of Data Engineering

Self-Adapting Data Pipelines: The Intelligent Future of Data Engineering

5
Comments
17 min read
10 Best Platforms to Learn Data Engineering in 2026

10 Best Platforms to Learn Data Engineering in 2026

Comments
4 min read
Building a Task Manager with Apache NiFi: From Custom Scheduler to Distributed Workflows

Building a Task Manager with Apache NiFi: From Custom Scheduler to Distributed Workflows

4
Comments
9 min read
Azure Data Factory (ADF) - A Beginner's Guide to Cloud Data Integration

Azure Data Factory (ADF) - A Beginner's Guide to Cloud Data Integration

2
Comments
4 min read
Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Apache Kafka Deep Dive: Core Concepts, Data Engineering Applications, and Real-World Production Practices

Comments
4 min read
Building a Reddit Sentiment Pipeline using Python, PostgreSQL, VADER, Airflow, Grafana, Prometheus and StatsD

Building a Reddit Sentiment Pipeline using Python, PostgreSQL, VADER, Airflow, Grafana, Prometheus and StatsD

1
Comments 1
5 min read
“𝗘𝗧𝗟 𝗶𝘀 𝗘𝘃𝗼𝗹𝘃𝗶𝗻𝗴 — 𝗔𝗿𝗲 𝗬𝗼𝘂?”

“𝗘𝗧𝗟 𝗶𝘀 𝗘𝘃𝗼𝗹𝘃𝗶𝗻𝗴 — 𝗔𝗿𝗲 𝗬𝗼𝘂?”

5
Comments
1 min read
From OLTP to OLAP: Streaming Databases into MotherDuck with Estuary

From OLTP to OLAP: Streaming Databases into MotherDuck with Estuary

1
Comments
7 min read
Complete Guide: Dockerizing Spark, Kafka, and Jupyter for YouTube Pipeline

Complete Guide: Dockerizing Spark, Kafka, and Jupyter for YouTube Pipeline

Comments
9 min read
Apache Kafka & Amazon MSK: The Beating Heart of Real-Time Data

Apache Kafka & Amazon MSK: The Beating Heart of Real-Time Data

1
Comments
4 min read
Migrating Oracle Fusion Cloud Data to Azure Fabric: A Practical Guide

Migrating Oracle Fusion Cloud Data to Azure Fabric: A Practical Guide

Comments
3 min read
Migrating Oracle Fusion Cloud Data to Azure Fabric: A Practical Guide

Migrating Oracle Fusion Cloud Data to Azure Fabric: A Practical Guide

Comments
3 min read
Handling Distributed Transactions with Orchestrator Pattern (Withdrawal & Deposit Example)

Handling Distributed Transactions with Orchestrator Pattern (Withdrawal & Deposit Example)

1
Comments
2 min read
Streaming Data Using Apache Kafka

Streaming Data Using Apache Kafka

1
Comments
2 min read
SUPCON Uses SeaTunnel to Build an Efficient Data Collection Framework, Achieving 0 Failures in Core Data Synchronization Tasks!

SUPCON Uses SeaTunnel to Build an Efficient Data Collection Framework, Achieving 0 Failures in Core Data Synchronization Tasks!

Comments
14 min read
Building a Streaming Data Pipeline with Kafka and Spark: Real-Time Analytics Implementation Guide

Building a Streaming Data Pipeline with Kafka and Spark: Real-Time Analytics Implementation Guide

1
Comments
10 min read
🚀 How I Learned That Thinking “Small” Can Save Hours of SQL Headaches

🚀 How I Learned That Thinking “Small” Can Save Hours of SQL Headaches

1
Comments
2 min read
GETTING TO KNOW:STAR AND SNOWFLAKE SCHEMA.

GETTING TO KNOW:STAR AND SNOWFLAKE SCHEMA.

1
Comments
3 min read
Hackeando o Data Engineering: Os Padrões que Todo Engenheiro Precisa Conhecer

Hackeando o Data Engineering: Os Padrões que Todo Engenheiro Precisa Conhecer

Comments
1 min read
Apache Kafka in Data Engineering

Apache Kafka in Data Engineering

Comments
1 min read
loading...