Dataengineering

👋 Sign in for the ability to sort posts by relevant, latest, or top.

Nithyalakshmi Kamalakkannan

Jan 2

End-to-End Real-Time Data Engineering on Databricks Using Spark Structured Streaming and Delta Lake

#dataengineering #handson #realtimeproject #spark

1 min read

Karthikeyan Rajasekaran

Jan 2

Building Bulletproof Data Pipelines: Orchestration, Testing, and Monitoring (Part 3 of 3)

#dataengineering #dagster #dataquality #testing

9 min read

Sanjay_Balaji

Nov 29 '25

LogInSight: A Lightweight CloudWatch Log Analytics Tool for Faster Debugging and Real-Time Insights

#fastapi #python #aws #dataengineering

3 min read

Lalit Mishra

Jan 1

The Proxy Economy: Residential, Datacenter, and ISP Rotation

#architecture #dataengineering #networking

5 min read

Thanh Truong

Jan 1

The Database Query That Could Cost a Company Millions(And Why Data Engineers Exist)

#technology #dataengineering #techhistory

5 min read

Vinicius Fagundes

Dec 21 '25

RAG Evaluation Metrics: Measuring What Actually Matters

#llm #rag #dataengineering #ai

10 min read

Baharath Bathula

Jan 1

Why Data SLAs Fail — and How to Enforce Them with a Unified Reliability Framework

#dataengineering #aws #machinelearning #analytics

2 min read

Gabriel

Jan 1

When code-gen suggests deprecated Pandas APIs — a subtle drift that broke a pipeline

#dataengineering #devops #python #ai

3 min read

Eliana Lam

Nov 29 '25

Building Streaming Iceberg Tables for Real-Time Logistics Analytics

#analytics #dataengineering #architecture #opensource

4 min read

Amos Augo

Nov 26 '25

Building a Scalable Community Health Worker Analytics Platform: My Journey with dbt and Snowflake

#dbt #snowflake #dataengineering

4 min read

Gabriel

Dec 31 '25

When an AI Suggests DataFrame.append: Missing Pandas Deprecations in Generated Code

#codequality #dataengineering #llm #python

3 min read

Data Tech Bridge

Dec 31 '25

The Great Table Format Debate: A Deep Dive into Apache Iceberg, Delta Lake, and Apache Hudi

#database #dataengineering #iceberg #apachehudi

18 min read

Data Tech Bridge

Dec 30 '25

Amazon Kinesis vs Amazon MSK: The Complete Guide for Stream Processing on AWS

#architecture #aws #dataengineering

29 min read

Sandeep

Dec 9 '25

Day 8: Accelerating Spark Joins - Broadcast, Shuffle Optimization & Skew Handling

#dataengineering #python #spark #bigdata

2 min read

Jubin Soni

Dec 30 '25

Mastering Serverless Data Pipelines: AWS Step Functions Best Practices for 2026

#aws #serverless #stepfunctions #dataengineering

5 min read

👋 Sign in for the ability to sort posts by relevant, latest, or top.

DEV Community

# dataengineering

End-to-End Real-Time Data Engineering on Databricks Using Spark Structured Streaming and Delta Lake

Building Bulletproof Data Pipelines: Orchestration, Testing, and Monitoring (Part 3 of 3)

LogInSight: A Lightweight CloudWatch Log Analytics Tool for Faster Debugging and Real-Time Insights

The Proxy Economy: Residential, Datacenter, and ISP Rotation

The Database Query That Could Cost a Company Millions(And Why Data Engineers Exist)

RAG Evaluation Metrics: Measuring What Actually Matters

Why Data SLAs Fail — and How to Enforce Them with a Unified Reliability Framework

When code-gen suggests deprecated Pandas APIs — a subtle drift that broke a pipeline

Building Streaming Iceberg Tables for Real-Time Logistics Analytics

Building a Scalable Community Health Worker Analytics Platform: My Journey with dbt and Snowflake

When an AI Suggests DataFrame.append: Missing Pandas Deprecations in Generated Code

The Great Table Format Debate: A Deep Dive into Apache Iceberg, Delta Lake, and Apache Hudi

Amazon Kinesis vs Amazon MSK: The Complete Guide for Stream Processing on AWS

Day 8: Accelerating Spark Joins - Broadcast, Shuffle Optimization & Skew Handling

Mastering Serverless Data Pipelines: AWS Step Functions Best Practices for 2026