DEV Community

# dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Building Bulletproof Data Pipelines: Orchestration, Testing, and Monitoring (Part 3 of 3)
karthikeyan_rajasekaran_c profile

Building Bulletproof Data Pipelines: Orchestration, Testing, and Monitoring (Part 3 of 3)

#dataengineering #dagster #dataquality #testing
1
Comments
9 min read
LogInSight: A Lightweight CloudWatch Log Analytics Tool for Faster Debugging and Real-Time Insights
sanjaydot profile

LogInSight: A Lightweight CloudWatch Log Analytics Tool for Faster Debugging and Real-Time Insights

#fastapi #python #aws #dataengineering
2
Comments
3 min read
The Proxy Economy: Residential, Datacenter, and ISP Rotation
deepak_mishra_35863517037 profile

The Proxy Economy: Residential, Datacenter, and ISP Rotation

#architecture #dataengineering #networking
Comments 3
5 min read
Automating Serverless Data Ingestion: How to Connect External APIs to BigQuery using Python and Cloud Functions
aminelaatfa profile

Automating Serverless Data Ingestion: How to Connect External APIs to BigQuery using Python and Cloud Functions

#googlecloud #bigquery #marketing #dataengineering
Comments
12 min read
The Database Query That Could Cost a Company Millions(And Why Data Engineers Exist)
thanh_truong_a99577c6b879 profile

The Database Query That Could Cost a Company Millions(And Why Data Engineers Exist)

#technology #dataengineering #techhistory
Comments
5 min read
Why Data SLAs Fail — and How to Enforce Them with a Unified Reliability Framework
baharath_bathula profile

Why Data SLAs Fail — and How to Enforce Them with a Unified Reliability Framework

#dataengineering #aws #machinelearning #analytics
Comments
2 min read
Building Streaming Iceberg Tables for Real-Time Logistics Analytics
elianalamhost profile

Building Streaming Iceberg Tables for Real-Time Logistics Analytics

#analytics #dataengineering #architecture #opensource
Comments
4 min read
The Great Table Format Debate: A Deep Dive into Apache Iceberg, Delta Lake, and Apache Hudi
datatechbridge profile

The Great Table Format Debate: A Deep Dive into Apache Iceberg, Delta Lake, and Apache Hudi

#database #dataengineering #iceberg #apachehudi
1
Comments
18 min read
Amazon Kinesis vs Amazon MSK: The Complete Guide for Stream Processing on AWS
datatechbridge profile

Amazon Kinesis vs Amazon MSK: The Complete Guide for Stream Processing on AWS

#architecture #aws #dataengineering
Comments
29 min read
Day 8: Accelerating Spark Joins - Broadcast, Shuffle Optimization & Skew Handling
sandeepk27 profile

Day 8: Accelerating Spark Joins - Broadcast, Shuffle Optimization & Skew Handling

#dataengineering #python #spark #bigdata
Comments
2 min read
2025 Year in Review: Apache Iceberg, Polaris, Parquet, and Arrow
alexmercedcoder profile

2025 Year in Review: Apache Iceberg, Polaris, Parquet, and Arrow

#architecture #bigdata #opensource #dataengineering
Comments
6 min read
Mastering Serverless Data Pipelines: AWS Step Functions Best Practices for 2026
jubinsoni profile

Mastering Serverless Data Pipelines: AWS Step Functions Best Practices for 2026

#aws #serverless #stepfunctions #dataengineering
Comments
5 min read
A Stranger In a New Town: CsvPath metadata fields
david_kershaw_b6916404da6 profile

A Stranger In a New Town: CsvPath metadata fields

#metadata #dataengineering #csv #datascience
Comments
6 min read
Unified Data Fabric: Serverless Spark on ROSA Integrating with AWS Glue Catalog
mgonzalezo profile

Unified Data Fabric: Serverless Spark on ROSA Integrating with AWS Glue Catalog

#serverless #kubernetes #aws #dataengineering
8
Comments 1
39 min read
Interesting links - November 2025
rmoff profile

Interesting links - November 2025

#data #dataengineering #kafka #flink
Comments
19 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.