DEV Community

# dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Designing a Cost-Efficient Parallel Data Pipeline on AWS Using Lambda and SQS
datapenguin profile

Designing a Cost-Efficient Parallel Data Pipeline on AWS Using Lambda and SQS

#aws #dataengineering #lambda #sqs
3
Comments
6 min read
The Offline Data Engineer: Building Resilient API Pipelines that Work on an Airplane
markolekic profile

The Offline Data Engineer: Building Resilient API Pipelines that Work on an Airplane

#python #dataengineering #api #sql
4
Comments
5 min read
Understanding Kafka Architecture, Schema Registry, ksqlDB, PostgreSQL, Couchbase, and Microservices
jumptotech profile

Understanding Kafka Architecture, Schema Registry, ksqlDB, PostgreSQL, Couchbase, and Microservices

#architecture #dataengineering #microservices #beginners
2
Comments
3 min read
Building a 75,000-Product Image Feature Dataset for the Amazon ML Challenge 2025
bolewara profile

Building a 75,000-Product Image Feature Dataset for the Amazon ML Challenge 2025

#machinelearning #deeplearning #dataengineering #pca
1
Comments
4 min read
An Exploration of the Commercial Iceberg Catalog Ecosystem
alexmercedcoder profile

An Exploration of the Commercial Iceberg Catalog Ecosystem

#dataengineering
Comments
14 min read
🧠 ClickHouse LEFT JOINs: Why join_use_nulls Matters
ranjbaryshahab profile

🧠 ClickHouse LEFT JOINs: Why join_use_nulls Matters

#clickhouse #sql #dataengineering #join
5
Comments
2 min read
Getting Started Building a Data Platform
jorinvo profile

Getting Started Building a Data Platform

#dataengineering #database #sql
Comments
3 min read
Building a Universal Lakehouse Catalog: Beyond Iceberg Tables
alexmercedcoder profile

Building a Universal Lakehouse Catalog: Beyond Iceberg Tables

#bigdata #dataengineering #architecture #database
Comments
10 min read
Real-time Data Analytics at Scale: Integrating Apache Flink and Apache Doris with Flink Doris Connector and Flink CDC
apachedoris profile

Real-time Data Analytics at Scale: Integrating Apache Flink and Apache Doris with Flink Doris Connector and Flink CDC

#olap #apacheflink #dataengineering #database
Comments
10 min read
Optimizing Kafka Performance: Best Practices for High Throughput and Low Latency
zoolatech profile

Optimizing Kafka Performance: Best Practices for High Throughput and Low Latency

#architecture #dataengineering #performance
Comments
7 min read
Fixing Type Hints for Callable Objects with Custom Signatures in Dagster
jongwan93 profile

Fixing Type Hints for Callable Objects with Custom Signatures in Dagster

#machinelearning #python #dataengineering #opensource
1
Comments
3 min read
Apache Spark সহজভাবে জানি
sajjadrahman56 profile

Apache Spark সহজভাবে জানি

#spark #dataengineering #bigdata #tutorial
1
Comments
1 min read
Building a Test Data Platform After Watching Teams Secretly Use Production for Years
alex_hayward_ff2d88ff331e profile

Building a Test Data Platform After Watching Teams Secretly Use Production for Years

#devops #testing #cicd #dataengineering
1
Comments
3 min read
Kafka
jumptotech profile

Kafka

#dataengineering #beginners #opensource #architecture
3
Comments
10 min read
Chinese DBA's Story: Sui Haifeng - Grasp the two most important five-year periods of your career
tripcat profile

Chinese DBA's Story: Sui Haifeng - Grasp the two most important five-year periods of your career

#database #dataengineering #data
Comments
5 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.