DEV Community

#dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap
firfircelik profile

DuckDB 2.0 vs Polars vs Spark on One Machine: 150M Rows Under a 4.5 GiB Cap

#duckdb #dataengineering #python #spark
2
Comments 3
15 min read
How to Optimize Slow dbt Models: Incremental Models, DAGs, and Query Performance
harulmozhi profile

How to Optimize Slow dbt Models: Incremental Models, DAGs, and Query Performance

#dataanalytics #cloudcomputing #dataengineering #dataarchitecture
Comments
5 min read
Lakehouse vs Data Warehouse vs Data Lake: The Difference in One Picture
dino_david_c5d8f55f119b7f profile

Lakehouse vs Data Warehouse vs Data Lake: The Difference in One Picture

#dataengineering #databricks #deltalake #bigdata
Comments
2 min read
What AI Agents Taught Me About Building Databricks Apps
dcoferraz profile

What AI Agents Taught Me About Building Databricks Apps

#ai #databricks #dataengineering #softwareengineering
1
Comments 1
7 min read
Thought Experiment: If 100GB of structured/3D data could be losslessly compressed to 1GB, what would you build?
japanopendatalab profile

Thought Experiment: If 100GB of structured/3D data could be losslessly compressed to 1GB, what would you build?

#discuss #architecture #dataengineering #database
Comments
3 min read
Every Step Runs Inside poll(): The Kafka Consumer Flow, End to End
charanrachumalla profile

Every Step Runs Inside poll(): The Kafka Consumer Flow, End to End

#kafka #java #distributedsystems #dataengineering
Comments
4 min read
Apache Iceberg Maintenance Tools: From Spark Procedures to Autonomous Control Planes
jonisar profile

Apache Iceberg Maintenance Tools: From Spark Procedures to Autonomous Control Planes

#iceberg #dataengineering #lakehouse #opensource
Comments
16 min read
A test you can't run is a test you don't have
ashg2099 profile

A test you can't run is a test you don't have

#dataengineering #postgres #testing #python
Comments
4 min read
Replacing an AWS Glue ingestion job with DBMS_CLOUD_PIPELINE in Autonomous Database: lessons learned
ashish_sinha_5241c7673d93 profile

Replacing an AWS Glue ingestion job with DBMS_CLOUD_PIPELINE in Autonomous Database: lessons learned

#oci #aws #dataengineering #oracle
Comments
2 min read
Why Customer AI Needs an AI-Ready Customer Profile
gazerro profile

Why Customer AI Needs an AI-Ready Customer Profile

#ai #architecture #data #dataengineering
Comments
8 min read
When Zero-Copy and Mirroring Clash: An Iceberg Metadata Gotcha — Part 3
swaroop_krishna_e2f4b83b2 profile

When Zero-Copy and Mirroring Clash: An Iceberg Metadata Gotcha — Part 3

#dataengineering #snowflake #microsoftfabric #iceberg
1
Comments
3 min read
SQL Indexes Explained (For People Who Just Got Yelled At By a Slow Dashboard)
neha_christina_1ac8651819 profile

SQL Indexes Explained (For People Who Just Got Yelled At By a Slow Dashboard)

#sql #database #dataengineering #beginners
Comments 1
4 min read
Three Orders In, Three Rows Out, 42.9% More Revenue
jenniferevangelistade profile

Three Orders In, Three Rows Out, 42.9% More Revenue

#sql #dataengineering #databricks
Comments
5 min read
A Freshness Check Before the 8 a.m. Report
jenniferevangelistade profile

A Freshness Check Before the 8 a.m. Report

#apacheairflow #dataengineering
Comments
5 min read
From Messy CSV to Business Insight: A Small Python + SQL Analytics Pipeline
data_analyst_bunny profile

From Messy CSV to Business Insight: A Small Python + SQL Analytics Pipeline

#python #sql #dataengineering #beginners
Comments
4 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.