DEV Community

#dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
I built a mini columnar storage engine in pure Python — and finally understood why Parquet is so fast
hajirufai profile

I built a mini columnar storage engine in pure Python — and finally understood why Parquet is so fast

#dataengineering #python #database #performance
Comments
6 min read
Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables
datapenguin profile

Hitting the Iceberg REST Catalog Directly: Understanding the Differences Between Glue Data Catalog and S3 Tables

#aws #iceberg #dataengineering
9
Comments 2
17 min read
Why I Don’t Let the LLM Decide Issue State
ahmadhumayun profile

Why I Don’t Let the LLM Decide Issue State

#ai #python #dataengineering #marketinganalytics
Comments
5 min read
The customer_id that isn't a customer
gaur_data_c6511af396c4bce profile

The customer_id that isn't a customer

#agents #analytics #dataengineering #sql
Comments
8 min read
Your Scraper Collected 50 Rows. There Were 4,000.
0012303 profile

Your Scraper Collected 50 Rows. There Were 4,000.

#webscraping #python #dataengineering #pagination
Comments
7 min read
Deeper into Dataform 3: Auditing Dataform
ben_watson_data profile

Deeper into Dataform 3: Auditing Dataform

#dataform #dataengineering #gcp #bigquery
Comments
2 min read
How I Broke Down My ETL Pipeline Project Into Smaller Engineering Exercises
tanmay_bhurkunde profile
Tanmay

How I Broke Down My ETL Pipeline Project Into Smaller Engineering Exercises

#dataengineering #database #etl #sql
Comments
2 min read
Apache Data Lakehouse Weekly: July 1 to July 8, 2026
alexmercedcoder profile

Apache Data Lakehouse Weekly: July 1 to July 8, 2026

#news #data #dataengineering #opensource
2
Comments 1
24 min read
I built a data-contract validator in pure Python (no pandas, no PyYAML) and it caught a 30% revenue ghost
hajirufai profile

I built a data-contract validator in pure Python (no pandas, no PyYAML) and it caught a 30% revenue ghost

#python #dataengineering #datascience #opensource
Comments
6 min read
A practical pipeline for turning messy business documents into spreadsheets
alex_zheng_49d3089c0d3fdf profile

A practical pipeline for turning messy business documents into spreadsheets

#dataengineering #ai #productivity
Comments
2 min read
How Delta Lake Brings ACID to a Data Lake
roni_das_b1b76c5ee6583027 profile

How Delta Lake Brings ACID to a Data Lake

#systemdesign #dataengineering #bigdata #database
1
Comments 1
3 min read
A Fair Coin Isn't Enough: When a Perfectly Randomized Experiment Is Impossible to Analyze
hexisteme profile
John

A Fair Coin Isn't Enough: When a Perfectly Randomized Experiment Is Impossible to Analyze

#analytics #data #dataengineering #datascience
1
Comments 2
6 min read
gold 숫자가 이상할 때 source_hash, quality, lineage로 원인 좁히기
junhyun-dev profile

gold 숫자가 이상할 때 source_hash, quality, lineage로 원인 좁히기

#dataengineering #python #etl #learning
Comments
2 min read
Designing an API-First Value-Based Care Analytics Stack for MA Payers
vbc_risk_analytics profile

Designing an API-First Value-Based Care Analytics Stack for MA Payers

#analytics #api #dataengineering #systemdesign
1
Comments 1
3 min read
How do I answer "what did my data look like last month" in Postgres?
wondadav profile

How do I answer "what did my data look like last month" in Postgres?

#ai #dataengineering #sql #temporalqueries
2
Comments 2
4 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.