DEV Community

#dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
Your Scraper Collected 50 Rows. There Were 4,000.
0012303 profile

Your Scraper Collected 50 Rows. There Were 4,000.

#webscraping #python #dataengineering #pagination
Comments
7 min read
Deeper into Dataform 3: Auditing Dataform
ben_watson_data profile

Deeper into Dataform 3: Auditing Dataform

#dataform #dataengineering #gcp #bigquery
Comments
2 min read
How I Broke Down My ETL Pipeline Project Into Smaller Engineering Exercises
tanmay_bhurkunde profile
Tanmay

How I Broke Down My ETL Pipeline Project Into Smaller Engineering Exercises

#dataengineering #database #etl #sql
Comments
2 min read
Apache Data Lakehouse Weekly: July 1 to July 8, 2026
alexmercedcoder profile

Apache Data Lakehouse Weekly: July 1 to July 8, 2026

#news #data #dataengineering #opensource
2
Comments 1
24 min read
I built a data-contract validator in pure Python (no pandas, no PyYAML) and it caught a 30% revenue ghost
hajirufai profile

I built a data-contract validator in pure Python (no pandas, no PyYAML) and it caught a 30% revenue ghost

#python #dataengineering #datascience #opensource
Comments
6 min read
A practical pipeline for turning messy business documents into spreadsheets
alex_zheng_49d3089c0d3fdf profile

A practical pipeline for turning messy business documents into spreadsheets

#dataengineering #ai #productivity
Comments
2 min read
gold 숫자가 이상할 때 source_hash, quality, lineage로 원인 좁히기
junhyun-dev profile

gold 숫자가 이상할 때 source_hash, quality, lineage로 원인 좁히기

#dataengineering #python #etl #learning
Comments
2 min read
A Fair Coin Isn't Enough: When a Perfectly Randomized Experiment Is Impossible to Analyze
hexisteme profile
John

A Fair Coin Isn't Enough: When a Perfectly Randomized Experiment Is Impossible to Analyze

#analytics #data #dataengineering #datascience
1
Comments 2
6 min read
Designing an API-First Value-Based Care Analytics Stack for MA Payers
vbc_risk_analytics profile

Designing an API-First Value-Based Care Analytics Stack for MA Payers

#analytics #api #dataengineering #systemdesign
1
Comments 1
3 min read
How do I answer "what did my data look like last month" in Postgres?
wondadav profile

How do I answer "what did my data look like last month" in Postgres?

#ai #dataengineering #sql #temporalqueries
2
Comments 2
4 min read
Day 9 of 100 Days of ClickHouse®: Mastering Data Aggregation from GROUP BY to CUBE
kanishga_subramani_49ad73 profile

Day 9 of 100 Days of ClickHouse®: Mastering Data Aggregation from GROUP BY to CUBE

#analytics #database #dataengineering #sql
1
Comments
4 min read
The Search Engine Renaissance: How Apache Lucene and Elasticsearch Are Reclaiming the AI-Native Future
vigneshh profile

The Search Engine Renaissance: How Apache Lucene and Elasticsearch Are Reclaiming the AI-Native Future

#dataengineering #lucene #elasticsearch #serverless
1
Comments
8 min read
If the warehouse already has the data, why are we copying it elsewhere?
gazerro profile

If the warehouse already has the data, why are we copying it elsewhere?

#architecture #data #dataengineering #systemdesign
Comments
5 min read
bronze, silver, and gold standard data vault 2.0
shrouwoods profile

bronze, silver, and gold standard data vault 2.0

#datavault #dataengineering #modeling #architecture
1
Comments
8 min read
The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse
alexmercedcoder profile

The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse

#news #database #dataengineering #opensource
Comments 1
22 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.