DEV Community

#dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Apache Airflow for Java Engineers: Orchestrating Data Pipelines Beyond Python
said_olano profile

Apache Airflow for Java Engineers: Orchestrating Data Pipelines Beyond Python

#java #airflow #dataengineering #architecture
Comments
8 min read
dbt snapshots are not an audit log
aniketsoni profile

dbt snapshots are not an audit log

#sql #analytics #dataengineering #dataquality
1
Comments
4 min read
openFDA's count= aggregation shows you the top 1,000 values and never tells you it stopped
fetchsmith profile

openFDA's count= aggregation shows you the top 1,000 values and never tells you it stopped

#webscraping #api #opendata #dataengineering
Comments
6 min read
Your Python Script Works on 1,000 Rows. Then the Real CSV Arrives.
emilywoodsnyc profile

Your Python Script Works on 1,000 Rows. Then the Real CSV Arrives.

#python #dataengineering #programming #interview
Comments 1
8 min read
Running CDC in Production: The WAL Slot Trap
pipelinewhiteboard profile

Running CDC in Production: The WAL Slot Trap

#postgres #dataengineering #devops #aws
Comments
7 min read
Zero-downtime partition swaps on Athena: build in a new prefix, flip with SET LOCATION
csotelo profile

Zero-downtime partition swaps on Athena: build in a new prefix, flip with SET LOCATION

#aws #dataengineering #python #athena
Comments
5 min read
dlt's Auto-Schema Pipeline: How Python Data Loaders Infer Structure Without Predefined Models
mech_app_ai profile

dlt's Auto-Schema Pipeline: How Python Data Loaders Infer Structure Without Predefined Models

#database #dataengineering #opensource #python
Comments
4 min read
A Green Spark Job Isn't Enough: Building a Fail-Closed Pipeline Health Monitor in Python
rupak_kulkarni_08b45c5846 profile

A Green Spark Job Isn't Enough: Building a Fail-Closed Pipeline Health Monitor in Python

#python #dataengineering #devops #testing
Comments
6 min read
Hadoop Ecosystem Overview
godofgeeks profile

Hadoop Ecosystem Overview

#bigdata #data #dataengineering
Comments
8 min read
Web Scraping vs Official API: Which Should You Use?
aqib_javaid profile

Web Scraping vs Official API: Which Should You Use?

#webscraping #apiintegration #dataengineering #architecture
Comments
6 min read
Equality deletes have no address, and that's why your Iceberg reads crawl
kyien profile

Equality deletes have no address, and that's why your Iceberg reads crawl

#dataengineering #iceberg #sql #flink
Comments
7 min read
Iceberg vs Parquet: do you need a table format?
mohammed_arshadansari_f2 profile

Iceberg vs Parquet: do you need a table format?

#dataengineering #database #data #analytics
Comments
3 min read
AI in Data Engineer Interviews: Banned, Allowed, or Graded?
datadriven profile

AI in Data Engineer Interviews: Banned, Allowed, or Graded?

#dataengineering #interview #career #ai
1
Comments 1
9 min read
Proving exactly-once by killing the stream mid-backlog
tantn89 profile

Proving exactly-once by killing the stream mid-backlog

#dataengineering #spark #iceberg #kafka
Comments 3
8 min read
Spark Is Lazy, and That One Fact Explains Most Beginner Confusion
k_hunnurjirao_7ea28e971f profile

Spark Is Lazy, and That One Fact Explains Most Beginner Confusion

#pyspark #dataengineering #beginners #python
1
Comments
5 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.