DEV Community

#dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
The fastest Glue job is the one you never start: removing per-job overhead from a 191-job pipeline
kiran_gunturu_022085015ec profile

The fastest Glue job is the one you never start: removing per-job overhead from a 191-job pipeline

#aws #dataengineering #serverless #performance
Comments
8 min read
dbt Was Not Designed for Multi-Tenant SaaS. Here's How I Made It Work.
eu_ti_f127c5b5d7535b7174f profile

dbt Was Not Designed for Multi-Tenant SaaS. Here's How I Made It Work.

#dataengineering #sql #python #architecture
Comments
4 min read
Building a real-time price monitoring pipeline that does not poison your data
anakin_writers profile

Building a real-time price monitoring pipeline that does not poison your data

#webdev #dataengineering #scraping #analytics
Comments
4 min read
One Iceberg MCP Server, Seven Catalogs: What It Takes to Reach Each One
xbill profile

One Iceberg MCP Server, Seven Catalogs: What It Takes to Reach Each One

#mcp #iceberg #python #dataengineering
11
Comments
7 min read
Building an investing knowledge graph, part 3: same company, seven names
hannune profile

Building an investing knowledge graph, part 3: same company, seven names

#graphdb #knowledgegraph #dataengineering #entityresolution
2
Comments
4 min read
Why a Numaflow watermark wouldn’t move
akarazhev profile

Why a Numaflow watermark wouldn’t move

#dataengineering #distributedsystems #rust #opensource
Comments
7 min read
The Pipeline Worked. Then the Research Outgrew It.
daemonic01 profile

The Pipeline Worked. Then the Research Outgrew It.

#dataengineering #python #programming #productivity
Comments
4 min read
Subqueries vs CTEs: Query Optimizer Internals & Memory Spooling Explained
arpitmbangre profile

Subqueries vs CTEs: Query Optimizer Internals & Memory Spooling Explained

#database #sql #dataengineering #performance
Comments
2 min read
Why Data Reconciliation Matters in AWS Glue: Finding Missing Records with Anti Joins
rahul_r15 profile

Why Data Reconciliation Matters in AWS Glue: Finding Missing Records with Anti Joins

#dataengineering #bigdata #aws #python
Comments
4 min read
One Iceberg MCP Server, Seven Catalogs: What It Takes to Reach Each One
xbill profile

One Iceberg MCP Server, Seven Catalogs: What It Takes to Reach Each One

#mcp #iceberg #python #dataengineering
Comments 2
7 min read
Building a Data Audit Workbench That Holds Up in an Assessment
maref profile

Building a Data Audit Workbench That Holds Up in an Assessment

#compliance #dataengineering #audit
Comments 1
3 min read
How to Build an Open Lakehouse on Your Laptop
lisancao profile
lnc

How to Build an Open Lakehouse on Your Laptop

#opensource #dataengineering #architecture #infrastructure
Comments
42 min read
Your ERP says the job was profitable. Your machines disagree.
trcollinson profile

Your ERP says the job was profitable. Your machines disagree.

#aws #manufacturing #dataengineering #athena
Comments 1
16 min read
Time-Series Cardinality: Why One More Indexed Column Costs More Than a Million More Rows
tigerdata_dev profile

Time-Series Cardinality: Why One More Indexed Column Costs More Than a Million More Rows

#timeseriesdata #postgres #dataengineering
Comments
13 min read
The Crash That Never Happened: Cleaning Price Data Before You Trust It
jaroslav_marda_9f25d4d9c profile

The Crash That Never Happened: Cleaning Price Data Before You Trust It

#python #datascience #finance #dataengineering
1
Comments 9
5 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.