DEV Community

#dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Improving Data Ingestion Throughput with a Queue-Based Pipeline: Python + Duckdb
meemeealm profile

Improving Data Ingestion Throughput with a Queue-Based Pipeline: Python + Duckdb

#python #database #dataengineering #architecture
Comments
3 min read
How I cut Python JSON memory overhead from 1.9GB to ~0MB (11x Speedup)
nareshcn2 profile

How I cut Python JSON memory overhead from 1.9GB to ~0MB (11x Speedup)

#python #c #performance #dataengineering
Comments
2 min read
Database Branch Testing: How Isolated Environments Improve QA Confidence
manualwise profile

Database Branch Testing: How Isolated Environments Improve QA Confidence

#database #dataengineering
1
Comments
11 min read
Boosting Lightweight ETL on AWS Lambda & Glue Python Shell with DuckDB and Apache Arrow Dataset
datapenguin profile

Boosting Lightweight ETL on AWS Lambda & Glue Python Shell with DuckDB and Apache Arrow Dataset

#aws #duckdb #chdb #dataengineering
6
Comments
9 min read
Part 4 | Why State Machines Power Reliable Scheduling Systems
chen_debra_3060b21d12b1b0 profile

Part 4 | Why State Machines Power Reliable Scheduling Systems

#architecture #dataengineering #opensource #systemdesign
Comments
6 min read
Building a Production-Ready Serverless App on Google Cloud (Part 2: The Data Contract)
patitonav profile

Building a Production-Ready Serverless App on Google Cloud (Part 2: The Data Contract)

#ai #dataengineering #python #googlecloud
7
Comments
4 min read
Quantified Self: Building a Production-Grade ETL Pipeline for 10+ Wearables
beck_moulton profile

Quantified Self: Building a Production-Grade ETL Pipeline for 10+ Wearables

#python #dataengineering #opensource #airflow
2
Comments
4 min read
Our Data Extraction Pipeline Worked Perfectly… Until Month 6
baldur12_a7f03927a09696e1 profile

Our Data Extraction Pipeline Worked Perfectly… Until Month 6

#dataengineering #datascience #datastructures #dataextraction
1
Comments
2 min read
Share of Shelf Analysis: How to Scrape Zappos Search Results
anderecit profile

Share of Shelf Analysis: How to Scrape Zappos Search Results

#webdev #devops #webscraping #dataengineering
1
Comments
4 min read
Iterator Patterns: How to Process Millions of Records Without Running Out of Memory
kunwar-jhamat profile

Iterator Patterns: How to Process Millions of Records Without Running Out of Memory

#php #programming #dataengineering #performance
1
Comments
5 min read
Why My Metrics Pipeline with Telegraf Didn’t Work (and What I Learned)
mohhddhassan profile

Why My Metrics Pipeline with Telegraf Didn’t Work (and What I Learned)

#dataengineering #clickhouse #devops #observability
2
Comments
2 min read
Python was too slow for 10M rows—So I built a C-Bridge (and found the hidden data loss)
nareshcn2 profile

Python was too slow for 10M rows—So I built a C-Bridge (and found the hidden data loss)

#python #cpp #performance #dataengineering
Comments
2 min read
Why Salesforce to Power BI Integrations Break at Scale (And How to Design Them Properly)
alexcybersmith profile

Why Salesforce to Power BI Integrations Break at Scale (And How to Design Them Properly)

#salesforce #dataengineering #devops #analytics
Comments
7 min read
Data Engineering Interviews Are Broken (Here's Proof)
datadriven profile

Data Engineering Interviews Are Broken (Here's Proof)

#dataengineering #career #interview #python
Comments
6 min read
How I stopped bad data from reaching my warehouse using a single Airflow task
appdeviq profile

How I stopped bad data from reaching my warehouse using a single Airflow task

#dataengineering #airflow #python #api
Comments
4 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.