DEV Community

# dataengineering

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
How to Bypass the Pandas "Object Tax": Building an 8x Faster CSV Engine in C
nareshcn2 profile

How to Bypass the Pandas "Object Tax": Building an 8x Faster CSV Engine in C

#python #performance #dataengineering #datascience
Comments
2 min read
How Google Maps Predicts Traffic in Real Time: Live Data and ETA Explained
helloashish99 profile

How Google Maps Predicts Traffic in Real Time: Live Data and ETA Explained

#googlemaps #traffic #gps #dataengineering
Comments
3 min read
How to Use Dremio with Claude Code: Connect, Query, and Build Data Apps
alexmercedcoder profile

How to Use Dremio with Claude Code: Connect, Query, and Build Data Apps

#ai #dataengineering #mcp #tutorial
Comments
13 min read
How to Connect Power BI to a SQL (PostgreSQL) Database and Build a Unified Dashboard
damaac profile

How to Connect Power BI to a SQL (PostgreSQL) Database and Build a Unified Dashboard

#discuss #database #postgres #dataengineering
2
Comments
4 min read
Improving Data Ingestion Throughput with a Queue-Based Pipeline: Python + Duckdb
meemeealm profile

Improving Data Ingestion Throughput with a Queue-Based Pipeline: Python + Duckdb

#python #database #dataengineering #architecture
Comments
3 min read
How I cut Python JSON memory overhead from 1.9GB to ~0MB (11x Speedup)
nareshcn2 profile

How I cut Python JSON memory overhead from 1.9GB to ~0MB (11x Speedup)

#python #c #performance #dataengineering
Comments
2 min read
Database Branch Testing: How Isolated Environments Improve QA Confidence
manualwise profile

Database Branch Testing: How Isolated Environments Improve QA Confidence

#database #dataengineering
1
Comments
11 min read
Boosting Lightweight ETL on AWS Lambda & Glue Python Shell with DuckDB and Apache Arrow Dataset
datapenguin profile

Boosting Lightweight ETL on AWS Lambda & Glue Python Shell with DuckDB and Apache Arrow Dataset

#aws #duckdb #chdb #dataengineering
6
Comments
9 min read
Part 4 | Why State Machines Power Reliable Scheduling Systems
chen_debra_3060b21d12b1b0 profile

Part 4 | Why State Machines Power Reliable Scheduling Systems

#architecture #dataengineering #opensource #systemdesign
Comments
6 min read
Building a Production-Ready Serverless App on Google Cloud (Part 2: The Data Contract)
patitonav profile

Building a Production-Ready Serverless App on Google Cloud (Part 2: The Data Contract)

#ai #dataengineering #python #googlecloud
7
Comments
4 min read
Quantified Self: Building a Production-Grade ETL Pipeline for 10+ Wearables
beck_moulton profile

Quantified Self: Building a Production-Grade ETL Pipeline for 10+ Wearables

#python #dataengineering #opensource #airflow
2
Comments
4 min read
Our Data Extraction Pipeline Worked Perfectly… Until Month 6
baldur12_a7f03927a09696e1 profile

Our Data Extraction Pipeline Worked Perfectly… Until Month 6

#dataengineering #datascience #datastructures #dataextraction
1
Comments
2 min read
Share of Shelf Analysis: How to Scrape Zappos Search Results
anderecit profile

Share of Shelf Analysis: How to Scrape Zappos Search Results

#webdev #devops #webscraping #dataengineering
1
Comments
4 min read
Iterator Patterns: How to Process Millions of Records Without Running Out of Memory
kunwar-jhamat profile

Iterator Patterns: How to Process Millions of Records Without Running Out of Memory

#php #programming #dataengineering #performance
1
Comments
5 min read
Why My Metrics Pipeline with Telegraf Didn’t Work (and What I Learned)
mohhddhassan profile

Why My Metrics Pipeline with Telegraf Didn’t Work (and What I Learned)

#dataengineering #clickhouse #devops #observability
2
Comments
2 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.