PardoX: Processing 640M Rows on a Standard Laptop — The High-Performance Rust ETL Engine

#devchallenge #muxchallenge #showandtell #video

DEV's Worldwide Show and Tell Challenge Submission 🎥

This is a submission for the DEV's Worldwide Show and Tell Challenge Presented by Mux

What I Built

I built pardoX, a high-performance ETL engine written in Rust designed specifically for the "Mid Data" gap. While modern tools excel at gigabytes in the cloud, they often choke on local hardware. PardoX allows a standard corporate laptop (like an i5 with 16GB RAM) to process massive datasets (640M+ rows) with the efficiency of a high-end server, converting raw CSVs to optimized Parquet files at speeds exceeding 80 MB/s.

My Pitch Video

Demo

PardoX is currently in Private Beta as we refine the final engine.

Beta Launch: January 19, 2026.

Project Updates & Benchmarks: Read the full story on Medium.

Early Access: If you are a judge or a developer dealing with "Memory Errors" on your local machine, please contact me at iam@albertocardenas.com for a pre-release binary and testing instructions.

The Story Behind It

I built this because of the "Forgotten Sector." I was tired of being at the office at 7 PM, watching my laptop freeze while trying to load a 3GB CSV into Pandas. Most engineers don't have $5,000 workstations; we have corporate laptops. I wanted to build a tool that respects the user's time and hardware. PardoX is my "love letter" to data engineers working in the trenches who need industrial-scale power without the cloud's price tag or the JVM's overhead.

Technical Highlights

Rust-Powered: Leverages Rust's memory safety and performance without a Garbage Collector.
Zero-Copy Architecture: Data is streamed and processed with minimal allocations, preventing OOM (Out of Memory) errors.
Hardware-Awareness: The engine "reads" your CPU and RAM to surgically allocate thread pools for reading and compression, preventing system contention.
Smart Sharding: Automatically fragments massive outputs to prevent OS file-system choking, ensuring a "Zero Friction" data flow.
Performance: In recent tests, PardoX processed 640 million rows in 206 seconds, outperforming both DuckDB and Polars on the same local hardware.

Use of Mux (Additional Prize Category Participants Only)

I utilized Mux to host and stream the pitch video for this challenge. The integration was seamless, providing a high-performance video delivery that matches the "speed and efficiency" philosophy of my own project.