How we built a resilient data pipeline to sync on-premise PMS with a modern platform using flat files, Rust, and Elixir Broadway.
I joined Vetolib as a senior engineer. Six months later, the CTO left. Suddenly, I inherited the biggest technical challenge: syncing data from multiple Practice Management Systems—the kind of software that runs on a Windows machine in the back office, hasn't been updated since 2015, and definitely doesn't have a REST API.
"We're flying blind," the previous architecture told me. The PMS initiates sync, calls our API, then polls to check if we're done. Zero visibility into what's happening.
This is the story of how I rebuilt the data pipeline with flat files, a Rust Lambda, and Elixir Broadway—and went from callback chaos to processing 20,000+ bookings per PMS with full observability.
👉 Read the full war story on Alembic Labs
TL;DR Architecture
PMS (Legacy) → S3 (CSVs) → Rust Lambda → SQS Queues → Broadway → Results Mailbox
Key wins:
- Flat files as the universal integration layer (works with ANY legacy system)
- One queue per PMS for isolation and visibility
- Broadway backpressure handling burst loads gracefully
- Circuit breaker catching rotten data before it pollutes the DB
The Numbers
- 3+ PMS actively syncing
- Initial sync: 8,000-20,000 bookings (100k+ total records with dependencies)
- Ongoing sync: 100-500 bookings per cycle
- Zero "we only found out via angry phone call" incidents
What's your gnarliest legacy integration story? Drop it in the comments 👇
Top comments (0)