I build an open-source CDC engine (single Rust binary). A user asked if it handles big workloads, so I ran 50M pre-seeded docs from each source through a snapshot bootstrap into OpenSearch, engine capped at 2 vCPU / 1GiB, and verified every run against the exact source count. This is impressive for just a single rust binary
I can now comfortably move away from kafka and debezium.
try it here https://ventstream.dev/docs
I need your feedback and contributions
Top comments (2)
Very cool! Would love to hear about some of the performance decisions and what makes to so fast. Thanks for sharing!!
Thanks! A few decisions do most of the work:
Bootstrap is a keyset-paginated snapshot with prefetch — it pages through the source by primary key (never OFFSET), and while one chunk is being written to the sink, the next is already being fetched. The engine never holds more than a couple of chunks in memory, which is why RSS stays flat (~250MB) no matter how big the dataset is.
Deterministic document ids make everything else legal. Every doc maps to the same id every time, so a retry is an overwrite, not a duplicate. That's what lets the dispatcher keep up to 16 bulk requests in flight against OpenSearch (with external versioning so reordering can't corrupt anything) instead of writing one batch at a time.
Docs pass through mostly as raw JSON — no deserialize/reserialize round-trip on the hot path.
OpenSearch is configured for ingest during bootstrap — refresh disabled, no replicas, one final refresh before the count check.
Graph sources stream instead of paginate. For Neo4j, the bootstrap runs one streamed Bolt query over the graph rather than paging with LIMIT/ORDER BY — the driver pulls results lazily in fetch-size batches, so there's no per-page re-planning or sorting on the server, memory stays bounded, and scan cost is strictly O(n) no matter how large the graph gets. That's how a 50M-node graph bootstraps in the same ~6-minute class as the relational sources.
i love if you can star to support the project
github.com/ventstream/ventstream
please add an issue if you notice anything off