Keeping downstream systems in sync with a database is one of those problems every team ends up solving badly — cron jobs that miss deletes, sync scripts that drift, search indexes you can only trust after the nightly rebuild. The only time you find out something went wrong is when a user reports stale data.
VentStream solves this by reading the database's change stream directly and keeping replicas in sync within milliseconds — including the hard parts: joined documents, delete propagation, and exactly-once delivery through deterministic document ids.
What it does
Sources: Postgres, MySQL, MongoDB, Neo4j, Kafka
Sinks: OpenSearch / Elasticsearch, Meilisearch, Redis
Realtime: GraphQL subscriptions and WebSocket fan-out with cursor resume — reconnect and pick up exactly where you left off
You can declare joins once — an order embeds its customer and line items — and the engine keeps the composed documents in sync from the change stream. A delete on a child row updates the parent document in the index. No sync jobs, no invalidation code.
One thing we just shipped: an MCP server for AI agents
The engine now includes a built-in MCP server, so AI agents (Claude Desktop, Claude Code, any MCP client) can query your synced data directly:
- Agents get live answers without holding any database credentials
- Scoped access tokens per agent — an agent sees only the targets you allow
- The joins spec doubles as schema documentation the agent discovers by itself
Point Claude at your replicas with one subcommand and it can answer "what's the status of order 4351?" from data that's milliseconds fresh.
Try it
One line, macOS and Linux:
curl -fsSL https://ventstream.dev/install.sh | sh
Stream a MongoDB database into Elasticsearch — this is the whole setup:
VS_ROLES=cdc VS_CDC_SOURCE=mongodb \
VS_MONGO_URI='mongodb+srv://user:pass@cluster.example.net/' \
VS_MONGO_DATABASE=shop \
VS_MONGO_STATE_DIR=./state \
VS_MONGO_BOOTSTRAP_MODE=snapshot \
VS_OS_ENDPOINT=http://localhost:9200 \
VS_INDEX_TEMPLATE='${header:ventstream.cdc.relation}' \
ventstream
Docs: https://ventstream.dev/docs
Repo: https://github.com/ventstream/ventstream
Looking for testers and contributors
The engine is running in production against sustained load, but real-world databases are where sync tools earn their keep — different schemas, different scales, different failure modes. If you run Postgres, MySQL, MongoDB, or Neo4j and have ever fought a sync pipeline, I'd genuinely like to hear what breaks or what's missing.
Open to contributors as well — the connector surface (Typesense and ClickHouse sinks are on the list), the MCP tools, and docs are all good places to start. Issues and discussions are open on the repo.
Top comments (2)
Promising project. I’d be careful with the phrase “exactly-once delivery through deterministic document IDs.” Deterministic IDs make duplicate upserts idempotent, but delivery across a source log and an external sink is usually still at-least-once. Ordering and stale writes remain separate problems.
For each projection key, carry the source position/version (LSN, binlog coordinate, resume token, etc.) and have the sink reject an event older than the applied version. Make the snapshot-to-stream handoff explicit with a high-water mark, retain delete tombstones long enough to defeat late events, and checkpoint only after the sink acknowledges the corresponding version.
Joined documents are the hardest case: one child change may fan out to many parents, and a crash can leave a partially updated set. I’d want deterministic dependency expansion, resumable fan-out batches, a dead-letter path, and periodic source-vs-sink reconciliation with repair.
For the MCP side, return freshness evidence with every answer: source position, applied position, observed lag, projection/schema version, generated-at time, and whether reconciliation is currently clean. Also enforce agent scope in the query/projection layer, not only in tool discovery, and include the authorization fingerprint in any cache key. That makes “milliseconds fresh” and “only the targets you allow” auditable properties rather than architecture claims.
This is a great breakdown — thank you. You're right about the phrasing: delivery is at-least-once; what deterministic ids give is idempotent application, so the effect is exactly-once.
On the mechanics — most of what you describe is how it actually works:
Checkpoints advance only after the sink acknowledges — the source tails from a sink-confirmed cursor, so a crash replays from the last applied position rather than trusting the log position.
The Redis sink carries per-key versions and rejects writes older than the applied version — the stale-write guard you describe. Snapshot→stream handoff is an explicit position capture before bootstrap, then tailing from it.
Joined docs: child→parent expansion goes through a maintained reverse index, batches are idempotent so crash-recovery replays them safely, there's a dead-letter path with exact offsets, and there's a reconciliation pass that diffs sink doc-ids against the source's live key set and repairs — deletes included. Delete handling under Postgres REPLICA IDENTITY DEFAULT is one of the harder cases we specifically test.
The part I'm taking away as a genuinely missing piece: freshness evidence on MCP answers. Today the server surfaces pipeline health as a separate signal, but returning source/applied positions, observed lag, and a generated-at with each answer would make freshness auditable per response instead of claimed — that's going on the roadmap.
(Scope is already enforced in the query layer, not just discovery — out-of-scope targets are indistinguishable from nonexistent ones — and there's no answer cache, so no cache-key concern yet; noted for when there is one.)
If you're interested in poking at any of this, the reconciliation and joins code is where the bodies are buried — happy to point you at it.