DEV Community: Bashiru Bukari

CDC without Kafka: keeping Postgres, OpenSearch, and Redis in sync with one binary

Bashiru Bukari — Sat, 29 Aug 2026 16:27:01 +0000

Every time I needed to keep Postgres in sync with something else — a search index, a cache, or another database — I seemed to end up with the same stack:

Debezium → Kafka Connect → Kafka → a consumer I still had to write.

There’s nothing wrong with that stack. Kafka is great at what it does.

But sometimes I just wanted this:

Postgres changes → update OpenSearch.

Or:

Postgres changes → update Redis.

Running a message broker for that felt like a lot of infrastructure for a fairly simple problem.

So I built VentStream.

It’s a single Rust binary that reads changes directly from databases, can join related records, and writes the resulting documents to your target.

This post is about why I built it, some of the decisions that turned out to matter, and the numbers I got while testing it.

It's open source and Apache-2.0 licensed.

The problem that started this

At work, we had a sync job that ran every ten minutes.

It basically did this:

Read the dataset from Postgres.
Join some of the tables in application code.
Rebuild the search index.
Wait ten minutes.
Do it all again.

The result was predictable.

The search index could be 10–12 minutes behind the database, the database was constantly being scanned, and we were doing work even when nothing had changed.

The annoying part was the joins.

An orders row by itself wasn't particularly useful to the search index. We wanted something more like:

{
  "id": "order_123",
  "status": "paid",
  "customer": {
    "id": "customer_456",
    "name": "John Doe"
  }
}

So every ten minutes we were effectively asking the database for everything, joining everything, and rebuilding everything.

CDC is obviously a better fit.

Instead of asking:

"What does the whole database look like?"

you ask:

"What changed?"

Postgres already knows the answer through logical replication.

The problem is that most CDC setups assume you're going to put those changes into Kafka.

If you already run Kafka, that's fine.

If you don't, you're adding a fairly large piece of infrastructure just to move changes from one database to another system.

That's the gap I wanted to explore.

What VentStream does

The basic idea is pretty simple:

        Database
           │
           │ CDC
           ▼
      ┌───────────┐
      │ VentStream│
      └─────┬─────┘
            │
            ▼
     Search / Cache / DB

VentStream connects directly to the source's native change feed, keeps the state it needs for joins, and writes to the destination.

No Kafka required.

Currently, the sources include:

Source	Change mechanism
PostgreSQL / Supabase	Logical replication (`pgoutput`)
MySQL / MariaDB	Row-based binlog
MongoDB	Change streams
Neo4j 5.17+ Enterprise	CDC log
Kafka / Redpanda	Debezium envelopes or raw topics

And the sinks include:

Sink	How VentStream writes
OpenSearch / Elasticsearch	Bulk API + external versioning
Meilisearch	Task-confirmed writes
Redis	Keyspace or view materialization
SurrealDB	Native RPC + real record IDs

The important part for me is that this is all one process.

A real Postgres → OpenSearch config

Here's what a Postgres → OpenSearch configuration currently looks like:

schema_version: 1

roles: [cdc]

source:
  kind: postgres
  postgres:
    host_ref: env:VS_PG_HOST
    port: 5432
    user_ref: env:VS_PG_USER
    password_ref: env:VS_PG_PASSWORD
    database_ref: env:VS_PG_DATABASE
    publication_ref: env:VS_PG_PUBLICATION
    slot_ref: env:VS_PG_SLOT

    bootstrap:
      mode: snapshot
      chunk_size: 10000

sink:
  kind: opensearch
  opensearch:
    endpoint_ref: env:VS_OS_ENDPOINT
    index_routing:
      strategy: by_output_relation

runtime:
  health_listen: 127.0.0.1:4043
  dlq_path: ./ventstream-state/dlq.jsonl

  memory:
    enabled: true
    budget_bytes: 268435456

  joins:
    state_dir: ./ventstream-state/joins

There are a few things I care about here.

Secrets are referenced through environment variables.

Memory has an explicit budget.

And the dead-letter queue is just a file.

If something goes wrong, I can actually open it and inspect it.

The three decisions that mattered most

Removing Kafka isn't particularly interesting by itself.

The harder question is:

Can you remove Kafka without losing the reliability properties people use it for?

There were three areas where this mattered a lot.

1. Don't move the cursor until the sink has the data

This is probably the most important part of the system.

With Postgres logical replication, the replication slot has a confirmed_flush_lsn.

Once Postgres knows you've processed everything up to a particular LSN, it can eventually remove the older WAL.

So you really don't want to say:

"I've read this change, so I'm done."

Reading it isn't enough.

VentStream only advances the replication watermark after the sink has acknowledged the write.

It also tracks a contiguous sequence of successfully processed changes.

For example:

Batch 1 ✓
Batch 2 ✓
Batch 3 ✓
Batch 4 ✓
Batch 5 ✗
Batch 6 ✓

The watermark stays at batch 4.

It doesn't jump to batch 6 because that would create a hole.

That means if the process dies after batch 6, it can restart from the last confirmed point rather than assuming the missing batch was processed.

I tested this by pushing 250,000 mutations through the pipeline and repeatedly hard-killing the process.

The source and sink ended up with zero drift in those validation runs.

That's the kind of failure case I care about much more than a pretty benchmark number.

2. Do the joins where the change happens

This was the other big reason I wanted to build this.

Suppose we have:

customers
orders

An order might reference a customer:

orders.customer_id → customers.id

If the customer changes, the order document in OpenSearch may also need to change.

With a basic CDC setup, you get something like:

customer changed
       │
       ▼
   CDC event
       │
       ▼
   your consumer
       │
       ├── find affected orders
       ├── fetch customer
       ├── rebuild documents
       └── update OpenSearch

I wanted the CDC engine itself to understand that relationship.

VentStream keeps join state locally and produces the composed document.

So when a customer changes, the affected orders can be recomposed and emitted again.

This also exposed an interesting Postgres detail.

TOAST columns

With pgoutput, unchanged TOAST columns may not be included in an UPDATE.

So if you treat every update as:

"Replace the whole document with whatever was in this event"

you can accidentally wipe out large text fields that weren't part of the update.

VentStream merges the incoming change into the stored row instead.

So an update like:

status = "paid"

doesn't accidentally turn:

description = "a very large text field..."

into null.

These are the kinds of details that don't show up in the basic CDC diagrams but become very important once you're actually running the thing.

3. A bad row shouldn't kill the entire pipeline

This one came from an actual failure mode.

Imagine the pipeline receives a record it can't process.

Maybe the payload isn't a JSON object.

Maybe it doesn't look like a valid CDC event.

Previously, the simplest thing to do was to stop.

But that creates a nasty loop:

bad event
   ↓
pipeline stops
   ↓
restart
   ↓
same bad event
   ↓
pipeline stops

The cursor can't move past it because the event wasn't processed.

So one bad record can effectively brick the pipeline.

The solution is a dead-letter queue.

But there's an important detail.

VentStream writes the event to the DLQ and calls fsync() before advancing the cursor past its LSN.

So the ordering is:

bad event
   ↓
write DLQ
   ↓
fsync
   ↓
advance cursor

Not:

bad event
   ↓
advance cursor
   ↓
write DLQ

The second version has a nasty failure window.

If the machine loses power between those two operations, the source may already have moved past the event while the DLQ entry only existed in the page cache.

Now the event is gone.

There are also failures that I deliberately don't put in the DLQ.

For example, if the source database disappears or the sink can't be reached, that's not a bad row.

That's an environment problem.

The event should be retried.

Otherwise you end up quarantining perfectly valid data just because your network went down.

So, how fast is it?

These aren't polished benchmark-suite numbers.

They're local validation runs on my machine.

I'm sharing them because they helped me decide whether the architecture was actually working.

SurrealDB

200,000 rows with graph edges:

Full bootstrap: 11 seconds
Peak RSS: 23 MiB
RSS while tailing: 8.3 MiB
Re-running the bootstrap produced the same 200,010 edges
No duplicate edges

Postgres → SQL-denormalized documents

2 million rows:

RSS stayed between 24 and 86 MiB
Memory remained bounded throughout the run

That's because the memory budget isn't just a configuration value sitting there for decoration.

It actually controls admission.

Neo4j projection

1 million documents plus a 250,000-row cascade:

26–100 MiB RSS

Postgres → OpenSearch

60,000 rows through the complete join path:

0 DLQ entries
Exact row-for-row match between the source and index

The main reason the memory usage stays relatively low is architectural.

There isn't a broker sitting in the middle serializing and deserializing every event.

There's no JVM.

And the joins happen once when the data changes instead of once during every full rescan.

But should you actually replace Kafka?

Probably not.

I don't think "Kafka is unnecessary" is a useful conclusion from this.

Kafka is very good at being a distributed log.

You probably want Kafka when:

many independent consumers need the same change stream
consumers need to replay events independently
you need long-term retention
the CDC stream is becoming an organisation-wide event backbone
multiple teams and applications depend on the same events

That's a different problem.

If your architecture looks like this:

Postgres
   │
   ├── Search index
   └── Redis

a single-process CDC engine can make a lot of sense.

If it looks like this:

                         ┌── Analytics
                         │
Postgres → Kafka →───────┼── Search
                         │
                         ├── Data warehouse
                         │
                         └── 20 other consumers

Kafka is probably the better tool.

They're not mutually exclusive either.

VentStream can consume Debezium-formatted Kafka topics, so you can keep Kafka as the event backbone and use VentStream for the join/materialization part.

Try it

If you want to play with it:

curl -fsSL https://ventstream.dev/install.sh | sh

Or with Docker:

docker run ghcr.io/ventstream/ventstream:<version>

There's also a demo in the repository that starts Postgres, Neo4j, and OpenSearch and streams joined documents between them.

There's a live SurrealDB demo too, showing orders joined with customers.

Site/docs: https://ventstream.dev
Source: https://github.com/ventstream/ventstream
Comparison: https://ventstream.dev/compare
Live demo: https://surreal-demo.ventstream.dev

What's still rough?

Quite a bit.

It's early. The 0.1.x version numbers are there for a reason.

A few things I'm not happy with yet:

PostgreSQL → PostgreSQL isn't supported as a sink yet.
The DLQ replay behaviour for poison rows needs better documentation.
Bootstrap could deduplicate upstream lookups more aggressively.
There are still plenty of edge cases that only show up when you run CDC for long enough.

That's also why I'm putting this out there now.

I'd rather have people try it against real systems and tell me where it breaks than pretend it's finished.

If you run VentStream and it does something weird, please open an issue.

Those reports are much more useful to me than another benchmark where everything runs perfectly.

One last thing

The main reason I built this isn't because I think Kafka is bad.

It's because I kept seeing relatively simple data-sync problems turn into relatively large infrastructure projects.

Sometimes you need Kafka.

Sometimes you just need:

database → change feed → transform → destination

And for that second case, I wanted to see how far a single binary could go.

That's what VentStream is trying to be.

50M docs bootstrapped per source into OpenSearch on 2 vCPU / 1GB on ventstream

Bashiru Bukari — Thu, 20 Aug 2026 23:30:55 +0000

I build an open-source CDC engine (single Rust binary). A user asked if it handles big workloads, so I ran 50M pre-seeded docs from each source through a snapshot bootstrap into OpenSearch, engine capped at 2 vCPU / 1GiB, and verified every run against the exact source count. This is impressive for just a single rust binary

I can now comfortably move away from kafka and debezium.

try it here https://ventstream.dev/docs
I need your feedback and contributions

How to sync PostgreSQL to Meilisearch in real time — without dual-writes

Bashiru Bukari — Sun, 16 Aug 2026 10:00:03 +0000

Originally published on the VentStream blog.

If you run PostgreSQL and want Meilisearch to stay in step with it, the usual first attempt is writing to both from application code. Two writes, no shared transaction — and a set of headaches that never really goes away:

Partial-failure drift — a crash between the two writes leaves the index lying about the database.
Retry duplicates — replaying the code path writes the same document twice.
Forgotten deletes — a row disappears from Postgres and lives on in search forever.
Backfill scripts — a one-off "initial load" path that behaves differently from the live one and drifts.
Sync logic everywhere — every new code path that touches the table has to remember to touch the index.

The database already has the truth and a change feed — use it.

The guide: Postgres to Meilisearch in four steps

The flow is: PostgreSQL (WAL) → publication → VentStream → Meilisearch. VentStream is an open-source (Apache-2.0) Rust engine that attaches to Postgres logical replication and materializes documents into the index.

1. Publish the table

The table owner decides what leaves the database, in the database:

CREATE PUBLICATION vs_pub FOR TABLE orders;

2. Write the config

This is the entire file. Secrets are environment references, never inline values:

schema_version: 1
roles: [cdc]

source:
  kind: postgres
  postgres:
    host_ref: env:VS_PG_HOST
    user_ref: env:VS_PG_USER
    password_ref: env:VS_PG_PASSWORD
    database_ref: env:VS_PG_DATABASE
    publication_ref: env:VS_PG_PUBLICATION
    slot_ref: env:VS_PG_SLOT
    bootstrap:
      mode: snapshot

sink:
  kind: meilisearch
  meilisearch:
    endpoint_ref: env:VS_MEILI_ENDPOINT
    api_key_ref: env:VS_MEILI_KEY
    index_routing:
      mode: fixed
      index: orders

That is the whole setup. The engine also supports table joins — composing an order with its line items into one document — but we're keeping this one simple.

3. Set the secrets

Every env: reference in the config resolves from the engine's environment at startup, so the file itself stays safe to commit. Put the values wherever you normally keep secrets — an env file, systemd Environment= directives, or a Kubernetes Secret. Locally, an env file next to your ventstream.yaml does fine — the shell exports it before the engine starts; the engine itself only reads environment variables:

# .env — the values the config references
VS_PG_HOST=db.internal
VS_PG_USER=ventstream
VS_PG_PASSWORD=…
VS_PG_DATABASE=shop
VS_PG_PUBLICATION=vs_pub
VS_PG_SLOT=vs_slot
VS_MEILI_ENDPOINT=https://search.internal:7700
VS_MEILI_KEY=…

4. Install and run

The installer covers macOS and Linux. On Windows, use WSL2 or the container image.

curl -fsSL https://ventstream.dev/install.sh | sh

set -a && source ./.env && set +a
VS_ENGINE_CONFIG=./ventstream.yaml ventstream

What you see

The engine snapshot-bootstraps your existing rows in keyset-paginated chunks, then tails the WAL from the exact watermark where the snapshot ended — same document shapes, same ids, one continuous motion. There is no "initial load script" that behaves differently from the live path. Once it is tailing, live changes become searchable within milliseconds to seconds of the commit, depending on the sink's ingestion speed. One detail to know: the sink namespaces index names, so index: orders shows up in Meilisearch as vs_orders (the prefix is configurable).

Why it stays correct

The short version of how VentStream keeps the index and the table in lockstep:

Every document gets a deterministic id derived from the primary key — canonical form public.orders:["42"], carried as _vs_id; the Meilisearch primary key _vs_pk is its URL-safe encoding. Re-emits overwrite, never duplicate.
Updates upsert in place. Deletes always find their target. Even a primary-key UPDATE removes the old document and writes the new one.
Writes are Meilisearch task-confirmed in FIFO order, and the engine's cursor only advances after the sink confirms durability — so after a crash or restart, the engine resumes from the last confirmed write rather than skipping or replaying events.

How fast is it?

Depending on the configuration and the resources you give it, the engine sustains from thousands up to tens of thousands of events per second. Our verified benchmarks reached roughly 930k documents per minute bootstrapping 4 million rows into Meilisearch on a laptop (M3 Max, single node, exact-count-verified), with the engine around 10–14% of one core. Meilisearch's own ingestion is usually the floor, not the engine.

When flat rows stop being enough — say you want each order indexed with its line items — VentStream can compose parent and child rows into one document with a joins spec. That deserves its own walkthrough; we'll cover it in a follow-up post.

Run it wherever you like

The engine is open source and runs anywhere a binary runs. When you want managed configuration, health, and operations across a fleet, the same binary attaches to VentStream Cloud with a single agent key — no key, and it never talks to the platform at all.

Questions or a source/sink you'd like to see supported? Open an issue — I read all of them.

VentStream — open source CDC engine that syncs your database to search, cache, and AI agents in realtime

Bashiru Bukari — Tue, 11 Aug 2026 18:21:56 +0000

Keeping downstream systems in sync with a database is one of those problems every team ends up solving badly — cron jobs that miss deletes, sync scripts that drift, search indexes you can only trust after the nightly rebuild. The only time you find out something went wrong is when a user reports stale data.

VentStream solves this by reading the database's change stream directly and keeping replicas in sync within milliseconds — including the hard parts: joined documents, delete propagation, and exactly-once delivery through deterministic document ids.

What it does

Sources: Postgres, MySQL, MongoDB, Neo4j, Kafka

Sinks: OpenSearch / Elasticsearch, Meilisearch, Redis

Realtime: GraphQL subscriptions and WebSocket fan-out with cursor resume — reconnect and pick up exactly where you left off

You can declare joins once — an order embeds its customer and line items — and the engine keeps the composed documents in sync from the change stream. A delete on a child row updates the parent document in the index. No sync jobs, no invalidation code.

One thing we just shipped: an MCP server for AI agents

The engine now includes a built-in MCP server, so AI agents (Claude Desktop, Claude Code, any MCP client) can query your synced data directly:

Agents get live answers without holding any database credentials
Scoped access tokens per agent — an agent sees only the targets you allow
The joins spec doubles as schema documentation the agent discovers by itself

Point Claude at your replicas with one subcommand and it can answer "what's the status of order 4351?" from data that's milliseconds fresh.

Try it

One line, macOS and Linux:

curl -fsSL https://ventstream.dev/install.sh | sh

Stream a MongoDB database into Elasticsearch — this is the whole setup:

VS_ROLES=cdc VS_CDC_SOURCE=mongodb \
VS_MONGO_URI='mongodb+srv://user:pass@cluster.example.net/' \
VS_MONGO_DATABASE=shop \
VS_MONGO_STATE_DIR=./state \
VS_MONGO_BOOTSTRAP_MODE=snapshot \
VS_OS_ENDPOINT=http://localhost:9200 \
VS_INDEX_TEMPLATE='${header:ventstream.cdc.relation}' \
ventstream

Docs: https://ventstream.dev/docs

Repo: https://github.com/ventstream/ventstream

Looking for testers and contributors

The engine is running in production against sustained load, but real-world databases are where sync tools earn their keep — different schemas, different scales, different failure modes. If you run Postgres, MySQL, MongoDB, or Neo4j and have ever fought a sync pipeline, I'd genuinely like to hear what breaks or what's missing.

Open to contributors as well — the connector surface (Typesense and ClickHouse sinks are on the list), the MCP tools, and docs are all good places to start. Issues and discussions are open on the repo.