DEV Community

Bashiru Bukari
Bashiru Bukari

Posted on

CDC without Kafka: keeping Postgres, OpenSearch, and Redis in sync with one binary

#postgres #rust #database #opensource

Every time I needed to keep Postgres in sync with something else — a search index, a cache, or another database — I seemed to end up with the same stack:

Debezium → Kafka Connect → Kafka → a consumer I still had to write.

There’s nothing wrong with that stack. Kafka is great at what it does.

But sometimes I just wanted this:

Postgres changes → update OpenSearch.

Or:

Postgres changes → update Redis.

Running a message broker for that felt like a lot of infrastructure for a fairly simple problem.

So I built VentStream.

It’s a single Rust binary that reads changes directly from databases, can join related records, and writes the resulting documents to your target.

This post is about why I built it, some of the decisions that turned out to matter, and the numbers I got while testing it.

It's open source and Apache-2.0 licensed.

The problem that started this

At work, we had a sync job that ran every ten minutes.

It basically did this:

  1. Read the dataset from Postgres.
  2. Join some of the tables in application code.
  3. Rebuild the search index.
  4. Wait ten minutes.
  5. Do it all again.

The result was predictable.

The search index could be 10–12 minutes behind the database, the database was constantly being scanned, and we were doing work even when nothing had changed.

The annoying part was the joins.

An orders row by itself wasn't particularly useful to the search index. We wanted something more like:

{
  "id": "order_123",
  "status": "paid",
  "customer": {
    "id": "customer_456",
    "name": "John Doe"
  }
}
Enter fullscreen mode Exit fullscreen mode

So every ten minutes we were effectively asking the database for everything, joining everything, and rebuilding everything.

CDC is obviously a better fit.

Instead of asking:

"What does the whole database look like?"

you ask:

"What changed?"

Postgres already knows the answer through logical replication.

The problem is that most CDC setups assume you're going to put those changes into Kafka.

If you already run Kafka, that's fine.

If you don't, you're adding a fairly large piece of infrastructure just to move changes from one database to another system.

That's the gap I wanted to explore.

What VentStream does

The basic idea is pretty simple:

        Database
           │
           │ CDC
           ▼
      ┌───────────┐
      │ VentStream│
      └─────┬─────┘
            │
            ▼
     Search / Cache / DB
Enter fullscreen mode Exit fullscreen mode

VentStream connects directly to the source's native change feed, keeps the state it needs for joins, and writes to the destination.

No Kafka required.

Currently, the sources include:

Source Change mechanism
PostgreSQL / Supabase Logical replication (pgoutput)
MySQL / MariaDB Row-based binlog
MongoDB Change streams
Neo4j 5.17+ Enterprise CDC log
Kafka / Redpanda Debezium envelopes or raw topics

And the sinks include:

Sink How VentStream writes
OpenSearch / Elasticsearch Bulk API + external versioning
Meilisearch Task-confirmed writes
Redis Keyspace or view materialization
SurrealDB Native RPC + real record IDs

The important part for me is that this is all one process.

A real Postgres → OpenSearch config

Here's what a Postgres → OpenSearch configuration currently looks like:

schema_version: 1

roles: [cdc]

source:
  kind: postgres
  postgres:
    host_ref: env:VS_PG_HOST
    port: 5432
    user_ref: env:VS_PG_USER
    password_ref: env:VS_PG_PASSWORD
    database_ref: env:VS_PG_DATABASE
    publication_ref: env:VS_PG_PUBLICATION
    slot_ref: env:VS_PG_SLOT

    bootstrap:
      mode: snapshot
      chunk_size: 10000

sink:
  kind: opensearch
  opensearch:
    endpoint_ref: env:VS_OS_ENDPOINT
    index_routing:
      strategy: by_output_relation

runtime:
  health_listen: 127.0.0.1:4043
  dlq_path: ./ventstream-state/dlq.jsonl

  memory:
    enabled: true
    budget_bytes: 268435456

  joins:
    state_dir: ./ventstream-state/joins
Enter fullscreen mode Exit fullscreen mode

There are a few things I care about here.

Secrets are referenced through environment variables.

Memory has an explicit budget.

And the dead-letter queue is just a file.

If something goes wrong, I can actually open it and inspect it.

The three decisions that mattered most

Removing Kafka isn't particularly interesting by itself.

The harder question is:

Can you remove Kafka without losing the reliability properties people use it for?

There were three areas where this mattered a lot.

1. Don't move the cursor until the sink has the data

This is probably the most important part of the system.

With Postgres logical replication, the replication slot has a confirmed_flush_lsn.

Once Postgres knows you've processed everything up to a particular LSN, it can eventually remove the older WAL.

So you really don't want to say:

"I've read this change, so I'm done."

Reading it isn't enough.

VentStream only advances the replication watermark after the sink has acknowledged the write.

It also tracks a contiguous sequence of successfully processed changes.

For example:

Batch 1 ✓
Batch 2 ✓
Batch 3 ✓
Batch 4 ✓
Batch 5 ✗
Batch 6 ✓
Enter fullscreen mode Exit fullscreen mode

The watermark stays at batch 4.

It doesn't jump to batch 6 because that would create a hole.

That means if the process dies after batch 6, it can restart from the last confirmed point rather than assuming the missing batch was processed.

I tested this by pushing 250,000 mutations through the pipeline and repeatedly hard-killing the process.

The source and sink ended up with zero drift in those validation runs.

That's the kind of failure case I care about much more than a pretty benchmark number.

2. Do the joins where the change happens

This was the other big reason I wanted to build this.

Suppose we have:

customers
orders
Enter fullscreen mode Exit fullscreen mode

An order might reference a customer:

orders.customer_id → customers.id
Enter fullscreen mode Exit fullscreen mode

If the customer changes, the order document in OpenSearch may also need to change.

With a basic CDC setup, you get something like:

customer changed
       │
       ▼
   CDC event
       │
       ▼
   your consumer
       │
       ├── find affected orders
       ├── fetch customer
       ├── rebuild documents
       └── update OpenSearch
Enter fullscreen mode Exit fullscreen mode

I wanted the CDC engine itself to understand that relationship.

VentStream keeps join state locally and produces the composed document.

So when a customer changes, the affected orders can be recomposed and emitted again.

This also exposed an interesting Postgres detail.

TOAST columns

With pgoutput, unchanged TOAST columns may not be included in an UPDATE.

So if you treat every update as:

"Replace the whole document with whatever was in this event"

you can accidentally wipe out large text fields that weren't part of the update.

VentStream merges the incoming change into the stored row instead.

So an update like:

status = "paid"
Enter fullscreen mode Exit fullscreen mode

doesn't accidentally turn:

description = "a very large text field..."
Enter fullscreen mode Exit fullscreen mode

into null.

These are the kinds of details that don't show up in the basic CDC diagrams but become very important once you're actually running the thing.

3. A bad row shouldn't kill the entire pipeline

This one came from an actual failure mode.

Imagine the pipeline receives a record it can't process.

Maybe the payload isn't a JSON object.

Maybe it doesn't look like a valid CDC event.

Previously, the simplest thing to do was to stop.

But that creates a nasty loop:

bad event
   ↓
pipeline stops
   ↓
restart
   ↓
same bad event
   ↓
pipeline stops
Enter fullscreen mode Exit fullscreen mode

The cursor can't move past it because the event wasn't processed.

So one bad record can effectively brick the pipeline.

The solution is a dead-letter queue.

But there's an important detail.

VentStream writes the event to the DLQ and calls fsync() before advancing the cursor past its LSN.

So the ordering is:

bad event
   ↓
write DLQ
   ↓
fsync
   ↓
advance cursor
Enter fullscreen mode Exit fullscreen mode

Not:

bad event
   ↓
advance cursor
   ↓
write DLQ
Enter fullscreen mode Exit fullscreen mode

The second version has a nasty failure window.

If the machine loses power between those two operations, the source may already have moved past the event while the DLQ entry only existed in the page cache.

Now the event is gone.

There are also failures that I deliberately don't put in the DLQ.

For example, if the source database disappears or the sink can't be reached, that's not a bad row.

That's an environment problem.

The event should be retried.

Otherwise you end up quarantining perfectly valid data just because your network went down.

So, how fast is it?

These aren't polished benchmark-suite numbers.

They're local validation runs on my machine.

I'm sharing them because they helped me decide whether the architecture was actually working.

SurrealDB

200,000 rows with graph edges:

  • Full bootstrap: 11 seconds
  • Peak RSS: 23 MiB
  • RSS while tailing: 8.3 MiB
  • Re-running the bootstrap produced the same 200,010 edges
  • No duplicate edges

Postgres → SQL-denormalized documents

2 million rows:

  • RSS stayed between 24 and 86 MiB
  • Memory remained bounded throughout the run

That's because the memory budget isn't just a configuration value sitting there for decoration.

It actually controls admission.

Neo4j projection

1 million documents plus a 250,000-row cascade:

  • 26–100 MiB RSS

Postgres → OpenSearch

60,000 rows through the complete join path:

  • 0 DLQ entries
  • Exact row-for-row match between the source and index

The main reason the memory usage stays relatively low is architectural.

There isn't a broker sitting in the middle serializing and deserializing every event.

There's no JVM.

And the joins happen once when the data changes instead of once during every full rescan.

But should you actually replace Kafka?

Probably not.

I don't think "Kafka is unnecessary" is a useful conclusion from this.

Kafka is very good at being a distributed log.

You probably want Kafka when:

  • many independent consumers need the same change stream
  • consumers need to replay events independently
  • you need long-term retention
  • the CDC stream is becoming an organisation-wide event backbone
  • multiple teams and applications depend on the same events

That's a different problem.

If your architecture looks like this:

Postgres
   │
   ├── Search index
   └── Redis
Enter fullscreen mode Exit fullscreen mode

a single-process CDC engine can make a lot of sense.

If it looks like this:

                         ┌── Analytics
                         │
Postgres → Kafka →───────┼── Search
                         │
                         ├── Data warehouse
                         │
                         └── 20 other consumers
Enter fullscreen mode Exit fullscreen mode

Kafka is probably the better tool.

They're not mutually exclusive either.

VentStream can consume Debezium-formatted Kafka topics, so you can keep Kafka as the event backbone and use VentStream for the join/materialization part.

Try it

If you want to play with it:

curl -fsSL https://ventstream.dev/install.sh | sh
Enter fullscreen mode Exit fullscreen mode

Or with Docker:

docker run ghcr.io/ventstream/ventstream:<version>
Enter fullscreen mode Exit fullscreen mode

There's also a demo in the repository that starts Postgres, Neo4j, and OpenSearch and streams joined documents between them.

There's a live SurrealDB demo too, showing orders joined with customers.

What's still rough?

Quite a bit.

It's early. The 0.1.x version numbers are there for a reason.

A few things I'm not happy with yet:

  • PostgreSQL → PostgreSQL isn't supported as a sink yet.
  • The DLQ replay behaviour for poison rows needs better documentation.
  • Bootstrap could deduplicate upstream lookups more aggressively.
  • There are still plenty of edge cases that only show up when you run CDC for long enough.

That's also why I'm putting this out there now.

I'd rather have people try it against real systems and tell me where it breaks than pretend it's finished.

If you run VentStream and it does something weird, please open an issue.

Those reports are much more useful to me than another benchmark where everything runs perfectly.

One last thing

The main reason I built this isn't because I think Kafka is bad.

It's because I kept seeing relatively simple data-sync problems turn into relatively large infrastructure projects.

Sometimes you need Kafka.

Sometimes you just need:

database → change feed → transform → destination
Enter fullscreen mode Exit fullscreen mode

And for that second case, I wanted to see how far a single binary could go.

That's what VentStream is trying to be.

Top comments (0)