DEV Community: Mujib

Building a Kafka-free CDC Pipeline in Go (and the DDL bug that almost broke my brain)

Mujib — Sat, 18 Jul 2026 05:52:43 +0000

So I've been obsessed with Postgres internals for a while now - WAL, replication slots, MVCC, all of it. I built pgstream a while back just to read the WAL and stream changes in real time, and it kind of took off (34 stars, got mentioned on Golang News Twitter which honestly made my whole week). But pgstream was just a reader. It didn't actually do anything with the data besides print it or forward it raw.

So I started building Rift. The idea was simple: I wanted a single Go binary that reads Postgres WAL changes and ships them to wherever you want - a webhook, another Postgres database, Redis - without needing Kafka, Zookeeper, Debezium, or any of that JVM-based infra that honestly feels like overkill for 90% of side projects and even a lot of production use cases.

Why not just use Debezium?

Look, Debezium is great. It's battle-tested, it's what actual companies run But spinning up Kafka Connect just to sync a couple tables between two Postgres instances feels like using a shipping container to move a backpack. I wanted something you could go install and run in like 30 seconds, with one yaml file.

So that's what Rift is. Here's roughly what it does:

~Reads the WAL using logical replication (pglogrepl + pgx, no C bindings, no cgo headaches)
~Decodes INSERT/UPDATE/DELETE events
~Optionally filters/transforms them with a JS script (using goja, so no need to shell out to Node)
~Ships them to one or more destinations - webhook, Postgres, Redis
~If a destination is down, events get queued to disk (BoltDB) instead of getting dropped or blocking everything

That last part matters a lot more than I expected once I actually started testing it.

The DDL bug that ate my weekend

Once the basic INSERT/UPDATE/DELETE pipeline was working, I wanted to go further — track schema changes too. Like if someone runs ALTER TABLE users ADD COLUMN city TEXT on the source db, Rift should know about it, not just silently break next time it tries to sync a row with a new column it's never seen.

The way I did this was with a Postgres event trigger. Basically:

Rift installs a function + event trigger in the source database
Any DDL command (CREATE TABLE, ALTER TABLE, CREATE INDEX, whatever) fires the trigger
The trigger writes a row into an internal table called rift_ddl_log
Since rift_ddl_log is a normal table, its INSERTs show up in the WAL too
Rift picks those up and forwards them to destinations as DDL events

Sounds clean right? Except I completely missed something obvious: the INSERT into rift_ddl_log is itself a WAL event, and Rift was treating it exactly like any other table's insert - meaning it tried to sync rift_ddl_log rows into my destination databases too. Which obviously don't have a rift_ddl_log table. So I'd get:

ERROR: relation "rift_ddl_log" does not exist

Over and over. Every 5 seconds. Forever. Because the failed event kept getting shoved into the disk queue and retried.

My first fix attempt was dumb - just checking if event.Table == "rift_ddl_log" and skipping it. Didn't work. Turns out the table name coming through was schema-qualified in some paths (public.rift_ddl_log) and not in others, so a strict equality check missed half the cases. Took me embarrassingly long to figure that out because I was staring at the filter logic instead of actually logging what event.Table looked like at the point it failed.

Fix ended up being a small isInternalTable() helper that checks both exact match and suffix match, so it doesn't matter if the schema prefix is there or not:

func isInternalTable(table string) bool {
    internal := []string{"rift_ddl_log", "public.rift_ddl_log"}
    for _, t := range internal {
        if strings.EqualFold(table, t) || strings.HasSuffix(table, t) {
            return true
        }
    }
    return false
}

Once that was in, rift_ddl_log inserts got filtered out cleanly ([FILTERED] INSERT rift_ddl_log in the logs now, instead of an infinite error loop), while actual schema changes like CREATE TABLE, ALTER TABLE, CREATE INDEX still got captured and forwarded correctly.

Honestly this was a good reminder that "just add a string check" bugs are rarely actually about the check - they're about not knowing exactly what shape your data is in at that point in the pipeline. Should've logged the raw value first instead of guessing.

The other thing I learned: DDL isn't the same as schema replication

While testing this, I hit something that made me realize a gap in the whole design. I inserted a row into a table Rift didn't know about yet, and got:

error sending to analytics-db: ERROR: relation "users" does not exist

At first I thought "wait, didn't the DDL tracking just handle this?" But no — DDL tracking detects and forwards schema changes as metadata (like "hey, someone ran CREATE TABLE users"), it doesn't automatically apply that DDL to the destination database. Those are two very different things. Rift knows the schema changed, but it doesn't (yet) go run CREATE TABLE users (...) on the destination for you.

Right now that's a real limitation I'm noting honestly in the release notes instead of pretending it's not there. Auto-applying DDL to Postgres destinations is genuinely tricky - you have to worry about column type differences, existing data conflicts, permissions, and probably a dozen edge cases I haven't thought of yet. It's on my list for a future version, but I'd rather ship something honest than something that pretends to do more than it does.

Making it feel like an actual tool

For the first while I was just running everything with go run main.go, which is fine for testing but feels pretty amateur if you're trying to get someone else to actually use your project. So I rebuilt the CLI using Cobra — now it's:

rift run
rift run -c custom-config.yaml
rift version

instead of a raw script. go install github.com/mujib77/rift@latest and you're done - no cloning, no build step, it just becomes a command on your machine like git or docker. Small change but it genuinely changes how the project feels, both to use and honestly just to work on.

Where it's at now

Rift can currently:

-Stream INSERT/UPDATE/DELETE from Postgres to webhook, Postgres, or Redis destinations
-Filter/transform events with a JS filter script
-Track DDL changes via event triggers (with the filtering bug above, finally, actually fixed)
-Survive destination downtime using an embedded disk queue, no external broker needed
-Run as a proper installed CLI binary

It's still early - 1 star as of writing this, so if you're reading this and it sounds useful, go kick the tires and tell me what breaks. That's genuinely the fastest way I learn what's actually wrong with it versus what I assume is wrong with it.

Repo's here: https://github.com/mujib77/rift

If you've built something similar or have thoughts on how you'd handle the DDL-to-destination sync problem, I'd love to hear it in the comments - still figuring that part out.https://github.com/mujib77

I spent a week researching why devs hate Supabase, LangChain, PostHog & Neon (and what I found)

Mujib — Mon, 01 Jun 2026 14:16:02 +0000

If you hang out on Reddit or Hacker News long enough, you’ll notice a pattern. Every few days someone posts:

“Supabase RLS is driving me insane”
“LangChain is death by a thousand abstractions”
“Neon cold starts are killing my API response times”
“PostHog autocapture is just noise”

I’ve used all four tools. And yeah, I’ve felt the pain. But I wanted to know: how bad is it really? Not just vibes, but actual hours lost. So I did something stupid for a student with free time.

I researched it Properly.

I read through GitHub issues, Reddit threads, Hacker News comments, and Discord rants. I even used AI to help me aggregate and categorize the complaints. (Yes, AI helped - I’m not hiding that I’m a student, not a journalistic purist.)

Here’s what I learned -

The short version:

Add it up for a team using all four? That’s 40+ hours a week of non-feature work. Basically one engineer just fighting tools.

And here’s the kicker: a lot of these problems already have solutions. People just don’t know about them.

Supabase RLS — the biggest trap
Supabase markets itself as “build in a weekend” But then you hit Row Level Security.

“I know I should learn Postgres better, but I feel misled.” — some Reddit user, speaking for thousands.

RLS policies aren't just access rules. They're PostgreSQL expressions that run inline with every query. When they fail, you get a generic “permission denied” and zero clue why.

One engineer said debugging RLS means opening nine different log tabs and manually correlating timestamps. Nine tabs For one bug.

LangChain — why I’m glad I didn’t build my AI agent with it
I almost used LangChain for Kommit (my AI commit CLI). Thank god I didn’t.

People describe debugging LangChain as “peeling an onion of abstractions until you cry” A simple API call becomes: PromptTemplate → OutputParser → Chain → AgentExecutor → LCEL pipe operators.

One team removed LangChain entirely after a year. They said: “We spent as much time understanding LangChain internals as building features”

The worst part? Hidden LLM calls. One developer reported an agent that called the wrong tool 14 times in a loop and burned $40 of API credits before returning gibberish.

My take: Use the raw OpenAI/Anthropic SDK unless you really need agent orchestration. LangGraph is better but still complex.

PostHog — the data cleanup tax
PostHog’s autocapture is both a blessing and a curse It tracks everything Clicks, rageclicks, form submissions, you name it.

But then you spend hours filtering out the noise

One GitHub issue I found: a developer was trying to stop $rageclick events from firing when users click up/down arrows on a date picker. Normal behavior Not a rageclick

PostHog has a “drop events” transformation, but that’s reactive. You’re cleaning up after the mess

What works: Disable autocapture from day one and implement explicit event tracking with a schema. Boring, but saves weekends

Neon — serverless is cool until it sleeps
Neon’s scale-to-zero saves money. But when your database wakes up, you get a 300-700ms cold start. On the free tier, it’s even worse.

And the connection pooling? It uses PgBouncer in transaction mode. That means SET search_path and prepared statements break. Your app works locally (direct connection) but fails in production (pooled connection). Fun times.

Fix: Set a minimum compute size to keep it warm or accept the cold start. Also maintain two connection strings - one pooled, one direct for migrations.

The pattern I didn’t expect
Almost every problem falls into three buckets:

Leaky abstractions - the tool hides complexity until it breaks, then you need to understand the internals anyway.

Configuration fatigue - too many env vars, YAML files, and dashboard settings that interact in weird ways.

Data/schema drift - changes made in dashboards instead of migrations, or events accumulating without governance.

None of this is new. Joel Spolsky wrote about leaky abstractions in 2002. But somehow every new tool repeats the same mistakes.

What I’m doing with this
I’m not writing this to bash the tools. They’re all well-engineered and solve real problems.

But as a student looking for an internship, I wanted to understand: what do engineers actually hate? Because if I can build something that fixes a small piece of that, maybe I can get paid to build it instead of fighting it.

I already have a few ideas. But first, I’m turning this research into a small series on Dev.to. Next post will be: “Three unsolved problems I found in the research (and what I’m building next).”

One last thing
If you’re using Supabase, LangChain, PostHog, or Neon - what’s your biggest pain point? I’m genuinely curious. I might try to build a fix for one of them.

Drop a comment or Find me on GitHub or Dev.to

– Mujib, sophomore who spends too much time in the terminal.

How I Built a Real-Time PostgreSQL WAL Reader in GO

Mujib — Sat, 09 May 2026 08:41:40 +0000

Most developers use PostgreSQL every day without knowing
what happens under the hood when they run an INSERT. I
wanted to find out. So I built pgstream — a tool that
reads every database change in real time directly from
PostgreSQL's internal log.

What is the WAL?

Every time you insert, update, or delete a row, Postgres
doesn't immediately write to the actual data files. It
first writes the change to a file called the WAL — Write
Ahead Log. This is how Postgres guarantees nothing gets
lost if the server crashes mid-write.

I had read about WAL in docs before but never really
understood why it existed until I started building this.
It clicked when I realized — the WAL is basically a
journal. Postgres writes every intention down first,
then acts on it.

Logical Replication and pgoutput

WAL records are stored in raw binary. You can't just
read them like a text file. Postgres has a built-in
plugin called pgoutput that decodes that binary into
readable messages — INSERT, UPDATE, DELETE — with the
actual row data included.

To read from the WAL stream your program creates a
replication slot. Think of it as a bookmark. Postgres
tracks your position in the stream and holds WAL records
until your reader has consumed them. This means you can
stop and restart your reader without missing any changes.

Building pgstream

I built pgstream in Go using the pglogrepl library which
handles the low level replication protocol. The project
has four main parts:

Connector — opens a replication connection to Postgres
and creates the slot.

Decoder — this was the hardest part. It reads raw bytes
coming from Postgres and figures out what each message
means. Postgres sends different message types — Relation
messages that describe table schemas, Insert messages,
Update messages, Delete messages, and keepalive
heartbeats. The decoder has to handle all of them
correctly.

The tricky part was the Relation message. Before Postgres
sends an INSERT event it sends a Relation message
describing the table structure — column names, types.
You have to store that and reference it later when
decoding the actual row data. Without this you get raw
bytes with no idea which column is which.

Handler — takes the decoded event and prints it cleanly
to the terminal.

The part that surprised me

What surprised me most was how fast it is. The moment
you run an INSERT in pgAdmin the change appears in the
terminal almost instantly. There is basically no delay.

I knew replication was fast in theory but seeing it
happen live made it real. This is the same mechanism
companies use to sync databases across regions in
real time.

What I learned

I understood Postgres as a database before this. Now I
understand it as a system. WAL, logical decoding,
replication slots, LSN — these are the internals that
make Postgres reliable at scale.

Go was also new to me coming from JavaScript and C.
The concurrency model with goroutines and channels
is genuinely different from anything I had used before.

Try it yourself

pgstream is open source.
github.com/mujib77/pgstream

If you want to understand what's happening inside your
Postgres database — clone it, run it, and watch your
changes stream in real time.