Filament by Galaxy launched on Product Hunt on November 13 and hit #2 Product of the Day. It's an open-source (Apache 2.0) data movement engine written in Go: you define connections, resources, and pipelines, and it replicates data from databases, APIs, and streams into warehouses, lakes, and other databases — from a terminal CLI, or driven by an AI coding agent.
(Quick disambiguation, because it matters: this is Filament by Galaxy, the data engine. It has nothing to do with FilamentPHP, the Laravel admin framework.)
If you try it, you'll hit one decision before anything else, and it's the decision that determines whether your pipeline is correct or quietly wrong: how each resource is read. Filament gives you three read modes — full, incremental, and cdc — and pairs each with a write mode (replace, append, upsert, merge). This post is the short version of how to choose, including the two traps that bite almost everyone.
The three read modes
full reads every selected row, every run. It's the easiest mode to reason about and the easiest to verify: pair it with replace and the sink is a fresh snapshot of the source. It also sees deletes naturally, because a deleted row simply isn't in the next rebuild. For small tables, nightly reporting rebuilds, and your first week with the tool, this is the right default.
incremental reads only rows past a saved cursor — typically rows whose updated_at is newer than the last run's watermark. It's how you keep a large table fresh every few minutes without re-reading it. The requirements are strict, though: the table needs a primary key, plus a NOT NULL timestamp-style cursor column that genuinely advances on every insert and update. If one code path in your app updates a row without touching that column, that change is invisible forever.
cdc reads the ordered change stream from the database log itself — PostgreSQL logical replication, or the MySQL binlog. Inserts, updates, and deletes all arrive as events, in order. This is the only read mode that sees hard deletes and preserves change order, which is why it's the mode everyone wants. It's also the mode with the most setup attached, which brings us to the traps.
Trap 1: Incremental reads cannot see deletes
This is the one that produces silently-wrong dashboards. A hard delete removes the row — and an incremental read works by reading rows. No row, no event, nothing to propagate. Your destination keeps the deleted row forever, and nothing errors.
So the rule is simple: if deletes must disappear from the destination, you need either full + replace (rebuild and swap) or cdc + merge (apply deletes as they happen). incremental + upsert — the combination most people reach for first — keeps a "current state" table that is current except for deletions. Know that before you ship it, not after someone audits the numbers.
Trap 2: CDC does not run on the local CLI
This one surprises people because the local CLI feels complete: you can install Filament, run a pipeline in about five minutes with no database at all, and replicate PostgreSQL → ClickHouse from your laptop. Then you switch a connection to CDC and the run fails, complaining it can't durably admit replication streams.
That's by design. CDC pipelines need Filament's PostgreSQL-backed datastore to track replication stream state durably. The local CLI context uses SQLite, which isn't enough — so CDC means deploying Filament as a server first, then creating the source as a CDC connection (it's a property of the connection, not a per-table toggle, and a CDC route is CDC for the whole run; you can't mix CDC and standard resources in one pipeline).
CDC also asks things of your source database. On PostgreSQL: logical replication, a publication over the tracked tables, a replication slot, and a primary key on every tracked table — plus REPLICA IDENTITY FULL if updates to large columns need to carry old values. On MySQL: 8.0+, binlog available over TCP (a Unix-socket DSN is rejected for CDC), and binlog retention longer than the longest gap between your runs, or positions expire and the fix is a re-snapshot.
And a few operational truths worth knowing in advance: a replication slot can only be read by one consumer at a time ("slot is active" means something else is holding it); a slot nobody reads makes WAL grow on your source, so when you delete a CDC pipeline, check for orphaned slots; and CDC support on a connector doesn't automatically mean continuous (always-on) support — check the connector's page.
The cheat sheet
| Goal | Read | Write | Watch out for |
|---|---|---|---|
| Rebuild a reporting table nightly | full |
replace |
A failed run restarts that resource from the beginning |
| Keep a big table fresh every few minutes | incremental |
upsert |
Hard deletes are invisible; the cursor column must really advance |
| Keep an audit history of every change | cdc |
append |
Deletes appear as events; rows are not removed |
| Maintain a live replica, deletes included | cdc |
merge |
Needs a primary key; runs on a deployment, not the local CLI |
One more thing in Filament's favor: it validates the read/write pairing before any data moves — source policy, sink capabilities, primary keys, cursor configuration. An invalid combination is rejected up front instead of corrupting data halfway through. When a create command fails validation, read the error; it's usually telling you exactly which requirement is missing.
Going further
I spent a few days reading Filament's official documentation end to end — it's good, but it's spread across dozens of pages, and the decisions above sit scattered between concept guides and how-tos. I turned my notes into a starter pack: a 14-section plain-English tutorial (including a full PostgreSQL → ClickHouse walkthrough and a troubleshooting table for the errors new users actually hit), 12 copy-paste prompts for driving Filament with an AI coding agent safely, and 3 workflow templates. Prefer to start free? The cheat sheet from this article (decision table + both traps + 2 full prompts) is a free download on Payhip: https://payhip.com/b/YG6a3
It's $19 on Payhip if you want the distilled version: https://payhip.com/b/KFJ4e
Disclosure: the pack is an independent guide, written from the official docs and the open-source repo. It is not affiliated with or endorsed by Galaxy. Filament itself is free and open source — the docs are there if you'd rather assemble all of this yourself.
Top comments (0)