In my last post, I wrote about the negative control that failed—the test in my fault-injection suite that injects nothing and expects silence. 56 seeded faults passed. The one that seeded nothing screamed.
This post is about the sequel to that, which turned out to be the same lesson wearing different clothes.
The choice
Upstrace started as a DuckDB-only tool—time to add a second warehouse.
Snowflake was the obvious answer. It's what's on the job ads, it's what people ask about, and it would have made a better headline than what I actually did. I could have signed up for a trial, made it work, written "supports Snowflake" in the README, and shipped.
Here's what happens on day 31.
The trial expires. From then on, every commit I push can break the Snowflake path, and I have no way of finding out. The README still says it works. The first person to discover otherwise is a stranger opening an issue.
That isn't a feature. It's a claim with an expiry date.
Postgres runs in GitHub Actions as a service container. Free. On every push. Which means the claim checks itself.
What CI actually asserts
The interesting part isn't that it runs on both. It's what it checks.
- name: Baseline
run: |
upstrace profile
upstrace profile
- name: No false positives on unchanged data
run: upstrace drift --fail-on warning
- name: Inject a unit change
run: |
psql -f fault.sql
dbt run --profiles-dir
upstrace profile
- name: The fault is detected
run: |
if upstrace drift --fail-on critical; then
echo "::error::A unit change was injected and no critical signal was raised."
exit 1
fi
- name: The root cause is the source, not a downstream model
run: |
upstrace rca | tee rca.txt
grep -q "root: orders (source)" rca.txt || exit 1
The first two are the negative control from the last post, promoted from a benchmark scenario to a CI gate. Profile twice over data that did not change, and fail the build if anything at all is raised.
The last one is the check I'd argue hardest for.
When the fault lands, Upstrace raises 22 drift signals. Twenty-one of them are downstream: staging inherited the change, the fact table inherited it, the daily aggregate inherited it. Exactly one names the source.
So if my lineage rule broke tomorrow — if the root-cause logic started blaming the aggregate instead of the source — the "did it detect something" check would still pass. Twenty-two signals would still appear. The build would still be green, and I'd learn nothing.
Only asserting the root by name fails when the thing I actually built stops working.
Which is the same asymmetry as the negative control. "It produced output" is not the property I care about.
The finding reproduced itself
The Postgres demo is a completely different schema. Orders and daily revenue aggregates, not taxi trips. Different column types, different table sizes, different SQL engine. The fault is a 1.6× scale applied to the last 30 days out of 90.
severity model column metric baseline current change days
critical orders amount mean_value 35.2144 56.3430 60.0% 30
high orders amount mean_value 35.1582 42.1915 20.0% -
Same column. Same run. Two rows.
Measured per day, the mean moved 60%. Measured across the whole table, it moved 20%.
The fault touches a third of the history, so a whole-table average dilutes it by roughly two thirds. This is the reason Upstrace profiles per partition rather than per table — on the original dataset, making that change took detection from 44 of 56 scenarios to 55.
I had quietly assumed some of that was an artefact of the taxi data. It isn't. It's arithmetic, and every whole-table metric has it. Running on a second engine with a second schema is what turned an assumption into something I know.
That's a side effect of portability I didn't plan for: a second implementation is a second opinion on your own results.
The part that saved me a week
One implementation note, because I nearly got this wrong.
psycopg and DuckDB's Python API do not agree. execute() returns a cursor rather than something you can fetch from directly. Placeholders are %s rather than ?. My first instinct was to open the codebase and fix the hundred-odd call sites.
Instead I wrapped the driver:
class _PgConnection:
def __init__(self, raw):
self._raw = raw
def execute(self, sql, params=None):
if params is None:
return self._raw.execute(sql)
return self._raw.execute(_qmark_to_pyformat(sql), params)
def __getattr__(self, name):
return getattr(self._raw, name)
About thirty lines, including the ? → %s translation. Every existing call site kept working unchanged.
The alternative would have meant editing every query in the codebase in order to add a second warehouse — and destabilising the DuckDB path, which was the one thing already known to work.
Adapting the driver to the codebase was much cheaper than adapting the codebase to the driver.
Everything else that differs between the two engines — DESCRIBE vs information_schema.columns, how floats round, bulk insert, type names — now lives in a single file behind a seven-method interface. A third warehouse means writing those seven methods and touching nothing else.
What I didn't do
Two things are still DuckDB-only, and I'd rather say so than let someone find out.
The dashboard reads DuckDB directly instead of going through the dialect layer, so on a Postgres project it returns a 503 explaining that. drift, rca and report all work on both — the HTML report gives you the same incident view without a server.
And the fault-injection benchmark itself is DuckDB-only. It's built on read_parquet and reservoir sampling. It's a test fixture, not part of the engine.
Trying it
pip install "upstrace[postgres]"
dialect: postgres
warehouse: postgresql://user:password@host:5432/database
Every command behaves identically after that. There's a complete worked example — dbt project, deterministic 90-day seed, and a fault.sql you can inject to watch root-cause analysis produce a real incident — in postgres-demo/.
Repo: github.com/ashg2099/upstrace
Top comments (0)