DEV Community: Ashwin Sridhar

How do you actually test for a failure that leaves no trace?

Ashwin Sridhar — Fri, 04 Sep 2026 07:53:30 +0000

Code and results: github.com/ashwin-sridhar/silent-failure-harness

There's a specific kind of bug I have a hard time explaining to people who haven't hit it. Not a crash. Not a stack trace. A test suite that's green, a system that looks "up," and a business event that just — isn't there. Nobody's system logged an error, because from its point of view nothing happened. That's not a metaphor. That's the literal failure mode.

I wanted to know whether the pattern I'd designed around this was actually safe, or just safe-sounding. So I tried to break it on purpose.

What "silent failure" actually means

At-least-once delivery plus a crash at exactly the wrong instant produces a state where the sender believes the event was delivered and the receiver has no record it ever arrived — and both of them are technically correct, from where they're standing. There's no error anywhere in that story. That's what makes it hard to test for: you're not looking for an exception, you're looking for an absence.

The setup

Two variants of a minimal webhook handler, and a way to kill the process on demand at an exact, repeatable point:

Variant A — ack-then-persist
  1. Receive request
  2. Return 200
  3. Write event to the database

Variant B — persist-then-ack
  1. Receive request
  2. Write event to the database (INSERT ... ON CONFLICT DO NOTHING)
  3. Return 200

The crash is a SIGKILL sent to the process at a pinned point right after step 2 in each variant — not "at some point during the request," but deterministically after the ack or after the write, using an env flag. No graceful shutdown. The same kind of failure a host reboot or an OOM kill actually produces.

For each trial: send the event, let the process kill itself, restart it clean, resend the identical event ID — exactly what a provider does when it never gets a 200. Then check the actual source of truth: how many rows exist for that event ID. Not the logs. The row count.

Finding 1: ack-ordering is not a "usually" — it's unconditional

50 trials per variant.

variant	lost	correct	duplicate	/ 50
ack-then-persist	50	0	0	50
persist-then-ack	0	50	0	50

Ack-then-persist lost the event on every single trial. The 200 went out, the process died before the insert ran, and there was no retry to save it — the provider saw a successful delivery and had no reason to send it again. Fifty for fifty, not a tendency.

Persist-then-ack didn't lose one. One trial out of fifty is worth noting honestly rather than ignoring: the crash request itself returned a 200 anyway, meaning the kill didn't land in time to suppress the ack that trial — a timing artifact of the harness, not a case where the retry path saved a loss. It still counted as correct, since the write had already committed either way, but it's not evidence of the same mechanism as the other 49.

Finding 2: the race I expected to see wasn't the race that was actually there

I set up a second test for the dedup mechanism itself — ten concurrent requests, same event ID, fired at once, comparing a naive check-then-insert against an atomic INSERT ... ON CONFLICT DO NOTHING. I expected check-then-insert to sometimes let a duplicate row through.

It never did. Twenty trials, ten concurrent requests each, for both mechanisms:

dedup	exactly 1 row	2+ rows	/ 20
check-then-insert	20	0	20
atomic-conflict	20	0	20

Zero duplicate rows, either way — but that's not the dedup logic being safe. That's event_id TEXT UNIQUE at the schema level doing its job regardless of what the application code above it does. A second insert physically cannot land, no matter how the race plays out.

The race still happened. It just didn't show up as a row — it showed up as an error. Across 200 individual concurrent requests against check-then-insert (20 trials × 10), 181 came back 200 and 19 came back 500. Two goroutines both ran the SELECT, both saw no existing row, both proceeded to INSERT — one won, one hit the unique constraint it never checked for and blew up. Against atomic-conflict, all 200 requests came back 200. Same exact race, same ten-way collision — ON CONFLICT DO NOTHING just absorbed it.

What this actually means for you

Persist before you ack. Not usually — always. Fifty trials, zero exceptions, in either direction.
Put a real UNIQUE constraint on your dedup key at the schema level, regardless of your application logic. It's the thing that actually stopped a duplicate row from ever landing here, independent of whether the code above it was written carefully.
Prefer atomic INSERT ... ON CONFLICT DO NOTHING over check-then-insert — not because it prevents data corruption, the constraint already does that — but because it decides what happens to the losing side of a race. Check-then-insert hands the loser a raw 500. From the sender's side, that looks like a failed delivery, which means another retry, which means another race. Atomic conflict resolves it cleanly on the first attempt.

The part where I admit something

I built this test expecting to catch a data-integrity bug — duplicate rows from a sloppy dedup implementation. The schema-level constraint made that outcome structurally impossible before either mechanism got a chance to prove anything. What the test actually caught was a behavioral difference, not a data one: whether the loser of an inevitable race gets an idempotent-feeling 200 or an ugly exception. That's a more useful thing to know than what I went in looking for, and I wouldn't have found it without running the concurrent version instead of trusting that "no duplicate rows" meant "no race."

The constraint saved the data. It didn't save the response.

Full harness and raw results if you want to run it yourself: github.com/ashwin-sridhar/silent-failure-harness

Does Postgres RLS actually ruin performance? Let’s look at the data.

Ashwin Sridhar — Fri, 29 May 2026 07:45:42 +0000

There's a particular kind of conundrum that has derailed more architectural decisions than I care to count. You know the one - performance optimization. What do you trade off for what and in what scenario and what might it achieve. I had to sit with one of those conundrums recently.

For a multi-tenant platform, I'd chosen Postgres Row-Level Security for tenant isolation — each tenant's rows locked behind a policy, enforced at the database level, no application code involved. Clean, elegant, one less thing to accidentally screw up in your ORM layer. Then I spent a weekend second-guessing myself.

So I did what you should always do before making an architectural decision: I measured it.

What RLS actually does (the one-sentence version)

You attach a policy to a table. Every query against that table gets an invisible WHERE clause appended by Postgres, based on who's asking. A regular app user asking for rows gets only their rows. A superuser bypasses it entirely. That's it. The question is whether that invisible WHERE clause costs you anything meaningful at scale.

The setup

1 million rows in a single table. One tenant owns roughly 10% of them (~100k rows). I ran 50 timed executions per condition, threw away the first 3 as warmup, and measured p50, p95, and p99 latency. (p95 means 95% of your queries finished faster than this number — it's your realistic bad day, not your theoretical worst case.)

Four conditions:

A — superuser,  no index    → RLS completely bypassed
B — app_role,   no index    → RLS active, planner on its own
C — superuser,  with index  → RLS bypassed, index benefit only
D — app_role,   with index  → RLS active + index (this is production)

Three query types: a simple LIMIT 100 fetch, a filtered scan with a second condition, and a full COUNT(*). The idea was to cover the range from "barely touches the table" to "has to read everything."

Finding 1: RLS overhead is basically noise

Compare A and B. Same hardware, same data, same queries — the only difference is whether RLS policies are being evaluated.

At p95, A vs B differs by less than 2% across every query type. On the count query — the heaviest one, which scans the entire table — A clocks in at 73.3ms and B at 74.9ms. That 1.6ms is Postgres evaluating your RLS policy. It is, to use a technical term, nothing.

The performance concern, as it turns out is focussed on the wrong thing. Policy evaluation is not your bottleneck.

Finding 2: The index is doing all the work

Now look at C and D. Same queries, but I've added an index on tenant_id.

The count query drops from ~73ms (A and B, no index) to 2.2ms (C) and 2.7ms (D). That's a 26× speedup. RLS overhead within the indexed conditions? Still less than 25%.

Here's what's happening under the hood. Without the index, Postgres launches a parallel sequential scan — it reads every single row in the table and throws away the ones that don't belong to your tenant. With the index, it goes straight to your tenant's rows and in the COUNT case, never even visits the heap at all.

The EXPLAIN plans make this embarrassingly obvious:

Without index (condition A):

Parallel Seq Scan on jobs
  Filter: (tenant_id = ...)
  Rows Removed by Filter: 299,876

With index (condition D):

Index Only Scan using idx_jobs_tenant_id
  Index Cond: (tenant_id = ...)
  Heap Fetches: 0

299,876 rows examined vs 0 heap fetches. The index doesn't just help — it changes the shape of the query entirely.

The nuance nobody talks about: secondary predicates

Here's where it gets interesting. The filtered query — which adds a second condition on top of the tenant filter — barely improves with the index. About 42ms without, about 40ms with. A 5% improvement where the count query got 26×.

Why? Because the second predicate forces Postgres to visit the actual heap rows to check the condition. The bitmap index scan narrows the candidate set using the tenant index, but it still has to physically read all ~11,000 heap blocks to evaluate the second filter. You can't index-only scan your way out of a heap fetch.

The fix, if this matters for you, is a composite index: (tenant_id, your_second_column). That lets the planner push both conditions into the index scan and skip the heap entirely. I didn't test that here — that's a follow-up post — but the query plans point directly at it.

What this actually means for you

Three things you can take away from this:

1. Index your tenant_id column. Not optional. Without it, every aggregation query scans your entire table regardless of how tight your RLS policy is. With it, your database goes from doing 300k wasted row evaluations to zero.

2. Stop worrying about RLS overhead. The policy evaluation cost is real but it's measured in microseconds. The architectural benefits — tenant isolation enforced at the database level, impossible to accidentally leak rows from a missing WHERE clause in your application code — are worth far more than 1.5ms.

3. Watch your secondary predicates. If your common queries filter on more than just tenant_id, think about whether a composite index makes sense. The planner is smart but it can only use what you give it.

The part where I admit something

I'll be honest — I already knew the likely outcome before running this. The Postgres community broadly understands that RLS overhead is index-shaped, not policy-shaped. But "broadly understood" and "here are actual numbers from a real table at 1M rows" are different things.

One caveat worth naming: this benchmark tests a simple single-condition policy — the kind that covers most multi-tenant SaaS use cases. Complex policies with subqueries or permission table joins are a different story, and not one this experiment speaks to. That's a follow-up for another day.

Measure things. Then decide.