Our tenant isolation was correct. Two tenants, separate realms, separate apps,
RLS confirmed both ways — the victim's token returns [] for the storm
tenant's table, and a script asserts it on every run. Nobody could read anybody
else's rows.
Then we pointed a load storm at one of them and the other one fell over.
The product tour, silent by design. Full-resolution video.
What we measured
Tenant A issued heavy 200-row reads at 60 concurrent. Tenant B issued the light
100-row read a grid does when you open it. Same stack, same database, 2,000
rows each.
| phase | p50 | p99 | requests |
|---|---|---|---|
| baseline | 3ms | 31ms | 200 |
| under storm | 3,194ms | 10,007ms | 200 |
| recovered | 3ms | 5ms | 100 |
Tenant B degraded 322× against a published bound of 2×, and 10.5% of its
requests were refused outright. The 10,007ms p99 is a timeout ceiling, not a
measurement — we never found out how slow it really was.
These are single-machine figures: Docker, the harness and the browser all on
one box. That is not a production result and we are not going to present it as
one. It is the right shape of measurement for the question — does one tenant's
load reach another tenant — and the wrong shape for "how fast is Appdor".
The part that made it obvious this was contention and not a bug: the storm
tenant's own throughput collapsed to 12.7 req/s. The instance did not throttle
one tenant in favour of another. It fell over for both. And recovery was
immediate — 5ms p99 the moment the storm stopped.
The mechanisms existed and nothing called them
This is the uncomfortable part. The fairness module was already in the tree.
It had tests. The only function of it the running product ever reached was the
one that publishes the promise to the capability catalogue.
A module that is imported by nothing on the request path is documentation with
a test suite. We had shipped the paperwork.
The fix was in-flight caps, not rate limits
The controller that now sits in front of PostgREST caps concurrent in-flight
work per tenant. That distinction is the whole result:
| before | after | |
|---|---|---|
| victim p99 under storm | 10,007ms | 25ms |
| degradation | 322× | well inside the 2× bound |
| victim requests refused | 10.5% | 0% |
The "after" column is the 2026-09-07 re-run against the current gateway: 3,000
storm requests at 60 concurrent, two live tenants with tokens minted for
separate users. Baseline p99 28ms, under storm 25ms, recovered 15ms.
The defensible claim is "well inside the bound", not a specific ratio, and
that is deliberate. The same harness has returned 0.50×, 0.14×, 0.73× and 0.89×
across runs of unchanged code, because the ratio is computed against whichever
baseline the first phase happened to measure — and a cold stack measured 200ms
p99 where a warm one measured 44ms. A sub-1.0 ratio is evidence the baseline is
still warming, not evidence that a storm makes the victim faster. Quoting the
prettiest of those numbers would be the same species of mistake as the one this
post is about.
The storm's request rate was modest. Its concurrent count was the entire
connection pool. A token bucket would not have fixed this, and we can show that
rather than assert it: at the shipped 200/s limit the rate limiter absorbed
2,308 refusals; raising it to 5,000/s moved every one of those to the queue and
changed the outcome not at all — 85% shed either way. When you can move a
mechanism's threshold by 25× and the result does not care, that mechanism is
not the one doing the work.
A wider queue made it worse
We tried a 200-deep per-tenant queue on the theory that shedding less is
kinder. It converted refusals into waiting: the storm took 4.9 minutes instead
of 7 seconds, and the victim was still at 2.41× baseline when the run ended.
It failed the recovery check.
Refusing a request in 1ms is a better answer than serving it in 10 seconds.
Queues are how you turn a load problem into a latency problem and then lose
track of it.
What we still cannot claim
The criterion this was measured against is still marked unmet in our own
audit, and that is the most useful sentence in this post.
It asks for a sustained automation event storm and a public API burst,
simultaneously. The harness drives the burst only. So the measurement supports
a narrower claim than the criterion makes, and the honest version is "we fixed
the case we can drive" rather than "we meet the bar we set ourselves". The
audit row says so in the same words; we would rather you read it from us.
The other thing the ratio cannot tell you is why. A failure here says the
promise broke, not which mechanism broke it — that is what the per-tenant
gauges are for. A fairness number is a smoke alarm, not a diagnosis.
Why this is in a spreadsheet-database you run yourself
Appdor runs as one container you start yourself, so "multi-tenant" is not an
abstraction our SaaS operator worries about on your behalf — it is your
Postgres, your pool, and your two departments. The isolation you get from RLS
is about reading. Nothing in RLS stops one tenant from consuming the resource
every other tenant needs, and the failure looks like a slow app, not like a
security incident, which is why it survives so long.
The storm is a script in the repo:
npm run test:fairness
It fails loudly, in the direction of the number above, if the controller ever
comes unwired again — which is the only reason we trust it now, having already
been wrong once about a module that existed.
Originally published at appdor.com. I build Appdor, so weigh accordingly.
Top comments (0)