In the last piece of my v0.1.0 series I wrote a sentence I was proud of and not yet entitled to: "isolation is structural or it's imaginary." I deferred it. Single-tenant was the honest scope for a first release.
v0.2.0 shipped multi-tenancy on HivePlane. Then I ran an adversarial suite against my own boundary, and the sentence stopped being a thesis and became a bug list.
Seven findings in the table below. Nine if you count the two the table combines. Every one was a place the system inferred tenancy instead of asserting it.
The sentence, tested
| Finding | Where I inferred tenancy | What an attacker did |
|---|---|---|
| D-6 | Workloads referenced platform tools by global key | A tenant's workload couldn't load a tool — keys weren't (id, tenant)
|
| #569 |
worker_id PK was global, not (id, tenant)
|
One tenant hijacked another's worker by reusing its id |
| #540 |
secret.resolve() trusted the ref's declared tenant |
A cross-tenant secret read was a payload field away |
| #562 | Rate limiter keyed on X-Hiveplane-Tenant header |
A spoofable header is not an identity |
| #534 / #535 | Approvals and security events defaulted to default
|
Defense events and approvals landed under the wrong tenant |
| #567 | Fleet incident state wasn't tenant-scoped | One tenant's active incident leaked into another's fleet view |
| #547 | Result-cache writes weren't scoped | A cross-tenant cache write polluted another tenant's results |
Read that table as the argument. Tenant isolation enforced by a header, a model default, or a declared field is imaginary. It must be enforced at the storage primary key and the authenticated principal — never a client-supplied value.
The worker hijack
This is the one I tell people about first. A worker enrolled under tenant A. Tenant B's code submitted a run and addressed it by worker id — worker_id: "worker-7". Because the primary key was worker_id alone, not (worker_id, tenant_id), the lookup succeeded. Tenant B was about to execute a run on tenant A's worker.
No policy said no. The store said yes. The store is the truth.
The fix is one of those changes that's obvious in retrospect and embarrassing in practice: every durable key becomes composite. (worker_id, tenant_id). (workload_id, tenant_id). (tool_id, tenant_id). A read outside the acting tenant now looks like the record does not exist — because in that tenant's world, it doesn't.
tip: If your isolation depends on a
WHERE tenant_id = ?that the application remembers to add, you have an inference, not a boundary. Make the primary key composite and the store cannot return the wrong tenant's row even if the query forgets.
The spoofable header
The rate limiter keyed on the X-Hiveplane-Tenant header. With auth disabled (the local default), the header is trusted plumbing. With auth enabled, a caller who knew another tenant's id could set the header and borrow that tenant's rate budget.
The single place a request becomes a TenantContext is get_tenant_context. With auth on, the authenticated principal's tenant must match the header; only a system principal may select an arbitrary tenant. That's the one assertion that turns a header from identity into hint. Everything before it was inference.
The default tenant is a security boundary you forgot to name
Half the findings were the same bug in a different subsystem: a record with no explicit tenant defaulted to default. Approvals, security events, incident state, cache writes — they all had a tenant field, and when nobody set it, the model filled in default. So a security event raised in tenant A's run landed in tenant A's audit unless something forgot to thread the context — then it landed in default, where the operator of tenant B could see it.
The lesson: a default tenant is not a safe default. It's a silent cross-tenant leak waiting for the one code path that forgets to pass the context. The fix is that an unattributed event doesn't default — it dead-letters and raises. The unattributed counter must be zero.
What I learned
Inference is not enforcement. A WHERE clause the application adds, a header the application trusts, a default the model fills in — each is a guess. Every one of my nine bugs was a guess that was wrong under pressure. The fix is the same everywhere: composite primary keys, TenantScopeError on cross-tenant writes, reads that look like "not found," and an adversarial field test that attacks each boundary instead of testing happy paths. The bug list is the argument that the sentence was right.
The v0.2.0 field test demonstrates the 34 release gates end to end, including per-tenant budget/policy/key isolation with a viewer who cannot approve. Full evidence in the field-test report; the migration guide documents the breaking schema change (forward-only migration 0003, back up first).
References
-
HivePlane —
pip install hiveplane==0.2.0 - Field test report (36/36)
- Migration guide (v0.1.0 → v0.2.0)
- Multi-tenant isolation plan
- Secrets, RBAC & access audit design
Where does your platform infer tenancy instead of enforcing it? Name one place a client-supplied field becomes an identity. That's your next bug.
Top comments (0)