I spent a few days building a repo about multi-tenant retrieval. One knowledge base, many customers, and one promise: a customer can never retrieve another customer's documents. The filter runs before the candidate set is scored, the guard re-checks every chunk on the way out, every query lands in an audit log.
Then I ran an adversarial pass whose only job was to break the promise rather than confirm it. It broke it in four lines of setup, and the hole was in the function I had written specifically to prevent identity confusion.
The line that was supposed to be boring
Entitlement is compared on tenant identifiers, so identifiers get canonicalised once, at construction, and compared exactly afterwards:
def normalise_id(value: str) -> str:
return value.strip().casefold()
The reasoning felt obvious. " ACME " and acme are the same customer, somebody will paste one with a trailing space eventually, and a fence that says "access denied" because of a space is a support ticket. casefold() rather than lower() because casefold is the Unicode-correct one, the one you are told to use for caseless comparison.
That is exactly why it is wrong here.
The tenant that was two tenants
The adversary created a customer called Straße-Werke and gave a completely unrelated principal a grant on strasse-werke. The unrelated principal read the documents.
>>> "Straße-Werke".casefold() == "strasse-werke".casefold()
True
casefold() is designed for caseless matching, and caseless matching is deliberately many-to-one. It is not a case conversion, it is a mapping onto a common form, and several of those mappings collapse characters that are not case variants of each other at all:
>>> "ß".casefold() # one character becomes two
'ss'
>>> "K".casefold() # KELVIN SIGN
'k'
>>> "fi".casefold() # LATIN SMALL LIGATURE FI
'fi'
>>> "ς".casefold() == "σ".casefold() # final sigma
True
Every one of those is a merge. Feed casefold() two identifiers that an upstream registry considers distinct, and you get one entitlement key. Two customers become one customer.
Why this is worse than a normal access bug
Nothing failed. That is the part worth sitting with.
The filter ran correctly, against a key that was wrong. The candidate set was built correctly, from a key that was wrong. My guard, the deliberate second check that re-validates every chunk before it leaves, calls the same entitlement function, so it agreed. The audit log recorded a completely ordinary query:
QUERY principal=svc-partner allow=[strasse-werke/*/*] deny=[] returned=3
There is no exception, no anomaly, no unusual pattern. A reviewer reading that line sees a principal reading documents it is entitled to read, because by the time anything is logged, the two tenants are already the same tenant. Every layer of defence in depth inherits the same wrong key, so depth buys you nothing. The mistake happened before the first layer ran.
Then the same bug again, wearing different clothes
Having fixed the fold, I still had strip(). str.strip() with no argument removes all Unicode whitespace, not just the ASCII kind:
>>> "kronos\u3000".strip() == "kronos" # IDEOGRAPHIC SPACE
True
The same applies to U+00A0, U+2007, U+1680, U+205F, U+0085 and U+2028. So a registry that enforces uniqueness on the raw string happily accepts kronos and kronos plus an ideographic space as two different customers, and my fence folds them into one.
The detail that makes this a good trap: zero-width space and the other invisible characters that people usually test for, U+200B, the soft hyphen, the byte order mark, are not stripped. So a test suite that covers "invisible characters" in the obvious way passes while the actual merge path stays open.
Two fixes I tried and threw away
Use lower() instead of casefold(). It reads like the conservative choice. It does not help:
>>> "K".lower()
'k'
The Kelvin sign folds under lower() too, because that mapping is simple case conversion, not full folding. Swapping the function narrows the hole without closing it, which is the worst outcome available: it feels fixed.
Reject non-ASCII identifiers at construction. This closes it, and it is wrong for a different reason. It refuses legitimate identifiers from most of the world in order to fix a bug in my normalisation, and it hides the actual rule behind a character-set restriction that has nothing to do with the problem.
The fix
Canonicalise as little as possible, and refuse anything that would need canonicalising to be safe:
ASCII_FOLD = str.maketrans(ascii_uppercase, ascii_lowercase)
ASCII_WHITESPACE = " \t\n\r\v\f"
def normalise_id(value: str) -> str:
"""Trim ASCII whitespace, fold ASCII case, and nothing else."""
return value.strip(ASCII_WHITESPACE).translate(ASCII_FOLD)
Straße-Werke and strasse-werke now stay two tenants. A tenant named with non-ASCII characters keeps working, and keeps its identity.
Then the second half, which matters more than the first: anything still carrying whitespace or a non-printable character after the trim is refused at construction, naming the offending code point in the error. Not folded onto its neighbour, not silently accepted. A tenant id with an ideographic space in the middle of it is a data problem upstream, and the honest thing a fence can do is say so out loud instead of guessing which neighbour it meant.
The general shape of it
The lesson generalises past Unicode, and it is the reason I am writing this down rather than quietly pushing the patch.
When a normalised value becomes a security key, normalisation is part of the attack surface. Any many-to-one transform applied to an identity is a merge operation, and a merge between two identities is a privilege grant. Case folding, accent stripping, whitespace collapsing, punctuation removal, homoglyph mapping, lowercasing an email local part: all of them are helpful in a search box and all of them are dangerous the moment the output is compared for authorisation.
Two questions worth asking of any such function:
- Can two inputs that the system elsewhere considers distinct produce the same output?
- If they can, which layer is supposed to notice? If the answer is "the layer that compares the outputs", there is no such layer.
The reason I found this at all is that the adversarial pass was told to breach the fence, not to test it. A test written by the person who wrote the code asks "does it do what I meant". A test written to break it asks "what did I mean that was wrong". Those find different bugs, and the second kind is the kind that reaches production.
Why the repo concentrates risk there on purpose
The repo's actual argument is older and simpler than the Unicode story. Most multi-tenant retrieval filters after the ranking, which means a leak needs one caller, one cache, one new endpoint to forget. Filtering before the candidate set is scored moves all of that risk into a single function.
Which is precisely how I ended up here. Concentrating the risk is the right trade, and the bill it comes with is that the one function you concentrated it into is now the only thing worth attacking. That is why it carries 141 adversarial tests, and why four of them are about a character that is not a case variant of anything.
The repo is at github.com/vinimabreu/tenant-fence: the fence, the deliberately wrong version kept next to it so the suite can demonstrate the leak rather than describe it, and the tests that would have caught this on day one.
Top comments (1)
The ASCII-only fold closes this specific collision but it quietly swaps in the mirror bug — now Straße-Werke and strasse-werke resolve to different tenants, so a customer who happens to type the folded form lands in an empty tenant instead of leaking into someone else's. The thing that actually bit you isn't casefold, it's deriving a security key from a lossy transform of a display string at all; every folding scheme has some many-to-one corner, you just picked the one with German street names in it. I'd mint an opaque tenant id at creation and keep the human string as a display attribute, so "are these two the same tenant" becomes an alias you explicitly register rather than whatever Unicode decides that week. And if you keep a folded key anyway, put a unique index on the folded value — then the second strasse-werke fails loudly at write time instead of silently sharing an entitlement, which is the only reason the whole scenario gets off the ground.