Every so often a post reminds me that the most dangerous line of code in a system is the one that looks like it could not possibly be wrong. This week's version: calling .lower() on a string can be a security vulnerability.
If your first reaction is skepticism, mine was too. Lowercasing is the plumbing of programming. We do it to normalize usernames, compare header names, canonicalize domains, and check things against blocklists. It feels like arithmetic.
The problem is that case conversion is not a character-by-character mechanical operation. It is a linguistic one, defined by Unicode, and it has behavior that surprises almost everyone who has not been bitten before.
Case is not symmetric, and not always local
Two examples that break the mental model. Turkish has a dotless i, and correct locale-aware conversion maps between letters differently than English does, which means "the same" string can lowercase into two different results depending on locale settings. And there are characters outside ASCII whose lowercase form is an ASCII character, meaning a string that contains no k at all can become one that does after normalization.
Sit with that second one for a moment, because it is the security-relevant shape. If you validate a string, then normalize it, you have validated something that no longer exists. Your check ran against one value and your system acts on another.
That is the classic time-of-check versus time-of-use bug, except the mutation is not caused by an attacker racing you. It is caused by your own normalization call, quietly doing what the spec says it should do.
I want to be careful not to overstate the specifics here, since the exact behavior depends on language runtime, Unicode version, and locale configuration. The generalizable lesson is what interests me.
The pattern to look for in your own code
Anywhere a string travels through this sequence, there is potential for trouble:
- Accept input.
- Check it against a rule: an allowlist, a blocklist, a comparison with a known value.
- Transform it: lowercase, strip, unescape, decode, or normalize.
- Use it: as a filename, a header, a domain, a database key, a routing decision.
If step three sits between steps two and four, the check is guarding a value the system never actually uses. The fix is not usually to remove the transformation. It is to reorder: canonicalize once, immediately at the boundary, and then validate the canonical form. After that point, the value is settled and nothing downstream may touch it.
This is the same discipline we already accept for HTML escaping and SQL parameterization. Decide on one representation, produce it at the edge, and treat the inside of your system as trusted-by-construction rather than re-cleaned at every stop.
Security comparisons want boring rules
The other thing I have changed my mind about over the years is how much locale-awareness belongs near security decisions.
Human-facing text should be locale-aware. If you are sorting names for a user, or displaying a title, or building a search feature, you want the real linguistic behavior. That is what it is for.
Security comparisons want the opposite. They want a narrow, explicit, boring rule that does the same thing in every environment on every machine. If I am comparing a header name, I want an ASCII-only comparison. If I am matching a token against a list, I want exact bytes, chosen deliberately. If I need case-insensitivity for identifiers, I want the language's dedicated case-folding function rather than the display-oriented one, and I want a comment explaining which one and why.
The general principle: when a function's behavior depends on ambient configuration, do not put it on the path of a decision about access.
Why this keeps happening
Because these functions are marketed to us as trivial. Nobody reviews a .lower(). Nobody writes a test for a lowercase call. There is no linter rule that fires, and if there were, we would probably silence it.
Meanwhile the functions themselves are doing something genuinely complex on behalf of a global user base, and doing it well. Unicode is not the villain in this story. The villain is our assumption that "normalize the string" is a single obvious operation with one obvious result.
What I take away is smaller and more useful than "be afraid of Unicode." It is this: find the places where your code validates a value and then changes it. Those are the seams. In my experience, they are also where the interesting bugs have been hiding all along, patient and well-tested and completely wrong.
Top comments (0)