DEV Community

Cover image for Weird Unicode Behaviors
Philippe Arteau
Philippe Arteau

Posted on • Updated on • Originally published at

Weird Unicode Behaviors

So you know about Unicode codepoint and encoding (UTF-8, UTF-16), but are you aware that few standard conversions have surprising outcomes? In this short post, I'll list some of the most surprising behaviors.

Case Mapping

Case mapping is the behavior behind the uppercase and lowercase functions in your favorite language. Unexpected behaviors can sometimes lead to bugs, some of them affecting software security.

While the strings “go\u017Fecure” and “gosecure” are not equal, a code that applies the uppercase transformation to both strings could mistakenly interpret both strings as being equal.

Here is a demonstration in Python.

>>> "GO\u017FECURE" == "GOSECURE"
>>> "GO\u017FECURE".upper() == "GOSECURE"

Enter fullscreen mode Exit fullscreen mode

The same behavior applies to Java

$1 ==> true
Enter fullscreen mode Exit fullscreen mode

This behavior occurs because the characters ı (U+0131) and ſ (U+017F) are converted to an ASCII characters as part of Unicode specification. Aside from a few exceptions, you can assume that your language apply these transformation by default.


The purpose of normalization is to simplify expressions to allow matching equal or equivalent “meaning”.

Here is a demonstration using normalization functions in Ruby. The five unicode characters become six unicode characters (ASCII only).

irb(main):003:0> "\u216E\u32CE\uFF0E\u209C\u2134".unicode_normalize(:nfkc)
=> ""
Enter fullscreen mode Exit fullscreen mode

API can sometimes hide those transformations. For example in C#, the class Uri normalize the hostname from URI entered.

> Console.Write(new Uri("https://faceboo\").Host == "");
Enter fullscreen mode Exit fullscreen mode

Here is a list of APIs with such behavior (see Section Auditing Source Code).

Testing your application

If you have an application that susceptible to issues related to Unicode, you can use this simple cheat sheet.
This cheat sheet can be used by developers to build regression test cases to make sure no characters are being misinterpreted.

Unicode Cheat sheet

If you are curious, you can read the full article with a security focus.

Top comments (0)