UCE-8: fitting the world's living scripts into two bytes

Erdene-Ochir Sh — Sun, 09 Aug 2026 14:09:36 +0000

UCE-8 — Unicode 3-byte variable-width Compact Encoding — is a character
encoding sized for the world's living scripts, reaching 128 × 128 × 68 =
1,114,112 characters in at most three bytes.

UTF-8 charges most of the world's living languages three bytes per character.
UCE-8 charges two. This is how it does that, what it measurably saves, and
what it costs.

Every figure in this article was produced by running
verify.py against the reference implementation,
uce_hibrid.py. Reviewer notes on the first draft are
kept in reviewer notes.

The problem

In the text we use every day, characters are stored with a length of 1–4 bytes.

UTF-8:

English-language text = 1 byte
European languages + Middle East = 2 bytes
Asian and African languages = 3 bytes
Languages that have fallen out of use = 4 bytes

UCE-8, instead:

English-language text = 1 byte
Every language alive and in use today = 2 bytes
Languages that have fallen out of use = 3 bytes

Which tier a script lands in was decided by Unicode's code-point ordering, not
by how many people write in it. That is why Devanagari, Thai, Ethiopic and
Chinese — between them the everyday scripts of billions — sit in UTF-8's
expensive three-byte tier.

How it works

1 byte = 8 bits = 256 possible values. Call the first 128 the low byte and
the last 128 the high byte. The high bit is the length signal: a high
byte means "another byte follows," a low byte means "the character ends
here." Decoding stops the instant a low byte appears — no length field, no
lookahead.

byte pattern	length
low	1 byte
high, low	2 bytes
high, high, low	3 bytes

That gives a self-terminating, three-tier scheme:

tier	byte pattern	capacity	covers
1	`0xxxxxxx`	128	plain ASCII
2	`1xxxxxxx` `0xxxxxxx`	8,704	68 pages, 128 characters each
3	`1xxxxxxx` `1xxxxxxx` `0xxxxxxx`	1,114,112	every remaining code point

The two positions have names, and they are just the high/low split above seen
from the other side:

A lead byte is a high byte (0x80–0xFF). It always means "another follows," and it carries the character's slot — which of a page's 128 entries this is.
A trail byte is a low byte (0x00–0x7F). It terminates the character, and it is the page index. Of its 128 possible values, 68 are usable (see below), so tier 2 reaches 128 × 68 = 8,704 characters in 2 bytes where UTF-8 needs 3.

Note the order, because it inverts the usual arrangement: the second byte
says which script you are in, and the first says which letter of it. UTF-8
puts the identifying information in the leading byte; UCE-8 puts it in the
terminator. Devanagari न (U+0928) encodes as A8 35:

A8  =  0x80 | 40   lead byte  — slot 40 of the page
35  =  page index  trail byte — the page holding U+0900–U+097F

Hold the lead byte at A8 and change only the trail byte, and the same slot
number lands in a different script each time:

bytes	code point	character
`A8 35`	U+0928	न Devanagari NA
`A8 45`	U+0E28	ศ Thai SO SALA
`A8 33`	U+0628	ب Arabic BEH

That is the whole addressing scheme: the trail byte chooses the page, the lead
byte chooses the entry.

The tiers fit Unicode exactly. Tier 3's capacity is 128 × 128 × 68 =
1,114,112. Unicode's total code space is 17 × 65,536 = 1,114,112. The same
number. UCE-8 covers all of Unicode with no waste and nothing spare — and
encoding then decoding all 1,114,112 code points round-trips with zero
failures.

Budgeting the 68 pages

page type	pages	slots	what's in them
Alphabetic world scripts	30	3,840	Latin, Greek, Cyrillic, Armenian, Hebrew, Arabic, the Indic scripts, Thai, Lao, Tibetan, Myanmar, Georgian, Ethiopic, Khmer, Vietnamese, Kana + CJK punctuation, Mongolian, currencies
Chinese	32	4,096	the highest-frequency Han characters
Korean	6	768	the highest-frequency Hangul syllables
Total	68	8,704

Each category splits into two kinds of page:

36 block pages (28 world scripts + 8 Chinese) — a plain 128-code-point-aligned slice of Unicode, so the slot is arithmetic (code_point & 0x7F).
32 indexed pages (2 world + 24 Chinese + 6 Korean) — 128 entries of a curated list, so the slot is a lookup.

Chinese and Korean are indexed rather than block-mapped because Unicode orders
Han characters by radical and Hangul syllables by jamo combination, and
neither follows frequency. The first 1,920 Hangul syllables that 15 block
pages would cover reach only 28% of real Korean text; 6 indexed pages reach
89%. All 3,500 characters of China's national standard primary set
(通用规范汉字表) fit in 2 bytes, with 24 slots spare.

The flip side of frequency-ordering is a fallback: 768 of 11,172 Hangul
syllables and 4,096 of 20,992 CJK ideographs reach 2 bytes. The rest — rare
syllables, proper names, loanwords, most Traditional Chinese — cost 3.

What the trail byte can and can't be

Of the 128 possible trail values, 60 are excluded: the 33 control codes
(0x00–0x1F plus DEL) and space, plus 26 punctuation marks that carry
structural meaning wherever text flows —

" & ' / < = > \ `     HTML, shell quoting, JSON strings, path separators
% , - . : ?           URL encoding, CSV, file extensions, key:value
# $ * + ; @ [ ] { | } shell, glob, email, JSON arrays and objects

Only 6 punctuation marks survive as trail bytes — ! ( ) ^ _ ~ — and none is
a metacharacter in JSON, CSV, URLs, file paths, or the shell.

Two different claims live here, and only one of them is true. A UCE-8
stream will not corrupt a parse: no excluded metacharacter can ever appear
in encoded output. But the 68 trail bytes that are used are mostly ASCII
digits and letters, so a UCE-8 stream is not safe for naive matching.
Devanagari's page index is 0x35 — the digit 5 — so नमस्ते encodes to:

A8 35 AE 35 B8 35 CD 35 A4 35 C7 35

Search that word plus the text price: 5 dollars for the byte "5" and you
get 7 matches, one of which is a real digit. Every grep, LIKE '%…%', and
substring index over raw UCE-8 bytes inherits this.

What it actually saves

The raw saving is deterministic: a character that cost 3 bytes costs 2, a flat
33% for that tier. Real text is a mix, so real documents land lower. And
because text is usually compressed in transit and at rest, the compressed
figure is the one that decides most real deployments:

Sample	UTF-8	UCE-8	raw	gzip	brotli
Chinese (simplified)	78	52	33.3%	27.7%	23.3%
Korean	95	69	27.4%	28.0%	18.9%
Japanese	101	70	30.7%	18.4%	12.9%
Thai	160	107	33.1%	11.2%	23.3%
Amharic	92	63	31.5%	13.7%	11.3%
Hindi	169	117	30.8%	9.3%	11.0%
Chinese (traditional)	81	59	27.2%	21.2%	—
JSON with Chinese values	55	47	14.5%	12.5%	—

Read this honestly and three things follow.

The raw saving is real and reliable. 27–33% across every script the
2-byte tier was built for. If you are storing text uncompressed — in memory,
in a fixed-width column, in an embedded system without a compressor — that is
what you get, every time.

Compression eats much of the advantage. UTF-8's repeated 3-byte lead bytes
are precisely the redundancy gzip and brotli exist to remove, so a large part
of what UCE-8 removes by design, a compressor removes anyway. Hindi's 30.8%
becomes 9.3% under gzip.

On long compressed Chinese prose, UCE-8 can lose. Measured across
documents from 121 to 12,199 characters, brotli-compressed UCE-8 came out
about 2% larger than brotli-compressed UTF-8 — stably, at every length.
Brotli models UTF-8's regular structure better than it models UCE-8's.

So the defensible claim is narrower than "33% for everyone," and it is still a
real one: UCE-8 wins clearly on uncompressed text, and wins on compressed CJK,
where the frequency-ordered index pages do information-theoretic work a
general-purpose compressor cannot replicate.

Who it helps

These scripts all sit in UTF-8's 3-byte range (U+0800–U+FFFF). "Page" is the
page index — page 090 is U+0900–U+097F.

Every page's address is a plain ASCII byte, so the whole page map fits in one
ASCII table — row is the high nibble, column the low nibble. Under each row of
pages is the ASCII character that byte actually is:

      0   1   2   3   4   5   6   7   8   9   A   B   C   D   E   F
  0  ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·
  1  ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·   ·
  2  ·  008  ·   ·   ·   ·   ·   ·  010 038  ·   ·   ·   ·   ·   ·
     ␠   !   "   #   $   %   &   '   (   )   *   +   ,   -   .   /
  3 040 050 058 060 068 090 098 0A0 0A8 0B0  ·   ·   ·   ·   ·   ·
     0   1   2   3   4   5   6   7   8   9   :   ;   <   =   >   ?
  4  ·  0C0 0C8 0D0 0D8 0E0 0E8 0F0 100 120 128 130 178 1E8 300 308
     @   A   B   C   D   E   F   G   H   I   J   K   L   M   N   O
  5 WLD WLD 4E0 4E8 540 5B8 628 7E8 8B8 8D0 CHN  ·   ·   ·  CHN CHN
     P   Q   R   S   T   U   V   W   X   Y   Z   [   \   ]   ^   _
  6  ·  CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN
     `   a   b   c   d   e   f   g   h   i   j   k   l   m   n   o
  7 CHN CHN CHN CHN CHN CHN KOR KOR KOR KOR KOR  ·   ·   ·  KOR  ·
     p   q   r   s   t   u   v   w   x   y   z   {   |   }   ~   ␡

Rows 0 and 1 are the 32 control codes, which is why nothing is printable there.
· marks a value that can never be a page index — those plus the structural
punctuation are the 60 exclusions. WLD / CHN / KOR are indexed pages: a
curated 128-entry list rather than an aligned Unicode block. What is left is
68 usable pages × 128 = 8,704 characters in two bytes.

Devanagari sits at row 3, column 5 — ASCII 5. That is why नमस्ते encodes
full of the digit 5.

Page	Script	Unicode block	Speakers (M)	Benefit
090	Devanagari	U+0900–U+097F	600.0	full
098	Bengali	U+0980–U+09FF	284.0	full
0A0	Gurmukhi	U+0A00–U+0A7F	30.0	full
0A8	Gujarati	U+0A80–U+0AFF	57.0	full
0B0	Oriya	U+0B00–U+0B7F	41.0	full
0B8	Tamil	U+0B80–U+0BFF	78.0	full
0C0	Telugu	U+0C00–U+0C7F	83.0	full
0C8	Kannada	U+0C80–U+0CFF	43.0	full
0D0	Malayalam	U+0D00–U+0D7F	45.0	full
0D8	Sinhala	U+0D80–U+0DFF	17.0	full
0E0	Thai	U+0E00–U+0E7F	71.0	full
0E8	Lao	U+0E80–U+0EFF	7.7	full
0F0	Tibetan	U+0F00–U+0FFF	6.0	full
100	Myanmar	U+1000–U+109F	33.0	full
108	Georgian	U+10A0–U+10FF	3.6	full
120	Ethiopic	U+1200–U+137F	135.0	full
178	Khmer	U+1780–U+17FF	18.0	full
180	Mongolian	U+1800–U+18AF	10.0	traditional script — 27.0%
1E8	Vietnamese	U+1E00–U+1EFF	86.0	8.1% — page shared with Yoruba, Igbo, Welsh
300	Japanese	U+3040–U+30FF	122.0	30.7% — kana full, kanji partial
4E0	Chinese	U+4E00–U+9FFF	1,400.0	full — simplified; traditional 27.2%
AC0	Korean	U+AC00–U+D7A3	78.0	full — 89% of running text
	Total		3,248.3

That is roughly 3.2 billion people whose everyday writing drops from 3
bytes per character to 2.

How much each one actually gains

Every figure below is measured, not estimated. Most of the table gains the
full 3→2 bytes; five entries gain less, and they are worth knowing about
before anyone else finds them:

Script	Measured	Why it differs
Japanese	30.7%	Kana and CJK punctuation have dedicated pages; kanji coverage is incidental — 19 of 38 common kanji tested still cost 3 bytes
Chinese (traditional)	27.2%	The index is simplified-only; 28 of 39 common Traditional characters tested fall to tier 3
Mongolian (traditional)	27.0%	32 curated characters of the traditional script
Vietnamese	8.1%	Mostly plain ASCII Latin already — only the diacritic characters were ever 3-byte
Mongolian (Cyrillic)	0.0%	Cyrillic is 2 bytes in UTF-8 and 2 bytes in UCE-8 — unchanged

The Mongolian split is the one to keep straight: Mongolia writes in Cyrillic,
which UCE-8 leaves exactly the same size. The 27% gain belongs to the
traditional script, used mainly in Inner Mongolia. Counting all 10 million
Mongolian speakers as beneficiaries of the traditional-script pages overstates
the case, and it is the first thing a careful reader will check.

What it costs

UCE-8 is not a drop-in, and the case for it is only credible if the costs are
stated alongside the savings.

It is not UTF-8. A UCE-8 stream is not valid UTF-8 and a UTF-8 stream is not valid UCE-8. Every reader, editor, database, terminal and library in the path needs to know which one it is holding. This is an ecosystem break, not an upgrade.
Byte order is not code-point order. The indexed pages are ordered by frequency, so sorting encoded bytes does not sort characters. Binary search, range queries and index ordering over raw UCE-8 all change meaning.
Naive matching breaks, as shown above — trail bytes are ordinary ASCII digits and letters.
Compression narrows the win, and for long brotli-compressed Chinese reverses it.
Coverage is a frozen guess. CHINESE_CHARS, KOREAN_CHARS and WORLD_CHARS cannot be reordered without changing the meaning of every byte sequence ever written. Whatever the frequency tables got wrong is permanent.

Reference implementation

The whole encoder is three branches and a fallback. LEAD_BASE is 0x80 —
setting the high bit marks a byte as "more follows". TAIL_COUNT is 68, the
number of usable trail values.

def encode(code_point):
    """Encode a Unicode code point into 1, 2 or 3 bytes."""

    if code_point < 0:
        handle_error("Negative code point")

    if code_point > MAX_CODE_POINT:
        handle_error("Beyond the Unicode range")

    # TIER 1 - plain ascii passes straight through
    if code_point < 0x80:
        return bytes([code_point])

    # TIER 2a - block pages: world scripts and the 8 Chinese blocks
    trail = PAGE_TO_TRAIL.get((code_point >> 7) << 3)
    if trail is not None:
        return bytes([LEAD_BASE | (code_point & 0x7F), trail])

    # TIER 2b - indexed Chinese
    slot = CHINESE_SLOT.get(code_point)
    if slot is not None:
        return bytes([LEAD_BASE | (slot & 0x7F), CHINESE_TRAILS[slot >> 7]])

    # TIER 2c - indexed Korean
    slot = KOREAN_SLOT.get(code_point)
    if slot is not None:
        return bytes([LEAD_BASE | (slot & 0x7F), KOREAN_TRAILS[slot >> 7]])

    # TIER 2d - indexed world pages
    slot = WORLD_SLOT.get(code_point)
    if slot is not None:
        return bytes([LEAD_BASE | (slot & 0x7F), WORLD_TRAILS[slot >> 7]])

    # TIER 3 - everything else
    index = code_point - 0x80
    tail = index % TAIL_COUNT
    rest = index // TAIL_COUNT

    return bytes([
        LEAD_BASE | (rest >> 7),
        LEAD_BASE | (rest & 0x7F),
        TAIL_BYTES[tail],
    ])

Decoding reads the high bit of the second byte to pick the tier, then the
trail byte to pick the page:

def decode(data):
    """Decode a 1, 2 or 3 byte sequence back into a Unicode code point."""

    first = data[0]

    if first < 0x80:
        return first

    second = data[1]

    # TIER 2 - second byte clears the high bit, so it terminates
    if second < 0x80:
        page = TRAIL_TO_PAGE[second]

        if page is None:
            handle_error("Trail byte is not a page index")

        if page is WORLD:
            return WORLD_CODE_POINTS[(WORLD_PAGE[second] << 7) | (first & 0x7F)]

        if page is CHINESE:
            return CHINESE_CODE_POINTS[(CHINESE_PAGE[second] << 7) | (first & 0x7F)]

        if page is KOREAN:
            return KOREAN_CODE_POINTS[(KOREAN_PAGE[second] << 7) | (first & 0x7F)]

        return ((page >> 3) << 7) | (first & 0x7F)

    # TIER 3 - second byte keeps the high bit, a third follows
    tail = TAIL_INDEX[data[2]]

    if tail is None:
        handle_error("Third byte is not a legal terminating value")

    rest = ((first & 0x7F) << 7) | (second & 0x7F)

    return 0x80 + rest * TAIL_COUNT + tail

Encoding a whole string is a join, since every character is independent:

def uce8_encode(text):
    return b"".join(encode(ord(c)) for c in text)

Note what tier 3 does differently. Tiers 1 and 2 map a code point to a page
and slot; tier 3 has no page table left, so it spreads the remaining code
points across the 68 legal trail values arithmetically — index % TAIL_COUNT
picks the terminator, index // TAIL_COUNT becomes the two lead bytes. That
is why tier 3's capacity is exactly 128 × 128 × 68, and why the encoding never
emits an illegal terminator even in its fallback path.

Status

UCE-8 is an experimental encoding, not a deployed standard.

Full source: https://gitlab.com/eric.shagdarjav/uce

File	What it is
uce_hibrid.py	Reference implementation — encoder, decoder, frozen character tables
verify.py	Reproduces every figure in this article
index.html	Interactive demo — runs the real tables in the browser

uce_hibrid.py has no dependencies beyond the Python standard library. Run it
directly for the built-in test vectors, or python3 docs/verify.py to
reproduce every number here.

DEV Community: Erdene-Ochir Sh