UCE-8 — Unicode 3-byte variable-width Compact Encoding — is a character
encoding sized for the world's living scripts, reaching 128 × 128 × 68 =
1,114,112 characters in at most three bytes.
UTF-8 charges most of the world's living languages three bytes per character.
UCE-8 charges two. This is how it does that, what it measurably saves, and
what it costs.
Every figure in this article was produced by running
verify.py against the reference implementation,
uce_hibrid.py. Reviewer notes on the first draft are
kept in reviewer notes.
The problem
In the text we use every day, characters are stored with a length of 1–4 bytes.
UTF-8:
- English-language text = 1 byte
- European languages + Middle East = 2 bytes
- Asian and African languages = 3 bytes
- Languages that have fallen out of use = 4 bytes
UCE-8, instead:
- English-language text = 1 byte
- Every language alive and in use today = 2 bytes
- Languages that have fallen out of use = 3 bytes
Which tier a script lands in was decided by Unicode's code-point ordering, not
by how many people write in it. That is why Devanagari, Thai, Ethiopic and
Chinese — between them the everyday scripts of billions — sit in UTF-8's
expensive three-byte tier.
How it works
1 byte = 8 bits = 256 possible values. Call the first 128 the low byte and
the last 128 the high byte. The high bit is the length signal: a high
byte means "another byte follows," a low byte means "the character ends
here." Decoding stops the instant a low byte appears — no length field, no
lookahead.
| byte pattern | length |
|---|---|
| low | 1 byte |
| high, low | 2 bytes |
| high, high, low | 3 bytes |
That gives a self-terminating, three-tier scheme:
| tier | byte pattern | capacity | covers |
|---|---|---|---|
| 1 | 0xxxxxxx |
128 | plain ASCII |
| 2 |
1xxxxxxx 0xxxxxxx
|
8,704 | 68 pages, 128 characters each |
| 3 |
1xxxxxxx 1xxxxxxx 0xxxxxxx
|
1,114,112 | every remaining code point |
The two positions have names, and they are just the high/low split above seen
from the other side:
- A lead byte is a high byte (
0x80–0xFF). It always means "another follows," and it carries the character's slot — which of a page's 128 entries this is. - A trail byte is a low byte (
0x00–0x7F). It terminates the character, and it is the page index. Of its 128 possible values, 68 are usable (see below), so tier 2 reaches 128 × 68 = 8,704 characters in 2 bytes where UTF-8 needs 3.
Note the order, because it inverts the usual arrangement: the second byte
says which script you are in, and the first says which letter of it. UTF-8
puts the identifying information in the leading byte; UCE-8 puts it in the
terminator. Devanagari न (U+0928) encodes as A8 35:
A8 = 0x80 | 40 lead byte — slot 40 of the page
35 = page index trail byte — the page holding U+0900–U+097F
Hold the lead byte at A8 and change only the trail byte, and the same slot
number lands in a different script each time:
| bytes | code point | character |
|---|---|---|
A8 35 |
U+0928 | न Devanagari NA |
A8 45 |
U+0E28 | ศ Thai SO SALA |
A8 33 |
U+0628 | ب Arabic BEH |
That is the whole addressing scheme: the trail byte chooses the page, the lead
byte chooses the entry.
The tiers fit Unicode exactly. Tier 3's capacity is 128 × 128 × 68 =
1,114,112. Unicode's total code space is 17 × 65,536 = 1,114,112. The same
number. UCE-8 covers all of Unicode with no waste and nothing spare — and
encoding then decoding all 1,114,112 code points round-trips with zero
failures.
Budgeting the 68 pages
| page type | pages | slots | what's in them |
|---|---|---|---|
| Alphabetic world scripts | 30 | 3,840 | Latin, Greek, Cyrillic, Armenian, Hebrew, Arabic, the Indic scripts, Thai, Lao, Tibetan, Myanmar, Georgian, Ethiopic, Khmer, Vietnamese, Kana + CJK punctuation, Mongolian, currencies |
| Chinese | 32 | 4,096 | the highest-frequency Han characters |
| Korean | 6 | 768 | the highest-frequency Hangul syllables |
| Total | 68 | 8,704 |
Each category splits into two kinds of page:
-
36 block pages (28 world scripts + 8 Chinese) — a plain
128-code-point-aligned slice of Unicode, so the slot is arithmetic
(
code_point & 0x7F). - 32 indexed pages (2 world + 24 Chinese + 6 Korean) — 128 entries of a curated list, so the slot is a lookup.
Chinese and Korean are indexed rather than block-mapped because Unicode orders
Han characters by radical and Hangul syllables by jamo combination, and
neither follows frequency. The first 1,920 Hangul syllables that 15 block
pages would cover reach only 28% of real Korean text; 6 indexed pages reach
89%. All 3,500 characters of China's national standard primary set
(通用规范汉字表) fit in 2 bytes, with 24 slots spare.
The flip side of frequency-ordering is a fallback: 768 of 11,172 Hangul
syllables and 4,096 of 20,992 CJK ideographs reach 2 bytes. The rest — rare
syllables, proper names, loanwords, most Traditional Chinese — cost 3.
What the trail byte can and can't be
Of the 128 possible trail values, 60 are excluded: the 33 control codes
(0x00–0x1F plus DEL) and space, plus 26 punctuation marks that carry
structural meaning wherever text flows —
" & ' / < = > \ ` HTML, shell quoting, JSON strings, path separators
% , - . : ? URL encoding, CSV, file extensions, key:value
# $ * + ; @ [ ] { | } shell, glob, email, JSON arrays and objects
Only 6 punctuation marks survive as trail bytes — ! ( ) ^ _ ~ — and none is
a metacharacter in JSON, CSV, URLs, file paths, or the shell.
Two different claims live here, and only one of them is true. A UCE-8
stream will not corrupt a parse: no excluded metacharacter can ever appear
in encoded output. But the 68 trail bytes that are used are mostly ASCII
digits and letters, so a UCE-8 stream is not safe for naive matching.
Devanagari's page index is 0x35 — the digit 5 — so नमस्ते encodes to:
A8 35 AE 35 B8 35 CD 35 A4 35 C7 35
Search that word plus the text price: 5 dollars for the byte "5" and you
get 7 matches, one of which is a real digit. Every grep, LIKE '%…%', and
substring index over raw UCE-8 bytes inherits this.
What it actually saves
The raw saving is deterministic: a character that cost 3 bytes costs 2, a flat
33% for that tier. Real text is a mix, so real documents land lower. And
because text is usually compressed in transit and at rest, the compressed
figure is the one that decides most real deployments:
| Sample | UTF-8 | UCE-8 | raw | gzip | brotli |
|---|---|---|---|---|---|
| Chinese (simplified) | 78 | 52 | 33.3% | 27.7% | 23.3% |
| Korean | 95 | 69 | 27.4% | 28.0% | 18.9% |
| Japanese | 101 | 70 | 30.7% | 18.4% | 12.9% |
| Thai | 160 | 107 | 33.1% | 11.2% | 23.3% |
| Amharic | 92 | 63 | 31.5% | 13.7% | 11.3% |
| Hindi | 169 | 117 | 30.8% | 9.3% | 11.0% |
| Chinese (traditional) | 81 | 59 | 27.2% | 21.2% | — |
| JSON with Chinese values | 55 | 47 | 14.5% | 12.5% | — |
Read this honestly and three things follow.
The raw saving is real and reliable. 27–33% across every script the
2-byte tier was built for. If you are storing text uncompressed — in memory,
in a fixed-width column, in an embedded system without a compressor — that is
what you get, every time.
Compression eats much of the advantage. UTF-8's repeated 3-byte lead bytes
are precisely the redundancy gzip and brotli exist to remove, so a large part
of what UCE-8 removes by design, a compressor removes anyway. Hindi's 30.8%
becomes 9.3% under gzip.
On long compressed Chinese prose, UCE-8 can lose. Measured across
documents from 121 to 12,199 characters, brotli-compressed UCE-8 came out
about 2% larger than brotli-compressed UTF-8 — stably, at every length.
Brotli models UTF-8's regular structure better than it models UCE-8's.
So the defensible claim is narrower than "33% for everyone," and it is still a
real one: UCE-8 wins clearly on uncompressed text, and wins on compressed CJK,
where the frequency-ordered index pages do information-theoretic work a
general-purpose compressor cannot replicate.
Who it helps
These scripts all sit in UTF-8's 3-byte range (U+0800–U+FFFF). "Page" is the
page index — page 090 is U+0900–U+097F.
Every page's address is a plain ASCII byte, so the whole page map fits in one
ASCII table — row is the high nibble, column the low nibble. Under each row of
pages is the ASCII character that byte actually is:
0 1 2 3 4 5 6 7 8 9 A B C D E F
0 · · · · · · · · · · · · · · · ·
1 · · · · · · · · · · · · · · · ·
2 · 008 · · · · · · 010 038 · · · · · ·
␠ ! " # $ % & ' ( ) * + , - . /
3 040 050 058 060 068 090 098 0A0 0A8 0B0 · · · · · ·
0 1 2 3 4 5 6 7 8 9 : ; < = > ?
4 · 0C0 0C8 0D0 0D8 0E0 0E8 0F0 100 120 128 130 178 1E8 300 308
@ A B C D E F G H I J K L M N O
5 WLD WLD 4E0 4E8 540 5B8 628 7E8 8B8 8D0 CHN · · · CHN CHN
P Q R S T U V W X Y Z [ \ ] ^ _
6 · CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN CHN
` a b c d e f g h i j k l m n o
7 CHN CHN CHN CHN CHN CHN KOR KOR KOR KOR KOR · · · KOR ·
p q r s t u v w x y z { | } ~ ␡
Rows 0 and 1 are the 32 control codes, which is why nothing is printable there.
· marks a value that can never be a page index — those plus the structural
punctuation are the 60 exclusions. WLD / CHN / KOR are indexed pages: a
curated 128-entry list rather than an aligned Unicode block. What is left is
68 usable pages × 128 = 8,704 characters in two bytes.
Devanagari sits at row 3, column 5 — ASCII 5. That is why नमस्ते encodes
full of the digit 5.
| Page | Script | Unicode block | Speakers (M) | Benefit |
|---|---|---|---|---|
| 090 | Devanagari | U+0900–U+097F | 600.0 | full |
| 098 | Bengali | U+0980–U+09FF | 284.0 | full |
| 0A0 | Gurmukhi | U+0A00–U+0A7F | 30.0 | full |
| 0A8 | Gujarati | U+0A80–U+0AFF | 57.0 | full |
| 0B0 | Oriya | U+0B00–U+0B7F | 41.0 | full |
| 0B8 | Tamil | U+0B80–U+0BFF | 78.0 | full |
| 0C0 | Telugu | U+0C00–U+0C7F | 83.0 | full |
| 0C8 | Kannada | U+0C80–U+0CFF | 43.0 | full |
| 0D0 | Malayalam | U+0D00–U+0D7F | 45.0 | full |
| 0D8 | Sinhala | U+0D80–U+0DFF | 17.0 | full |
| 0E0 | Thai | U+0E00–U+0E7F | 71.0 | full |
| 0E8 | Lao | U+0E80–U+0EFF | 7.7 | full |
| 0F0 | Tibetan | U+0F00–U+0FFF | 6.0 | full |
| 100 | Myanmar | U+1000–U+109F | 33.0 | full |
| 108 | Georgian | U+10A0–U+10FF | 3.6 | full |
| 120 | Ethiopic | U+1200–U+137F | 135.0 | full |
| 178 | Khmer | U+1780–U+17FF | 18.0 | full |
| 180 | Mongolian | U+1800–U+18AF | 10.0 | traditional script — 27.0% |
| 1E8 | Vietnamese | U+1E00–U+1EFF | 86.0 | 8.1% — page shared with Yoruba, Igbo, Welsh |
| 300 | Japanese | U+3040–U+30FF | 122.0 | 30.7% — kana full, kanji partial |
| 4E0 | Chinese | U+4E00–U+9FFF | 1,400.0 | full — simplified; traditional 27.2% |
| AC0 | Korean | U+AC00–U+D7A3 | 78.0 | full — 89% of running text |
| Total | 3,248.3 |
That is roughly 3.2 billion people whose everyday writing drops from 3
bytes per character to 2.
How much each one actually gains
Every figure below is measured, not estimated. Most of the table gains the
full 3→2 bytes; five entries gain less, and they are worth knowing about
before anyone else finds them:
| Script | Measured | Why it differs |
|---|---|---|
| Japanese | 30.7% | Kana and CJK punctuation have dedicated pages; kanji coverage is incidental — 19 of 38 common kanji tested still cost 3 bytes |
| Chinese (traditional) | 27.2% | The index is simplified-only; 28 of 39 common Traditional characters tested fall to tier 3 |
| Mongolian (traditional) | 27.0% | 32 curated characters of the traditional script |
| Vietnamese | 8.1% | Mostly plain ASCII Latin already — only the diacritic characters were ever 3-byte |
| Mongolian (Cyrillic) | 0.0% | Cyrillic is 2 bytes in UTF-8 and 2 bytes in UCE-8 — unchanged |
The Mongolian split is the one to keep straight: Mongolia writes in Cyrillic,
which UCE-8 leaves exactly the same size. The 27% gain belongs to the
traditional script, used mainly in Inner Mongolia. Counting all 10 million
Mongolian speakers as beneficiaries of the traditional-script pages overstates
the case, and it is the first thing a careful reader will check.
What it costs
UCE-8 is not a drop-in, and the case for it is only credible if the costs are
stated alongside the savings.
- It is not UTF-8. A UCE-8 stream is not valid UTF-8 and a UTF-8 stream is not valid UCE-8. Every reader, editor, database, terminal and library in the path needs to know which one it is holding. This is an ecosystem break, not an upgrade.
- Byte order is not code-point order. The indexed pages are ordered by frequency, so sorting encoded bytes does not sort characters. Binary search, range queries and index ordering over raw UCE-8 all change meaning.
- Naive matching breaks, as shown above — trail bytes are ordinary ASCII digits and letters.
- Compression narrows the win, and for long brotli-compressed Chinese reverses it.
-
Coverage is a frozen guess.
CHINESE_CHARS,KOREAN_CHARSandWORLD_CHARScannot be reordered without changing the meaning of every byte sequence ever written. Whatever the frequency tables got wrong is permanent.
Reference implementation
The whole encoder is three branches and a fallback. LEAD_BASE is 0x80 —
setting the high bit marks a byte as "more follows". TAIL_COUNT is 68, the
number of usable trail values.
def encode(code_point):
"""Encode a Unicode code point into 1, 2 or 3 bytes."""
if code_point < 0:
handle_error("Negative code point")
if code_point > MAX_CODE_POINT:
handle_error("Beyond the Unicode range")
# TIER 1 - plain ascii passes straight through
if code_point < 0x80:
return bytes([code_point])
# TIER 2a - block pages: world scripts and the 8 Chinese blocks
trail = PAGE_TO_TRAIL.get((code_point >> 7) << 3)
if trail is not None:
return bytes([LEAD_BASE | (code_point & 0x7F), trail])
# TIER 2b - indexed Chinese
slot = CHINESE_SLOT.get(code_point)
if slot is not None:
return bytes([LEAD_BASE | (slot & 0x7F), CHINESE_TRAILS[slot >> 7]])
# TIER 2c - indexed Korean
slot = KOREAN_SLOT.get(code_point)
if slot is not None:
return bytes([LEAD_BASE | (slot & 0x7F), KOREAN_TRAILS[slot >> 7]])
# TIER 2d - indexed world pages
slot = WORLD_SLOT.get(code_point)
if slot is not None:
return bytes([LEAD_BASE | (slot & 0x7F), WORLD_TRAILS[slot >> 7]])
# TIER 3 - everything else
index = code_point - 0x80
tail = index % TAIL_COUNT
rest = index // TAIL_COUNT
return bytes([
LEAD_BASE | (rest >> 7),
LEAD_BASE | (rest & 0x7F),
TAIL_BYTES[tail],
])
Decoding reads the high bit of the second byte to pick the tier, then the
trail byte to pick the page:
def decode(data):
"""Decode a 1, 2 or 3 byte sequence back into a Unicode code point."""
first = data[0]
if first < 0x80:
return first
second = data[1]
# TIER 2 - second byte clears the high bit, so it terminates
if second < 0x80:
page = TRAIL_TO_PAGE[second]
if page is None:
handle_error("Trail byte is not a page index")
if page is WORLD:
return WORLD_CODE_POINTS[(WORLD_PAGE[second] << 7) | (first & 0x7F)]
if page is CHINESE:
return CHINESE_CODE_POINTS[(CHINESE_PAGE[second] << 7) | (first & 0x7F)]
if page is KOREAN:
return KOREAN_CODE_POINTS[(KOREAN_PAGE[second] << 7) | (first & 0x7F)]
return ((page >> 3) << 7) | (first & 0x7F)
# TIER 3 - second byte keeps the high bit, a third follows
tail = TAIL_INDEX[data[2]]
if tail is None:
handle_error("Third byte is not a legal terminating value")
rest = ((first & 0x7F) << 7) | (second & 0x7F)
return 0x80 + rest * TAIL_COUNT + tail
Encoding a whole string is a join, since every character is independent:
def uce8_encode(text):
return b"".join(encode(ord(c)) for c in text)
Note what tier 3 does differently. Tiers 1 and 2 map a code point to a page
and slot; tier 3 has no page table left, so it spreads the remaining code
points across the 68 legal trail values arithmetically — index % TAIL_COUNT
picks the terminator, index // TAIL_COUNT becomes the two lead bytes. That
is why tier 3's capacity is exactly 128 × 128 × 68, and why the encoding never
emits an illegal terminator even in its fallback path.
Status
UCE-8 is an experimental encoding, not a deployed standard.
Full source: https://gitlab.com/eric.shagdarjav/uce
| File | What it is |
|---|---|
| uce_hibrid.py | Reference implementation — encoder, decoder, frozen character tables |
| verify.py | Reproduces every figure in this article |
| index.html | Interactive demo — runs the real tables in the browser |
uce_hibrid.py has no dependencies beyond the Python standard library. Run it
directly for the built-in test vectors, or python3 docs/verify.py to
reproduce every number here.
Top comments (0)