DEV Community: Eugen

Styled Unicode Breaks Character Counters

Eugen — Sun, 23 Aug 2026 17:36:52 +0000

A 76-character bio, styled once, no longer has one useful length.

const s = "𝐚𝐞𝐬𝐭𝐡𝐞𝐭𝐢𝐜";
[...s].length                          // 9   code points (and 9 graphemes here)
s.length                               // 18  UTF-16 units — what String.length counts
new TextEncoder().encode(s).length     // 36  UTF-8 bytes  — what storage counts

Three defensible answers, factor of four on one word. I found this while auditing a counter beside field limits I do not own. The counter reported code points; the destination's unit was unknown. That is already enough to make a green "safe" verdict dishonest.

Styled letters from the Mathematical Alphanumeric Symbols block live in the Supplementary Plane: 1 code point = 2 UTF-16 units = 4 UTF-8 bytes. If you display one number next to a limit documented only as "characters", you are making a claim about a unit the destination never published.

What a real bio line does

Same 76-code-point line, four styles. Measured, not recalled:

style	code points	UTF-16	UTF-8 bytes
plain	76	76	76
Bold `𝐀`	76	134	250
Script `𝒜`	76	117	225
Small Caps `ᴀ`	76	76	155

A code-point counter calls Bold "76". The same text occupies 134 UTF-16 units or 250 UTF-8 bytes. If the destination budgets one of those units, a verdict based on another is meaningless. Documentation that says only "characters" does not resolve it.

Honest UI: three states, not a green number

Measure all three. If the destination explicitly publishes a byte limit, enforce it. If it publishes a character limit without naming the unit, keep bytes visible but do not compare them to the same number: byte and character budgets are not interchangeable.

const measure = (s) => ({
  codePoints: [...s].length,
  utf16: s.length,
  bytes: new TextEncoder().encode(s).length,
});

// ok    — fits by code points and UTF-16; bytes are reported separately
// risk  — fits by code points, not by UTF-16: the field decides, and it did not say
// over  — too long even by the most generous count
const state = (m, limit) =>
  m.codePoints > limit ? "over" : m.utf16 > limit ? "risk" : "ok";

risk is the honest answer when the platform has not told you the unit. The only way to name the unit is to paste a known-length styled string into the live field and see where it cuts. Until then, do not draw a green bar.

The next failure is truncation, not the count

.slice(), .substring(), and [0..n] on a JavaScript string cut at UTF-16 boundaries. An odd index inside a Supplementary character keeps half a surrogate pair. Many encoders and renderers replace that lone surrogate with U+FFFD:

"𝐚𝐞𝐬𝐭𝐡𝐞𝐭𝐢𝐜".slice(0, 7)   // "𝐚𝐞𝐬\uD835"  → renders 𝐚𝐞𝐬�

Truncate on code points — and on grapheme clusters if you allow combining marks (a̶ is two code points):

const cutCodePoints = (s, n) => [...s].slice(0, n).join("");
const cutGraphemes = (s, n) =>
  [...new Intl.Segmenter(undefined, { granularity: "grapheme" }).segment(s)]
    .slice(0, n)
    .map((g) => g.segment)
    .join("");

Where the rest of the measurements live

I did not want a second guessed number in the UI, so I ran this against a shipping 22-style catalog rather than a handful of examples. Length units, the NFKC rule that flattens 17 of those 22 styles, coverage holes per family, and how /\d/ in JavaScript disagrees with Python on 𝟐𝟎𝟐𝟔 are written up with the reproducing code in Unicode text in fields you do not own (CC0).

Checklist I actually use:

[ ] Count in all three units; never show one number as if it were the limit
[ ] Truncate on code points or graphemes, never on UTF-16 indices
[ ] Normalise before validating; know whether your storage path normalises
[ ] Test rendering on a device you did not compose the text on
[ ] Disclose coverage holes instead of silently substituting another letter

The catalog those numbers were measured on is the type-style-copy tool at fontius.app. The skill is the document; the tool is just where the 22 families live.

Research disclosure: The catalog measurements come from focused local tests against the 22 shipped styles. I did not establish which unit any named third-party platform uses; that requires testing the field itself.

AI-assistance disclosure: AI tools assisted with research navigation, code verification, and drafting. I reviewed the evidence and take responsibility for the claims and conclusions.

An Emoji Made Me Audit Byte Offsets

Eugen — Wed, 12 Aug 2026 16:00:00 +0000

I was building my own harness for solving engineering tasks when an emoji broke
one of my assumptions.

The file was not malformed. There was no exotic exploit payload. The fixture
contained a perfectly ordinary Unicode string:

Rendered:    A😀éB
Code points: U+0041 U+1F600 U+0065 U+0301 U+0042

It looks like four user-visible characters. In this exact NFD fixture, the
zero-based start offset of B is 4 Unicode code points, 5 UTF-16 code units in
JavaScript, and 8 UTF-8 bytes.

That difference is harmless until one component calculates a position and
another component gives that number mutation authority.

TL;DR: Text positions need explicit units. The NFD string above has 4
grapheme clusters, 5 code points, 6 UTF-16 code units and 9 UTF-8 bytes,
while B begins at zero-based offsets 4 / 5 / 8. Across local runtime probes
and pinned source inspection, I found that safe agent editing needs strict
admission, byte-preserving mutation, and review, tests and settlement bound
to exact source and candidate bytes.

I was building a task harness, not another chat UI

My goal was a task-oriented development harness:

conversation and design
  -> frozen task
  -> architect
  -> implementer
  -> tests
  -> reviewer
  -> verified result

The conversation is useful upstream, but the implementation stage needs a much
stricter contract. Each role should receive only the context and artifacts it
needs, and a reviewer should approve the exact candidate that was tested.

I used Python to prototype the edit path quickly and was considering TypeScript
for more of the lifecycle. The Unicode fixture was a tripwire for the boundary
between them. Its total length has four equally correct descriptions:

4 grapheme clusters
5 Unicode code points
6 UTF-16 code units
9 UTF-8 bytes

Nothing was corrupted. I caught this in a local, model-free test—not in a
customer outage or a breach. But it made me ask how far the mechanism could
travel, so I reconstructed the customer and adversarial scenarios below.

The first reconstructed failure needed no malicious input

Imagine a customer gives the harness this small configuration file:

# Deploy 🚀
timeout = 30

The requested change is boring: set the timeout to 60.

Suppose the TypeScript orchestration layer computes a range from a JavaScript
string and the Python or Rust edit layer consumes the same number as a UTF-8
byte offset. Every position after the rocket now refers to different data in
the two components.

The best outcome is a clean range-boundary refusal. An ordinary failure is an
anchor miss or an off-by-one edit. The dangerous outcome is that a valid but
different adjacent token is changed while the preview was calculated in the
other coordinate system.

This customer story is a reconstruction, not an incident report. The measured
fact underneath it is the zero-based 4 / 5 / 8 start-offset split in the
fixture above.

One edit crosses several representations

An agent edit crosses several representations:

model JSON
  -> runtime string
  -> decoded source text
  -> matched span
  -> candidate
  -> diff shown to a reviewer
  -> bytes written to disk

Every arrow can change the meaning of “the same text.”

The repository ultimately stores bytes. JavaScript strings are indexed as
UTF-16 code units. Python strings are indexed by code points. A UI usually wants
grapheme clusters. A fuzzy matcher may normalize text before comparing it.

Those are all valid choices in the right layer. The problem begins when a value
created in one coordinate system is consumed as another.

I later isolated the same counting problem in a smaller browser-facing example:
styled Unicode breaks character counters
when visible glyphs, code points, UTF-16 units, and UTF-8 bytes diverge.

Key takeaway: A numeric offset is not a contract until its coordinate
unit is part of the protocol.

Three runtimes, one malformed JSON string

I wrote three small model-free probes and ran this case on Node.js 22.17.0,
Python 3.14.4 and Rust 1.97.0. The fixture was the JSON string "\uD800", an
unpaired UTF-16 surrogate.

RFC 8259 explicitly warns
that JSON grammar can carry such a value even though it is not a Unicode scalar
value.

Here is what I observed:

Boundary	Observed result
Node `JSON.parse`	produced a one-unit string containing U+D800
Node `Buffer.from(..., "utf8")`	silently encoded U+FFFD: `ef bf bd`
Python `json.loads`	produced a `str` containing U+D800
Python strict UTF-8 encode	raised `UnicodeEncodeError`
Python encode with `errors="replace"`	emitted ASCII `?`: `3f`
Python `surrogatepass`	emitted `ed a0 80`, which is not well-formed UTF-8
Rust `char` / `String` boundary	U+D800 was not representable; strict UTF-16 conversion failed

“The JSON parsed” was not a sufficient admission rule in any of the three
runtimes.

Then, in a standalone Python script, I reimplemented a destructive ordering
pattern visible in one inspected source path:

open target with O_TRUNC
  -> encode candidate as UTF-8
  -> encoding fails on U+D800

The Python probe raised UnicodeEncodeError, but the target was already empty.
When I encoded the candidate before opening the target, the same error occurred
and the original bytes stayed unchanged.

That is a small ordering decision with a large difference in failure semantics.

Here is one realistic way those primitives could compose. An upstream length
limit slices a JavaScript string through the middle of an emoji. JSON still
serializes the remaining surrogate code unit. A Python worker parses it, opens
the destination with truncation, and only then discovers that the candidate
cannot be encoded as strict UTF-8. The result can be an empty file even though
every individual step looked routine.

I reproduced the slicing, parsing and truncate-before-encode mechanisms, but
not that complete production chain. It is a reconstructed failure scenario and
a regression test I now want, not a claim about a real customer request.

Invalid UTF-8 can be repaired without asking you

The next fixture was four bytes:

61 ff 62 0a

I asked the simulated editor to make an unrelated change: a to A.

A strict decoder rejected the source in Node, Python and Rust. A lossy decoder
produced this candidate in all three:

41 ef bf bd 62 0a

The requested byte changed from 61 to 41. The unrelated invalid byte ff
also became the three-byte encoding of U+FFFD.

The edit can report success. The output is valid UTF-8. The visible diff may
focus on the requested line. Yet the tool has permanently changed bytes outside
the intended mutation.

I repeated the fixture with the invalid byte at offset 1001. Validation of only
the first 1000 bytes passed in all three runtimes; full validation failed. This
matters because a byte sample is useful for classification, but it is not proof
that the complete source is valid text.

This is where my network-security instincts changed the question. If I were
red-teaming this boundary, offset 1001 is exactly where I would place the byte:
just beyond the classifier's evidence window, but still inside the later lossy
decode. That becomes a validation-coverage gap if the sample is treated as full
admission. Sampling itself is not the problem; giving an advisory classifier
authority over bytes it never inspected is.

Again, no attacker sent this to my system. I constructed the input locally to
test the boundary.

Key takeaway: A sample can classify input. It cannot authorize mutation
of bytes it never inspected.

Strict UTF-8 is necessary, but it is not format detection

I also tested UTF-16LE.

A file with the ff fe BOM was rejected by strict UTF-8, as expected. But a
BOM-less UTF-16LE file containing ASCII-range text passed strict UTF-8 in all
three runtimes:

6f 00 6c 00 64 00 0a 00

As UTF-8, that is valid text with four embedded NUL characters. As UTF-16LE, it
is simply old followed by a line ending.

The BOM-bearing form is not an exotic historical format. Windows PowerShell
has produced UTF-16LE files with a BOM through commands such as Out-File and
redirection; the
PowerShell encoding documentation
describes the version-dependent behavior. The harder BOM-less fixture above was
constructed for this probe; that PowerShell citation is not evidence that those
commands produced it.

A UTF-8-only edit tool does not need to support UTF-16. It does need to refuse
an unsupported format before mutation and prove that refusal left the source
unchanged.

This is the least dramatic scenario and probably the most practical one. A
customer can submit a legitimate PowerShell script created by an older Windows
tool, ask for a one-line change, and violate none of your stated assumptions.
The editor still needs a deterministic answer: preserve the declared encoding
or refuse before effect. Silently guessing and rewriting is not a customer
error.

Normalization can change “one match” into “two matches”

These two strings often render identically:

NFC: é        -> c3 a9
NFD: e + ◌́   -> 65 cc 81

They are canonically equivalent, but they are not byte-identical.

I placed one of each in a source. The NFD anchor had one exact match. After NFKC
normalization, the same logical anchor had two matches.

That means normalization did not merely make search friendlier. It changed the
cardinality of an authorized mutation.

Key takeaway: Exact byte matching owns mutation. Normalized or fuzzy
matching may propose candidates; if it changes zero/one/many cardinality, the
tool reports that fact instead of silently selecting the first result.

Unicode normalization is valuable.
Making it mutation-authoritative without an explicit contract is the problem.

Then I checked real coding-agent implementations

At first I suspected this was mainly a TypeScript/Python problem. Rust strings
cannot contain lone surrogates, so perhaps Rust editors avoided the whole class.

That hypothesis was wrong.

I inspected pinned source snapshots and traced decode -> locate -> candidate ->
write paths. I did not run the complete products, so these are bounded source
findings followed by standard-runtime reproductions of the selected primitives.

The table reports policy and ordering visible at pinned source snapshots. It is
not a list of confirmed product vulnerabilities, incidents or end-to-end runs.

Editor path	Relevant behavior found in source
OpenAI Codex `apply_patch`	strict `String::from_utf8` rejects invalid update sources
Pi coding-agent edit	`Buffer.toString("utf-8")` creates a lossy text authority
Deep Agents filesystem edit	strict source decode; the inspected local write path orders `O_TRUNC` before the text write encodes the candidate
Grok Standard search/replace	Rust explicitly chooses `String::from_utf8_lossy`; any later effect depends on the rest of its settlement path
Hermes default replace	reads through `cat`; the default local process transport decodes with `errors="replace"`, while the write path separately uses temp+rename

The useful conclusion is not “language X is safe.”

Language changes which failure modes are easy or impossible. The complete edit
contract decides whether they become file corruption.

Two Rust editors can choose opposite invalid-UTF-8 policies. A Python editor can
fail closed at decoding and still lose a file because encoding happens after
truncate. An atomic rename can settle a candidate correctly even when that
candidate was already derived from a lossy source.

The network-security analogy is parser differential, not packets

I am a network security engineer, so this shape felt familiar. HTTP/1.1 is
defined over octets, and
RFC 9112 warns that
parsing the message as Unicode too early can create security vulnerabilities.
The structural problem here also resembles a parser differential:

network path:
octets -> intermediary parser -> security decision -> origin parser

agent edit path:
file bytes -> runtime decoder -> model/reviewer decision -> filesystem writer

I am not claiming that an AI edit bug is HTTP request smuggling. The analogy is
about disagreement at boundaries.

If the security layer approves representation A while the effect layer writes
representation B, the approval is attached to the wrong object.

That becomes especially important in a multi-agent harness. A test result on
candidate A cannot authorize candidate B. A reviewer approving a pretty diff
cannot authorize bytes that were regenerated later from a different source
snapshot.

Best case, ordinary failure, worst case

The fixtures helped me separate failure levels:

Case	Best case	Ordinary unsafe result	Plausible worst case
invalid UTF-8	refuse before effect	U+FFFD silently replaces unrelated bytes	corrupted generated/signed fixture with a deceptively small visible diff
lone surrogate	reject/pre-encode before open	tool returns an encoding error	silent replacement or an empty target after truncate
mixed BOM/newlines	preserve untouched byte spans	whole-file representation rewrite	broken script, checksum or protocol corpus
BOM-less UTF-16LE	typed unsupported-format refusal	anchor not found among NULs	lossy transcode or downstream parser disagreement
emoji offset	explicit position unit and byte conversion	off-by-one, rejection or panic	adjacent identifier/operator/literal edited under a mismatched preview
NFC/NFD	exact phase preserves identity	fuzzy ambiguity	visually equivalent but byte-distinct target selected first
stale source	digest precondition refuses	last writer wins	reviewed candidate overwrites a different source version

“Worst case” here is threat modeling, not a claim that I reproduced a production
exploit.

I also looked for the iPhone text-crash story

The remembered iPhone examples are real. Apple's
iOS 11.2.6 notes say that certain
character sequences could cause apps to crash. Apple has also documented
crafted-text CoreText denial-of-service issues, including
CVE-2017-2461 and
CVE-2020-9829.

But I do not use those incidents as evidence for the edit-kernel bug.

They are an adjacent layer: text shaping and rendering availability. Apple's
public records do not establish UTF-8/UTF-16 offset conversion or file mutation
as the root cause.

My bounded synthetic test used up to 32,768 combining marks. Node segmentation
plus normalization and Python normalization completed in roughly one
millisecond on this machine. No renderer was invoked, and no historical crash
payload was used.

The correct conclusion is narrower: complex text is untrusted input for the
diff/report renderer too. Run that renderer with resource bounds and crash
isolation, but do not confuse a renderer test with byte-mutation integrity.

The edit contract I now want

I split “safe editing” into three contracts.

1. Strict admission

validate model strings as Unicode scalar values, not only valid JSON;
define supported source encodings;
keep full source bytes authoritative;
treat samples as hints, never full validation;
bound source, anchor and replacement sizes.

2. Representation-preserving mutation

name every position unit at protocol boundaries;
locate exact byte anchors and distinguish zero, one and many matches;
construct candidates from byte spans;
preserve every byte outside declared spans;
keep normalization, fuzzy search and grapheme UI outside mutation authority.

3. Identity-bound settlement

pre-encode the complete candidate before destructive effects;
record source and candidate digests;
bind tests and review to the candidate digest;
recheck source identity immediately before settlement;
use atomic replacement where available;
read back or otherwise identify the settled candidate.

In shorthand:

owned source bytes
  + exact byte spans
  + candidate bytes
  + source/candidate identities
  + freshness check
  + atomic settlement

Bytes alone are not the novel idea. The composition is.

What surprised me most

Four things changed my initial mental model:

JSON validity is not Unicode scalar validity.
Rust removes one state, not the need for an explicit decoder policy.
Strict UTF-8 validation does not identify BOM-less UTF-16LE.
Atomic candidate settlement does not prove which source produced it.

The emoji was only the first visible clue.

The deeper issue was the same one I look for in network systems: where do two
components stop agreeing about the bytes they are authorizing?

If you maintain an AI code editor, I would add these fixtures before adding
another fuzzy matching strategy:

invalid UTF-8 near the start and beyond any sample window;
lone surrogates through the real JSON tool boundary;
emoji positions labeled as UTF-8, UTF-16 and code points;
NFC/NFD duplicates;
UTF-8 BOM, mixed CRLF/LF/bare CR and missing final newline;
BOM and BOM-less UTF-16LE;
a source change between review and settlement;
bidi, zero-width and confusable diagnostics in the review UI.

They are tiny inputs. They exercise surprisingly large assumptions.

Research disclosure: This article combines pinned source inspection with focused
standard-runtime probes. I did not execute complete third-party coding-agent
products, reproduce an iPhone payload, or establish a CVE in an agent editor.
The three-part contract above is the design target for my harness, not a claim
that its complete live path already enforces it.

AI-assistance disclosure: AI tools assisted with source navigation, probe
implementation, drafting and independent editorial review. I reviewed the
resulting evidence and take responsibility for the claims and conclusions.