DEV Community

Cover image for An Emoji Made Me Audit Byte Offsets
Eugen
Eugen

Posted on Edited on

An Emoji Made Me Audit Byte Offsets

#ai #agents #programming #security

I was building my own harness for solving engineering tasks when an emoji broke
one of my assumptions.

The file was not malformed. There was no exotic exploit payload. The fixture
contained a perfectly ordinary Unicode string:

Rendered:    A😀éB
Code points: U+0041 U+1F600 U+0065 U+0301 U+0042
Enter fullscreen mode Exit fullscreen mode

It looks like four user-visible characters. In this exact NFD fixture, the
zero-based start offset of B is 4 Unicode code points, 5 UTF-16 code units in
JavaScript, and 8 UTF-8 bytes.

That difference is harmless until one component calculates a position and
another component gives that number mutation authority.

TL;DR: Text positions need explicit units. The NFD string above has 4
grapheme clusters, 5 code points, 6 UTF-16 code units and 9 UTF-8 bytes,
while B begins at zero-based offsets 4 / 5 / 8. Across local runtime probes
and pinned source inspection, I found that safe agent editing needs strict
admission, byte-preserving mutation, and review, tests and settlement bound
to exact source and candidate bytes.

I was building a task harness, not another chat UI

My goal was a task-oriented development harness:

conversation and design
  -> frozen task
  -> architect
  -> implementer
  -> tests
  -> reviewer
  -> verified result
Enter fullscreen mode Exit fullscreen mode

The conversation is useful upstream, but the implementation stage needs a much
stricter contract. Each role should receive only the context and artifacts it
needs, and a reviewer should approve the exact candidate that was tested.

I used Python to prototype the edit path quickly and was considering TypeScript
for more of the lifecycle. The Unicode fixture was a tripwire for the boundary
between them. Its total length has four equally correct descriptions:

4 grapheme clusters
5 Unicode code points
6 UTF-16 code units
9 UTF-8 bytes
Enter fullscreen mode Exit fullscreen mode

Nothing was corrupted. I caught this in a local, model-free test—not in a
customer outage or a breach. But it made me ask how far the mechanism could
travel, so I reconstructed the customer and adversarial scenarios below.

The first reconstructed failure needed no malicious input

Imagine a customer gives the harness this small configuration file:

# Deploy 🚀
timeout = 30
Enter fullscreen mode Exit fullscreen mode

The requested change is boring: set the timeout to 60.

Suppose the TypeScript orchestration layer computes a range from a JavaScript
string and the Python or Rust edit layer consumes the same number as a UTF-8
byte offset. Every position after the rocket now refers to different data in
the two components.

The best outcome is a clean range-boundary refusal. An ordinary failure is an
anchor miss or an off-by-one edit. The dangerous outcome is that a valid but
different adjacent token is changed while the preview was calculated in the
other coordinate system.

This customer story is a reconstruction, not an incident report. The measured
fact underneath it is the zero-based 4 / 5 / 8 start-offset split in the
fixture above.

One edit crosses several representations

An agent edit crosses several representations:

model JSON
  -> runtime string
  -> decoded source text
  -> matched span
  -> candidate
  -> diff shown to a reviewer
  -> bytes written to disk
Enter fullscreen mode Exit fullscreen mode

Every arrow can change the meaning of “the same text.”

The repository ultimately stores bytes. JavaScript strings are indexed as
UTF-16 code units. Python strings are indexed by code points. A UI usually wants
grapheme clusters. A fuzzy matcher may normalize text before comparing it.

Those are all valid choices in the right layer. The problem begins when a value
created in one coordinate system is consumed as another.

I later isolated the same counting problem in a smaller browser-facing example:
styled Unicode breaks character counters
when visible glyphs, code points, UTF-16 units, and UTF-8 bytes diverge.

Key takeaway: A numeric offset is not a contract until its coordinate
unit is part of the protocol.

Three runtimes, one malformed JSON string

I wrote three small model-free probes and ran this case on Node.js 22.17.0,
Python 3.14.4 and Rust 1.97.0. The fixture was the JSON string "\uD800", an
unpaired UTF-16 surrogate.

RFC 8259 explicitly warns
that JSON grammar can carry such a value even though it is not a Unicode scalar
value.

Here is what I observed:

Boundary Observed result
Node JSON.parse produced a one-unit string containing U+D800
Node Buffer.from(..., "utf8") silently encoded U+FFFD: ef bf bd
Python json.loads produced a str containing U+D800
Python strict UTF-8 encode raised UnicodeEncodeError
Python encode with errors="replace" emitted ASCII ?: 3f
Python surrogatepass emitted ed a0 80, which is not well-formed UTF-8
Rust char / String boundary U+D800 was not representable; strict UTF-16 conversion failed

“The JSON parsed” was not a sufficient admission rule in any of the three
runtimes.

Then, in a standalone Python script, I reimplemented a destructive ordering
pattern visible in one inspected source path:

open target with O_TRUNC
  -> encode candidate as UTF-8
  -> encoding fails on U+D800
Enter fullscreen mode Exit fullscreen mode

The Python probe raised UnicodeEncodeError, but the target was already empty.
When I encoded the candidate before opening the target, the same error occurred
and the original bytes stayed unchanged.

That is a small ordering decision with a large difference in failure semantics.

Here is one realistic way those primitives could compose. An upstream length
limit slices a JavaScript string through the middle of an emoji. JSON still
serializes the remaining surrogate code unit. A Python worker parses it, opens
the destination with truncation, and only then discovers that the candidate
cannot be encoded as strict UTF-8. The result can be an empty file even though
every individual step looked routine.

I reproduced the slicing, parsing and truncate-before-encode mechanisms, but
not that complete production chain. It is a reconstructed failure scenario and
a regression test I now want, not a claim about a real customer request.

Invalid UTF-8 can be repaired without asking you

The next fixture was four bytes:

61 ff 62 0a
Enter fullscreen mode Exit fullscreen mode

I asked the simulated editor to make an unrelated change: a to A.

A strict decoder rejected the source in Node, Python and Rust. A lossy decoder
produced this candidate in all three:

41 ef bf bd 62 0a
Enter fullscreen mode Exit fullscreen mode

The requested byte changed from 61 to 41. The unrelated invalid byte ff
also became the three-byte encoding of U+FFFD.

The edit can report success. The output is valid UTF-8. The visible diff may
focus on the requested line. Yet the tool has permanently changed bytes outside
the intended mutation.

I repeated the fixture with the invalid byte at offset 1001. Validation of only
the first 1000 bytes passed in all three runtimes; full validation failed. This
matters because a byte sample is useful for classification, but it is not proof
that the complete source is valid text.

This is where my network-security instincts changed the question. If I were
red-teaming this boundary, offset 1001 is exactly where I would place the byte:
just beyond the classifier's evidence window, but still inside the later lossy
decode. That becomes a validation-coverage gap if the sample is treated as full
admission. Sampling itself is not the problem; giving an advisory classifier
authority over bytes it never inspected is.

Again, no attacker sent this to my system. I constructed the input locally to
test the boundary.

Key takeaway: A sample can classify input. It cannot authorize mutation
of bytes it never inspected.

Strict UTF-8 is necessary, but it is not format detection

I also tested UTF-16LE.

A file with the ff fe BOM was rejected by strict UTF-8, as expected. But a
BOM-less UTF-16LE file containing ASCII-range text passed strict UTF-8 in all
three runtimes:

6f 00 6c 00 64 00 0a 00
Enter fullscreen mode Exit fullscreen mode

As UTF-8, that is valid text with four embedded NUL characters. As UTF-16LE, it
is simply old followed by a line ending.

The BOM-bearing form is not an exotic historical format. Windows PowerShell
has produced UTF-16LE files with a BOM through commands such as Out-File and
redirection; the
PowerShell encoding documentation
describes the version-dependent behavior. The harder BOM-less fixture above was
constructed for this probe; that PowerShell citation is not evidence that those
commands produced it.

A UTF-8-only edit tool does not need to support UTF-16. It does need to refuse
an unsupported format before mutation and prove that refusal left the source
unchanged.

This is the least dramatic scenario and probably the most practical one. A
customer can submit a legitimate PowerShell script created by an older Windows
tool, ask for a one-line change, and violate none of your stated assumptions.
The editor still needs a deterministic answer: preserve the declared encoding
or refuse before effect. Silently guessing and rewriting is not a customer
error.

Normalization can change “one match” into “two matches”

These two strings often render identically:

NFC: Ă©        -> c3 a9
NFD: e + ◌́   -> 65 cc 81
Enter fullscreen mode Exit fullscreen mode

They are canonically equivalent, but they are not byte-identical.

I placed one of each in a source. The NFD anchor had one exact match. After NFKC
normalization, the same logical anchor had two matches.

That means normalization did not merely make search friendlier. It changed the
cardinality of an authorized mutation.

Key takeaway: Exact byte matching owns mutation. Normalized or fuzzy
matching may propose candidates; if it changes zero/one/many cardinality, the
tool reports that fact instead of silently selecting the first result.

Unicode normalization is valuable.
Making it mutation-authoritative without an explicit contract is the problem.

Then I checked real coding-agent implementations

At first I suspected this was mainly a TypeScript/Python problem. Rust strings
cannot contain lone surrogates, so perhaps Rust editors avoided the whole class.

That hypothesis was wrong.

I inspected pinned source snapshots and traced decode -> locate -> candidate ->
write paths. I did not run the complete products, so these are bounded source
findings followed by standard-runtime reproductions of the selected primitives.

The table reports policy and ordering visible at pinned source snapshots. It is
not a list of confirmed product vulnerabilities, incidents or end-to-end runs.

Editor path Relevant behavior found in source
OpenAI Codex apply_patch strict String::from_utf8 rejects invalid update sources
Pi coding-agent edit Buffer.toString("utf-8") creates a lossy text authority
Deep Agents filesystem edit strict source decode; the inspected local write path orders O_TRUNC before the text write encodes the candidate
Grok Standard search/replace Rust explicitly chooses String::from_utf8_lossy; any later effect depends on the rest of its settlement path
Hermes default replace reads through cat; the default local process transport decodes with errors="replace", while the write path separately uses temp+rename

The useful conclusion is not “language X is safe.”

Language changes which failure modes are easy or impossible. The complete edit
contract decides whether they become file corruption.

Two Rust editors can choose opposite invalid-UTF-8 policies. A Python editor can
fail closed at decoding and still lose a file because encoding happens after
truncate. An atomic rename can settle a candidate correctly even when that
candidate was already derived from a lossy source.

The network-security analogy is parser differential, not packets

I am a network security engineer, so this shape felt familiar. HTTP/1.1 is
defined over octets, and
RFC 9112 warns that
parsing the message as Unicode too early can create security vulnerabilities.
The structural problem here also resembles a parser differential:

network path:
octets -> intermediary parser -> security decision -> origin parser

agent edit path:
file bytes -> runtime decoder -> model/reviewer decision -> filesystem writer
Enter fullscreen mode Exit fullscreen mode

I am not claiming that an AI edit bug is HTTP request smuggling. The analogy is
about disagreement at boundaries.

If the security layer approves representation A while the effect layer writes
representation B, the approval is attached to the wrong object.

That becomes especially important in a multi-agent harness. A test result on
candidate A cannot authorize candidate B. A reviewer approving a pretty diff
cannot authorize bytes that were regenerated later from a different source
snapshot.

Best case, ordinary failure, worst case

The fixtures helped me separate failure levels:

Case Best case Ordinary unsafe result Plausible worst case
invalid UTF-8 refuse before effect U+FFFD silently replaces unrelated bytes corrupted generated/signed fixture with a deceptively small visible diff
lone surrogate reject/pre-encode before open tool returns an encoding error silent replacement or an empty target after truncate
mixed BOM/newlines preserve untouched byte spans whole-file representation rewrite broken script, checksum or protocol corpus
BOM-less UTF-16LE typed unsupported-format refusal anchor not found among NULs lossy transcode or downstream parser disagreement
emoji offset explicit position unit and byte conversion off-by-one, rejection or panic adjacent identifier/operator/literal edited under a mismatched preview
NFC/NFD exact phase preserves identity fuzzy ambiguity visually equivalent but byte-distinct target selected first
stale source digest precondition refuses last writer wins reviewed candidate overwrites a different source version

“Worst case” here is threat modeling, not a claim that I reproduced a production
exploit.

I also looked for the iPhone text-crash story

The remembered iPhone examples are real. Apple's
iOS 11.2.6 notes say that certain
character sequences could cause apps to crash. Apple has also documented
crafted-text CoreText denial-of-service issues, including
CVE-2017-2461 and
CVE-2020-9829.

But I do not use those incidents as evidence for the edit-kernel bug.

They are an adjacent layer: text shaping and rendering availability. Apple's
public records do not establish UTF-8/UTF-16 offset conversion or file mutation
as the root cause.

My bounded synthetic test used up to 32,768 combining marks. Node segmentation
plus normalization and Python normalization completed in roughly one
millisecond on this machine. No renderer was invoked, and no historical crash
payload was used.

The correct conclusion is narrower: complex text is untrusted input for the
diff/report renderer too. Run that renderer with resource bounds and crash
isolation, but do not confuse a renderer test with byte-mutation integrity.

The edit contract I now want

I split “safe editing” into three contracts.

1. Strict admission

  • validate model strings as Unicode scalar values, not only valid JSON;
  • define supported source encodings;
  • keep full source bytes authoritative;
  • treat samples as hints, never full validation;
  • bound source, anchor and replacement sizes.

2. Representation-preserving mutation

  • name every position unit at protocol boundaries;
  • locate exact byte anchors and distinguish zero, one and many matches;
  • construct candidates from byte spans;
  • preserve every byte outside declared spans;
  • keep normalization, fuzzy search and grapheme UI outside mutation authority.

3. Identity-bound settlement

  • pre-encode the complete candidate before destructive effects;
  • record source and candidate digests;
  • bind tests and review to the candidate digest;
  • recheck source identity immediately before settlement;
  • use atomic replacement where available;
  • read back or otherwise identify the settled candidate.

In shorthand:

owned source bytes
  + exact byte spans
  + candidate bytes
  + source/candidate identities
  + freshness check
  + atomic settlement
Enter fullscreen mode Exit fullscreen mode

Bytes alone are not the novel idea. The composition is.

What surprised me most

Four things changed my initial mental model:

  1. JSON validity is not Unicode scalar validity.
  2. Rust removes one state, not the need for an explicit decoder policy.
  3. Strict UTF-8 validation does not identify BOM-less UTF-16LE.
  4. Atomic candidate settlement does not prove which source produced it.

The emoji was only the first visible clue.

The deeper issue was the same one I look for in network systems: where do two
components stop agreeing about the bytes they are authorizing?

If you maintain an AI code editor, I would add these fixtures before adding
another fuzzy matching strategy:

  • invalid UTF-8 near the start and beyond any sample window;
  • lone surrogates through the real JSON tool boundary;
  • emoji positions labeled as UTF-8, UTF-16 and code points;
  • NFC/NFD duplicates;
  • UTF-8 BOM, mixed CRLF/LF/bare CR and missing final newline;
  • BOM and BOM-less UTF-16LE;
  • a source change between review and settlement;
  • bidi, zero-width and confusable diagnostics in the review UI.

They are tiny inputs. They exercise surprisingly large assumptions.

Research disclosure: This article combines pinned source inspection with focused
standard-runtime probes. I did not execute complete third-party coding-agent
products, reproduce an iPhone payload, or establish a CVE in an agent editor.
The three-part contract above is the design target for my harness, not a claim
that its complete live path already enforces it.

AI-assistance disclosure: AI tools assisted with source navigation, probe
implementation, drafting and independent editorial review. I reviewed the
resulting evidence and take responsibility for the claims and conclusions.

References

Top comments (0)