DEV Community

Yiğit Erdoğan
Yiğit Erdoğan

Posted on

An AI agent read a Japanese novel as 208,953 replacement characters

#ai #typescript #debugging #opensource

OpenClaw is an open-source AI agent. When a message contains a link, its link-understanding step fetches the page and hands the text to the model, so the agent can answer questions about it. I pointed it at Natsume Sōseki's Kokoro on Aozora Bunko, and the model received 208,953 copies of U+FFFD, the replacement character. The Japanese text was gone.

The bug

Aozora Bunko serves its texts as Shift_JIS. The link fetcher didn't care what the page declared and did this:

const content = new TextDecoder().decode(buffer).trim();
Enter fullscreen mode Exit fullscreen mode

new TextDecoder() with no argument is a UTF-8 decoder. Shift_JIS bytes are not valid UTF-8, so almost every character became �. Any page served in a legacy encoding (Shift_JIS, EUC-JP, GBK, Big5) went through the same thing.

Nothing failed loudly, which is the annoying part. The model got a wall of replacement characters, and the replies came back garbled too.

The decoder was already there

It turned out web_fetch, the agent's general web tool, already had a decoder that does this properly. It checks, in order:

  1. A byte order mark (UTF-8, UTF-16LE, UTF-16BE)
  2. The charset parameter of the Content-Type header
  3. For HTML and XML responses without a header charset: an <?xml encoding="..."> declaration, <meta charset>, or <meta http-equiv="Content-Type">

If the label is one TextDecoder doesn't know, it falls back to UTF-8 instead of throwing.

So the fix is mostly an export:

// src/agents/tools/web-shared.ts
export function decodeResponseBytes(res: Response, bytes: Uint8Array, truncated = false): string {
Enter fullscreen mode Exit fullscreen mode 
// src/link-understanding/apply.ts
import { decodeResponseBytes } from "../agents/tools/web-shared.js";

const content = decodeResponseBytes(response, buffer).trim();
Enter fullscreen mode Exit fullscreen mode

Writing a second decoder would have been easy. Keeping two of them in sync would not.

Why the first attempt stalled

An earlier PR for the same bug used a strict decoder. That fixes Shift_JIS pages but breaks pages that declare no charset at all, and plenty of pages don't declare one. Reusing the web_fetch decoder keeps those on UTF-8, exactly where they were before.

The test

The regression test serves a page from a local HTTP server with this header:

content-type: text/plain; charset=Shift_JIS
Enter fullscreen mode Exit fullscreen mode

and the body [147, 250, 150, 123, 140, 234], which is 日本語 ("Japanese") in Shift_JIS. The test expects the agent to see 日本語. On main it fails, with the fix it passes, and nothing else in the suite changes.

Proving it on real pages

Six bytes in a unit test is a start, not proof. I ran the change through the production link-understanding path, with the strict SSRF guard on, against live public pages:

Page Replacement characters before After
Kokoro on Aozora Bunko 208,953 0
阿部寛のホームページ 13 0

Then the maintainer, obviyus, re-ran it through a built Gateway: a real session, a real chat message, and a model fixture that records exactly what the model receives. On main the Shift_JIS turns came back garbled. With the fix the model got 阿部寛のホームページ, and the UTF-8 control page stayed the same. It merged as openclaw/openclaw#188766.

If your agent fetches web pages

Check how it turns bytes into text. Both new TextDecoder() and Response.text() decode as UTF-8 no matter what the server says, so an agent built on either one has this bug on legacy-encoded pages. You don't need a library to fix it: read the BOM, the Content-Type charset and the document's own declaration, then fall back to UTF-8.

Top comments (0)