DEV Community

y4u
y4u

Posted on Originally published at uvp.y42u.net

EUC-JP, UTF-16, and a Suspicion of EBCDIC — Reading Legacy Encodings in 2026

#logging #encoding #unicode #devops

A log file from 1998 lands on your desk.

grep ERROR returns nothing. The file is 12 GB, and you can see the word ERROR on screen. When a search runs correctly and matches nothing, what do you suspect?

The answer isn't inside the command you typed.

Part 3 covered Shift_JIS (CP932), the encoding still in daily production use, and promised a sequel for EUC-JP, UTF-16 and surrogate-pair line drift. This is that sequel — one layer older and one layer newer, both united by the same property: the tools on your 2026 laptop don't read them without being told.

Four situations: an EUC-JP mainline system log, a UTF-16 Windows event export, an unidentifiable file from a mainframe, and lines that drift because of emoji. All four jam on the same single point.

Up front: UwView auto-detects UTF-8, Shift-JIS (CP932), EUC-JP and UTF-16, and when the guess is wrong you can switch interpretation in place without rebuilding the index. With Pro, a 47.73 GB file you have already opened reopens in 0.02–0.07 s with line numbers intact (measured on one setup; results vary — details at the end)

1. Reading an EUC-JP system log in 2026

Situation

A log from a system that has been running for twenty years. cat in a UTF-8 terminal produces garbage.

You've read Part 3, so you suspect CP932 and pipe it through iconv -f CP932. It's still garbage — just a different flavour of garbage. One run of symbols replaced by another run of symbols, and nothing that looks like text.

Why it happens

EUC-JP and CP932 both fail plausibly, in two bytes at a time.

EUC-JP kanji use bytes in 0xA1–0xFE for both halves. CP932 lead bytes live in 0x81–0x9F and 0xE0–0xEF. The ranges don't coincide exactly, but they overlap enough. Decode with the wrong one and a large share of bytes still map to some valid character. An error would tell you something; instead you get a different plausible-looking wrong answer.

EUC-JP also encodes half-width katakana as 0x8E plus one byte — a shape CP932 has no equivalent for. The more of those a log contains, the more convincingly wrong the mis-decode looks.

And a log you receive in 2026 has usually lived through a migration. Someone flipped the server locale from ja_JP.eucJP to ja_JP.UTF-8 on some Tuesday, which means the encoding can change partway through a single file.

Working around it, and where that stops

Detect, then convert.

nkf --guess app.log                       # ask for a verdict
file -i app.log                           # libmagic's opinion
iconv -f EUC-JP -t UTF-8 app.log | less   # if it survives, it was EUC-JP
head -c 4000 app.log | iconv -f EUC-JP -t UTF-8 >/dev/null && echo ok
Enter fullscreen mode Exit fullscreen mode

The fourth line is the practical one: feed a prefix and let success or failure decide.

Three limits. First, detection looks at the beginning of the file. In a migration-era log the head is EUC-JP and the tail is UTF-8, so a verdict based on the head is guaranteed to be wrong later. iconv accepts exactly one -f per file.

Second, every check costs a full pass. Bouncing between "maybe EUC-JP" and "maybe CP932" on a 12 GB log means reading 12 GB per hunch. On top of Part 3's point that a converted file is no longer the original, you now pay in waiting.

Third, you never get the reasoning. nkf --guess returns an answer, not evidence — not which bytes decided it. When the answer is wrong, there's no hint about what to try next, so you brute-force the candidate list.

2. A UTF-16 Windows log that Unix tools won't read

Situation

An exported Windows event log. grep -c Error returns 0.

wc -l gives an implausibly small number. grep says Binary file app.txt matches and refuses to show anything. file reports Little-endian UTF-16 Unicode text.

Why it happens

In UTF-16, an ASCII character occupies two bytes.

E isn't 0x45, it's 0x45 0x00. So the byte sequence 45 72 72 6F 72 that grep Error is hunting for never occurs in the file. Zero matches is the correct result, and nothing warns you.

wc -l misbehaves for the same reason. A line break is 0x0D 0x00 0x0A 0x00; the 0x0A is there to be counted, but a stray 0x0A 0x00 elsewhere in the text gets counted too. And with NUL bytes scattered throughout, most tools classify the file as binary — which is what Binary file matches really means.

The nastier variant: UTF-16 without a BOM exists. Some exporters omit it, and without those two bytes file becomes unreliable, sometimes returning just data or ASCII text.

Working around it, and where that stops

Convert first, then work normally.

xxd app.txt | head -3                          # look at the first bytes (FF FE = UTF-16LE)
iconv -f UTF-16LE -t UTF-8 app.txt > app_u8.txt
grep -a Error app.txt                          # force it past the binary check
tr -d '\000' < app.txt | grep Error            # strip NULs and read it anyway
Enter fullscreen mode Exit fullscreen mode

Do the first one first. Two bytes usually settle the question.

Three limits. First, you can't always re-export. If you can run Get-WinEvent | Export-Csv -Encoding UTF8 yourself, this is a non-problem. But if what you hold is evidence, or a one-shot export pulled from a departing employee's machine, there is no second copy — you need to read the original without converting it.

Second, tr -d '\000' is too blunt. The ASCII parts become readable and every non-ASCII character is destroyed — and nothing on screen says so. Looking fine while being wrong is the worst possible failure mode here.

Third, converting severs the byte offsets. As Part 9 argued, raw byte positions sometimes carry meaning. If a report has to state which byte of the original the corruption starts at, a position in the converted copy is useless.

3. Suspect EBCDIC, discover CP932, feel relieved, then get stuck

Situation

A file from a mainframe. cat shows nothing resembling text.

You suspect EBCDIC and run it through dd conv=ascii. Still unreadable. An hour later you discover it was ordinary CP932 all along. Relief.

The next day you get stuck on something else entirely. The file has no line breaks.

Why it happens

Files from that world are non-textual along axes other than the character set.

Several things at once. Fixed-length records — 250 bytes each, delimited by position rather than by newline. Space padding — short fields are blank-filled, so every grep hit drags a field of whitespace behind it. Shift-out / shift-in — if the sender still uses 0x0E / 0x0F to toggle between single-byte and double-byte modes, then different regions of one file are encoded differently.

Suspecting EBCDIC was the right first move. But running dd conv=ascii breaks the file a second time. If the source was CP932, applying an EBCDIC→ASCII table destroys the original bytes. That lost hour may have been spent staring at something you had already damaged.

Working around it, and where that stops

Look at the bytes. That's the only correct opening move.

xxd data.dat | head -20                    # raw bytes, before anything else
od -c data.dat | head -20
fold -w 250 data.dat | head                # wrap fixed-length records to look at them
tr -d '\016\017' < data.dat | iconv -f CP932 -t UTF-8 | less   # drop SO/SI
Enter fullscreen mode Exit fullscreen mode

Whether you type xxd first decides how the rest of the day goes.

Three limits. First, a file without newlines breaks the assumption every line-oriented tool makes. grep, awk and less all work in units of a line. Hand them 12 GB as one line and some implementations will try to hold that line in memory. fold gets you readable output, but line 12,043 of the folded view is not a coordinate in the original.

Second, SO/SI nesting can't be expressed as a single encoding flag. iconv -f CP932 assumes the whole file is CP932. A structure where single-byte and double-byte regions alternate under control-character toggles is outside that assumption. Strip them and it reads — and the record of which regions were double-byte is gone.

Third, one committed conversion and you can't go back. If the EBCDIC experiment had been dd conv=ascii > data.dat over the original, the investigation ends there. It's the most dangerous form of Part 1's "transform the file so you can open it."

4. Emoji, and the lines and columns that drift

Situation

Support-form text goes into the log verbatim. User input in 2026 contains emoji, naturally.

Your incident report cites "line 12,043." A colleague opens the file and finds a different line there. Or awk '{print $5}' is off by one field — but only on some rows.

Why it happens

"One character" has four definitions, and tools disagree about which they use.

Bytes, code points, UTF-16 code units, grapheme clusters. Emoji make all four differ. 😀 is four bytes in UTF-8, one code point, two UTF-16 units (a surrogate pair), one grapheme. 👨‍👩‍👧 is five code points including the joiners and one grapheme. When an application says "character 40," it doesn't say which definition it means.

The main culprit for drifting line numbers isn't emoji, though. It's raw newlines inside user input — write them to a log unescaped and one record becomes two or three lines. The second culprit is lone surrogates: when an application truncates a string "at 100 units," it can cut a surrogate pair in half. The result is not valid UTF-8, so downstream parsers and viewers either stop or substitute U+FFFD. The moment they substitute, the byte length changes and every column after it shifts.

Working around it, and where that stops

Pin down one definition and stay in it.

LC_ALL=C grep -n 'ERROR' app.log            # count in bytes, fast
grep -naP '[\x{1F300}-\x{1FAFF}]' app.log   # find lines containing emoji (PCRE)
python3 -c "import sys;[print(i+1,len(l),len(l.encode())) for i,l in enumerate(sys.stdin)]" < app.log
Enter fullscreen mode Exit fullscreen mode

The first is genuinely fast — LC_ALL=C skips multibyte interpretation entirely, and on large files you feel it.

Three limits. First, LC_ALL=C costs you non-ASCII search. Speed and readability trade against each other, and picking one loses the other. Wanting both on the same file means exporting a different environment and running twice.

Second, terminal column positions can't be trusted. Whether an emoji renders one cell wide or two depends on the terminal and the font, so the column you counted in less won't match awk's field position. You can't transcribe a visually counted column into a report.

Third, conversion tools make lone surrogates disappear. iconv halts, or discards them under //IGNORE; most viewers render U+FFFD. But those broken bytes are the evidence that an application truncated a string incorrectly. The processing that makes the file readable is deleting the thing you're investigating.

What all four had in common

Situation What you suspect first Where it actually jams Where general tools stop
EUC-JP system log Wrong encoding picked It changes mid-file One -f per file, no more
UTF-16 Windows log Your grep syntax ASCII became two bytes Zero matches without conversion
Mainframe extract Whether it's EBCDIC No newlines, SO/SI toggles Line-oriented assumptions collapse
Emoji and drift The emoji themselves Four definitions of "character" Fixing one definition loses another

Read the third column downwards and it resolves. Every one is a case of the tool deciding how to interpret the bytes before you get a say. iconv commits a conversion, grep decides it's binary and goes quiet, tr throws away the NULs, the viewer swaps broken bytes for U+FFFD. Every one of those behaviours is helpful, and none of them is reversible.

Then the fourth column. General tools stop at the exact moment you think "let me look at that again, interpreted differently." On a 12 GB file, that round trip is one full scan each time.

Three things you need.

  • Switching interpretation and comparing. Not converting into a second file, but re-displaying the same original under a different reading. If each switch triggers a re-read and an index rebuild, you won't make that round trip more than twice.
  • One definition of line and position. "Line 12,043" should point at the same line in everyone's environment. Seeking by byte and losing the line number showed up in Part 10; with legacy encodings you also have to answer "what counts as a line" first.
  • Unreadable things staying visibly unreadable. Lone surrogates, SO/SI control bytes, NUL padding — none of it tidied away. The fact that something is broken is the lead.

Back to the opening: a 1998 log, grep ERROR, zero matches. Zero isn't a failed search. It's the quietest possible report that the bytes are not what you assumed. Whether you reach for xxd at that moment decides how the rest of the day goes.

The tools I use

I build UwView (free), a viewer that displays, scrolls and searches a huge text file from the moment it opens. Indexing runs in the background and line numbers appear when it completes (most other viewers show only the head until indexing finishes). Nothing is converted and nothing is split: the original stays one file, unmodified. It has no edit mode, which makes opening a received evidence file a safe operation in itself.

  • Switch interpretation and compare: UTF-8, Shift-JIS (CP932), EUC-JP and UTF-16 are auto-detected, and switching manually takes effect immediately without rebuilding the index. Section 1's ping-pong between EUC-JP and CP932 stops costing a 12 GB scan per attempt.
  • Make reopening cheap: UwView Pro persists the index and compression, so every open after the first comes back with line numbers already there (measured at 0.02–0.07 s on a 47.73 GB text file; varies by environment). Trying one more hypothesis stops meaning waiting again.
  • Keep line numbers as coordinates on the original: nothing is extracted and nothing is folded, so "line 12,043" is line 12,043 of the original. Of section 4's drift problems, at least identifying the line returns to a shared coordinate.
  • Leave broken bytes alone: NUL padding stays NUL padding and control characters stay visible. The leads that sections 3 and 4 warned about losing remain on screen.

To be straight about it: UwView does not handle EBCDIC. Section 3's opening move belongs to xxd and dd; this tool's turn begins once you know it's CP932. UTF-16 is recognised via the BOM, but line splitting is based on \n — BOM-less UTF-16, and 0x0D 0x00 0x0A 0x00 line breaks, may not split cleanly, so treat UTF-8, Shift-JIS and EUC-JP as the primary targets. Region-by-region encoding switches via SO/SI aren't supported either. And there is no facility to convert an encoding and save it: this is a tool for reading, not for repairing.

And if a huge log is eating your disk and you want it compressed for storage while staying searchable at speed, give UwView Pro a look — persistent index, compressed-cache search, and ~1/9 storage make both reopening and searching a step faster (all OS, one-time or monthly). For legacy-encoding work, where you reopen the same file under one interpretation after another, a cheap second open converts directly into more attempts.

Links

From the developer: A full list of my apps, Kindle books and open-source work lives at GitHub: amru195704.

A note
This article is provided for reference and makes no guarantee of accuracy or completeness. Encoding-detection behaviour, conversion tables and control-character handling vary by tool implementation, version and locale settings. Record formats and the presence of shift codes in mainframe-sourced files depend heavily on the sending system's configuration. Measured figures come from one specific environment and are not a promise of the same result elsewhere. Command examples may need adjusting for your environment (GNU vs BSD, iconv and nkf implementation differences, whether your grep has PCRE support). If you spot an error or something inaccurate, please leave a comment and I'll check and correct it.

Top comments (0)