DEV Community

Daniil Kornilov
Daniil Kornilov

Posted on

Reject malformed UTF-8 before it reaches your CSV parser

#abotwrotethis #webdev #javascript #programming

A CSV import can have the right number of columns and still contain damaged text. The damage may happen before the CSV parser receives a string.

JavaScript's TextDecoder replaces malformed byte sequences by default. That is useful when displaying imperfect text, but an import workflow may need to reject those bytes instead.

Reproduce the difference

Run this in a modern browser console or Node.js:

const bytes = new Uint8Array([0x61, 0xc3, 0x28]);

console.log(new TextDecoder("utf-8").decode(bytes));
// a�(

try {
  new TextDecoder("utf-8", { fatal: true }).decode(bytes);
} catch (error) {
  console.log(error instanceof TypeError);
  // true
}
Enter fullscreen mode Exit fullscreen mode

The 0xc3 byte starts a UTF-8 sequence, but 0x28 cannot complete it. The default decoder substitutes U+FFFD, the replacement character. With fatal: true, decoding fails.

This matters when a damaged value becomes a lookup key. A parser can accept customer_id,name and a row containing a�(,Ada without knowing that the identifier changed during decoding.

Put the check at the byte boundary

For a small browser upload, decode the original bytes before parsing:

async function readUtf8(file) {
  const maxBytes = 2 * 1024 * 1024;
  if (file.size > maxBytes) {
    throw new RangeError("Choose a file no larger than 2 MiB.");
  }

  const buffer = await file.arrayBuffer();
  return new TextDecoder("utf-8", { fatal: true }).decode(buffer);
}
Enter fullscreen mode Exit fullscreen mode

The size check happens before the file is read into an array buffer. This example reads the whole file; it is intended for bounded uploads, not large streaming imports.

Handle a decoding failure in the upload interface: keep the previous imported data, show an actionable error, and let the user export UTF-8 from the source application or explicitly choose another supported encoding.

Avoid automatically retrying several encodings until one returns a string. Successful decoding under another encoding does not establish what the file's author intended.

Keep the guarantee narrow

Strict decoding establishes that the bytes are valid for the selected encoding. It does not establish that the values are correct, that the selected encoding matches the source, or that the file is valid CSV.

An ASCII-only file, for example, cannot establish whether the source application intended UTF-8 or another ASCII-compatible encoding. A literal U+FFFD already present in valid UTF-8 also passes this check.

After decoding, validate CSV syntax, column counts, required headers, and domain rules separately. Show a preview before replacing existing data.

References: MDN: TextDecoder options, MDN: decode errors, and the WHATWG Encoding Standard.

Follow this DEV profile for more practical JavaScript examples that handle input errors.

Top comments (0)