DEV Community: ii-x

CSV to JSON: the three cases that break every naive parser

ii-x — Wed, 04 Nov 2026 01:48:04 +0000

A CSV file is not comma-separated. It is comma-separated, with a quoting ruleset that almost every hand-written parser ignores. Three inputs break the naive version, and all three fail the same way: your columns silently shift by one.

Case 1 — a comma inside quotes

sku,name,price
00127,Widget, small,45.20

Naive line.split(',') gives you four fields for three columns. The price lands in a fourth column you do not have, and 45.20 disappears.

The rule: a field may be wrapped in double quotes, and inside those quotes a comma is literal data.

function parseLine(line) {
  const out = [];
  let cur = "";
  let inQuotes = false;
  for (let i = 0; i < line.length; i++) {
    const ch = line[i];
    if (inQuotes) {
      if (ch === '"') {
        if (line[i + 1] === '"') { cur += '"'; i++; }   // escaped quote
        else inQuotes = false;
      } else cur += ch;
    } else if (ch === '"') {
      inQuotes = true;
    } else if (ch === ",") {
      out.push(cur); cur = "";
    } else cur += ch;
  }
  out.push(cur);
  return out;
}

Case 2 — doubled quotes are an escape, not a close

sku,title
00127,"9"" x 12"" Poster"

That single field is 9" x 12" Poster. If you treat the second " as "closing quote", the rest becomes garbage and — worse — the parser never re-enters quote mode, so every following comma splits.

The tell is in the bytes: "" inside a quoted field is always one literal ".

Case 3 — the BOM nobody mentions

Excel on Windows writes UTF-8 files with a byte-order mark: EF BB BF before the first character.

EF BB BF 73 6B 75 ...   →   "sku"

Your first header becomes \uFEFFsku, so row["sku"] is undefined for every row. Nothing throws. You just get a JSON file full of null.

Two lines fix it:

function stripBom(s) {
  return s.charCodeAt(0) === 0xFEFF ? s.slice(1) : s;
}

Case 4 — CRLF, because it is always there

split("\n") leaves a trailing \r on every line, so your last column is "45.20\r". parseFloat handles it, JSON.stringify gives you "45.20\r", and your downstream comparison fails.

Normalise first: text.replace(/\r\n?/g, "\n").

The order that actually works

Strip the BOM.
Normalise line endings.
Split into logical records — respecting quotes across newlines (a quoted field may contain \n).
Parse each record with the quote state machine.
Only now map the first row to headers.

Step 3 is the one people skip. A quoted field containing a newline is legal CSV, and it is how every product description with a bullet list breaks.

What I checked

Ten assertions on a fixture built from real supplier exports: quoted commas, doubled quotes, BOM header, CRLF, an embedded newline in a description, and a row with trailing empty cells. All ten fail on the naive parser and pass on the one above.

The limitation worth stating: this is not a full RFC 4180 implementation. It does not handle a mix of , and ; delimiters (which European Excel uses for decimal commas), and it does not attempt to sniff the delimiter. If your files come from one tool, hard-code the delimiter; if they come from users, sniff it on the first line before parsing.

If you just want to convert a file right now without writing this: IIX Lab's CSV to JSON converter runs the same algorithm in your browser and never uploads the file. And if you are fighting the next problem after conversion — duplicate rows and leading zeros eating your SKUs — that is written up in the dedupe notes.

Extracting text from a PDF without a PDF library: the T* trap

ii-x — Mon, 02 Nov 2026 07:27:51 +0000

I needed to pull transaction rows out of a bank statement PDF, inside a browser, with no server and no 1 MB pdf.js bundle. What I learned is that the hard part is not reading the bytes — it is that a PDF has no concept of a line.

The naive approach, and why it returns one giant string

A simple text extractor walks the content stream and collects whatever sits inside (...) Tj and [...] TJ operators. That part works. Then you get this:

ACME BANK 01/05/2026 GROCERY STORE #12 45.20 01/06/2026 SALARY 2,450.00 CR 01/07/2026 ELECTRIC 120.75

Eight transactions, zero newlines. T* is a layout instruction — it moves the cursor, it does not emit a character. If your parser splits on \n, you get one row and you will spend an hour thinking the extractor is broken.

Fix one: split on a business anchor, not on whitespace

Do not try to reconstruct lines from PDF operators. Find a token that only appears at the start of every record. For statements that is the date.

// collect every date occurrence, then slice between them
const anchors = [];
for (const pat of DATE_PATTERNS) {
  for (const m of text.matchAll(pat)) anchors.push({ start: m.index, end: m.index + m[0].length, date: m[0] });
}
anchors.sort((a, b) => a.start - b.start);

for (let i = 0; i < anchors.length; i++) {
  const seg = text.slice(anchors[i].end, anchors[i + 1]?.start ?? text.length);
  // seg is one transaction, minus the date
}

The second benefit: a record that has no date is not a transaction, so it gets dropped for free.

Fix two: the amount regex has to defend against the previous row

This is the bug that cost me the most time. Amounts sit directly against the next date:

... GROCERY STORE #12 45.2001/06/2026 SALARY ...

A naive \d+\.\d{2} matches 45.2001. You then record a transaction worth four hundred and fifty point two — silently, with no error.

Two things fix it, and you need both:

Strip dates out of the segment before looking for amounts, so 01/06 cannot be read as a number.
Put boundaries in the regex so a number cannot start or end inside another number:

const AMOUNT_RE = /(?<![\d/])(?:[$€£]\s?)?(-?\d{1,3}(?:,\d{3})+(?:\.\d{2})?|-?\d+\.\d{2})(?![\d/])/g;

That (?<![\d/]) is doing the real work. Without it, 45.2001 is a valid match.

Fix three: CR and DR are not signs

A statement writes 2,450.00 CR. Your parser sees 2,450.00 and throws away the CR, so a deposit becomes a withdrawal. Read the two characters after the amount:

const tail = seg.slice(matchEnd, matchEnd + 8);
if (/\bCR\b/i.test(tail)) amount = Math.abs(amount);
else if (/\bDR\b/i.test(tail)) amount = -Math.abs(amount);

What I tested, and one honest limitation

Ten assertions: 8 rows found, ISO dates, CR positive, thousands separators parsed, no digits left in the description, CSV row count, and total equals the hand-computed sum.

The limitation: this only works on PDFs that have a text layer. A scanned statement is an image, and I did not add OCR — the whole point of this tool was to stay small enough to run in a browser tab.

The smallest working extractor

function extractText(pdfBytes) {
  const out = [];
  for (const m of pdfBytes.matchAll(Buffer.from('stream\r?\n(.*?)endstream', 's'))) { /* ... */ }
  // inflate each stream, then pull the strings out of Tj / TJ
}

That is roughly 30 lines with zlib. No dependency, no WebAssembly, no worker.

If you are debugging a similar extractor: the failure mode is almost never the inflate step. It is the assumption that the text has structure it does not have.