In the first post of this series we looked at what Tesseract can and cannot do for table OCR in the browser. The short version: it gives you text with coordinates, and that's it. No columns, no rows, no merged cells.
This post is about the gap between those two worlds. Given a pile of (text, x, y, width, height) boxes, how do you rebuild a grid that a spreadsheet can actually open? We went through this building OCR2Excel, and every single rule below exists because a real invoice or report broke it first.
Why "just sort by y, then by x" fails
The naive approach — sort boxes into lines by their y coordinate, then order each line by x — works on screenshot-perfect tables and fails on real documents. Three reasons:
-
Superscripts, dollar signs and footnote markers sit above the baseline. A
$rendered slightly higher than the digits it belongs to becomes its own "line". - Multi-line cells. A product description wrapping over three lines produces three separate y-bands. Sort by y and you get description fragments interleaved with the numbers from neighboring rows.
- Hand-converted PDFs. Text extracted from a PDF (as opposed to OCR of a scan) has arbitrary internal ordering — the visual grid is only in the coordinates, and sometimes not even there.
So the reconstruction has to be tolerant: lines are clusters, not slices.
Rows: cluster on vertical overlap
Instead of a fixed line height, treat two boxes as being "on the same row" if their vertical spans overlap by more than half of the smaller box's height. Then take the connected components of that relation.
function sameRow(a, b) {
const overlap = Math.min(a.y + a.h, b.y + b.h) - Math.max(a.y, b.y);
const minH = Math.min(a.h, b.h);
return overlap > minH * 0.5;
}
Union-find over all boxes with this predicate, sort each component by its median baseline, and you get rows that survive superscripts (they overlap their number) and survive moderate skew. What they don't survive is staircase tables — zebra-striped layouts where alternating rows are vertically offset by half a line. Those exist in the wild (some ERP exports produce them), and the honest answer is that they need a different heuristic, not a tuned threshold.
Columns: the histogram of x-edges
For columns, looking at box starts is a trap, because right-aligned numbers (which is what financial columns are, almost always) all have different widths and therefore different left edges.
What's stable is the gap structure. Project every box onto the x-axis and build a boolean coverage array: for each pixel column, is any box covering it? Column separators are the runs of empty space that persist across most rows.
// coverage[x] = true if any box spans x
// gaps = maximal runs where coverage[x] === false
// keep gaps wider than ~0.4 * median character width
Filter the gaps by width — anything narrower than roughly half a character is kerning noise, not a separator. What's left gives you column boundaries that work for right-aligned, centered, and left-aligned content alike, because you're counting the emptiness between things rather than the alignment of the things themselves.
This is also the step where you catch the classic invoice pathology from the second post in this series: a 9.75 that OCR read as 975 often shows up as a suspiciously narrow box. If your grid is built from gaps, you can attach a per-cell "confidence" based on how cleanly the cell's boxes sit inside their boundaries, and flag the row instead of silently shipping a wrong number to a spreadsheet.
Merged cells: absence of evidence is the evidence
Once you have a grid, merged cells are surprisingly easy to detect — they show up as empty cells. If row 5 has boxes covering columns 1–2 and nothing in column 3, but column 3 is populated in rows 4 and 6, that's either a merged cell or a genuinely empty field, and you cannot always tell them apart.
Our rule of thumb, tuned on real financial documents:
- If the empty cell is bounded left and right by boxes whose text continues a sentence — merge it (treat as
colspan). - If the whole row is empty except one box — that's a continuation line (multi-line cell that clustering missed), not a merge.
- If the empty region is a full column in a band of rows — it's a section separator; emit a real empty row rather than guessing.
We deliberately do not try to be clever here. Emitting an empty cell that the user deletes in two seconds is a recoverable error. Emitting a guessed colspan that shifts every number one column left is not.
Validation beats perfection
The last piece is accepting that reconstruction will be wrong sometimes, and spending the effort on detecting wrongness instead of preventing it:
- Numeric column consistency: if 80% of a column parses as numbers and one cell doesn't, flag that cell, don't re-run OCR.
- Row width sanity: a row with 9 cells in a 6-column table is a clustering failure, not user data — re-cluster before showing anything.
-
Total check: if the document contains a word like
Totaland a numeric column sums to something wildly different from the number near it, surface that comparison to the user.
That last check has caught more real-world errors for our users than every OCR accuracy improvement combined. OCR will misread a digit from time to time — that's physics. But a receipt where the line items don't sum to the total is knowably wrong, and telling the user is the difference between a tool they trust and one they silently stop using.
Try it
All of this runs client-side — no upload, no server round-trip. If you want to see the pipeline applied to your own invoices and tables: ocr2excel.com.
Posts in this series:
Top comments (0)