Table OCR in the browser: what Tesseract gives you, and what it doesn't

I spent the last few weeks building a tool that turns a photo of a table into an editable Excel file, and I made one decision early that shaped everything else: the OCR had to run in the browser. No upload, no server, no per-page cost.

That decision is popular with users and painful for the developer. Here is what I learned.

Tesseract gives you words, not tables

The single biggest misconception about OCR is that it understands layout. It doesn't. Tesseract returns text blocks, and at the finest level each word comes with a bounding box and a confidence score.

For a paragraph, that is enough. For a table, it is not. You get a flat bag of words with coordinates, and you have to reconstruct the grid yourself.

The naive approach - join words with spaces and call it a row - fails immediately on anything with more than two columns, because a wide cell and a narrow cell produce the same output. You lose the only thing that made it a table.

So the real work is the step after OCR:

1. Group words into rows by clustering their vertical centers. Words whose y-coordinates overlap within a tolerance belong to the same row.
2. Within a row, sort by x, then decide which gaps are column separators and which are just word spacing.
3. Build a column model from the whole page, not from one row, or a single badly-aligned row will shift every column after it.
4. Merge cells that span multiple columns once the grid is known.

None of that is part of Tesseract. It is all inference on top of the boxes, and it is where most of the bugs live.

The client-side tax

Running OCR in the browser means shipping the engine to the browser. With tesseract.js that is a WASM build plus trained data, and it is measured in megabytes. On a fast connection it is fine. On a phone on a train it is a cold start the user will notice.

Three things made this acceptable:

• Lazy loading. The engine is only fetched when a file is actually dropped. The landing page stays small.
• Honest UI. A progress bar that reflects the real pipeline stage, not a fake animation. OCR on a dense table is slow, and pretending otherwise makes it feel broken.
• No upload claim that is actually true. If the file never leaves the device, say it in the first screen, not in a footnote. It is the main reason people pick a browser tool over an API.

The tradeoff is real: a server-side pipeline could use a heavier model and get better accuracy on messy scans. I accepted lower accuracy on hard inputs in exchange for privacy and zero marginal cost per conversion. For clean screenshots and flat photos, the difference is small. For a crumpled receipt shot at an angle, it matters.

Your data is not always a number

This one caught me off guard. Spreadsheet users care about what a value looks like.

If a cell shows 1,314.00, and your export pipeline parses it as a float, you get 1314. Formatting lost, trailing zeros gone, the thousands separator gone. For a financial statement that is a defect, not a rounding preference.

The fix was to treat sheet export formats differently:

• XLSX - the value is a typed number, and formatting is applied as a cell format. The cell stores 1314, displays 1,314.00.
• CSV - a plain interchange format. Numbers go out as numbers, because something else will parse them.
• TXT / DOCX - these are read by humans, so the original string is preserved verbatim. No float round-trip, no toAoa-style coercion that quietly rewrites the text.

Writing 1,314.00 into a text file is not lazy. It is correct for that destination.

Keep the structure, not just the text

Merged cells, column widths and number formats are the difference between "a file with the right words in it" and "a spreadsheet someone can keep working in". If the output loses merged headers, the user has to rebuild them by hand, and your tool saved them nothing.

On the table-reconstruction side that means carrying span information through the pipeline instead of flattening to a uniform grid early. It makes the intermediate representation uglier and the export code simpler, which is the right trade.

What I would tell someone starting today

• Do not evaluate an OCR engine on paragraphs if your real input is tables. Test it on the worst table you can find.
• Build the row/column reconstruction as a separate, testable layer. You will rewrite it several times.
• Decide early what each export format is for. "All formats are equivalent" is how 1,314.00 becomes 1314.
• Measure the client-side cost on a mid-range phone, not on your laptop.

The tool is at ocr2excel.com if you want to see the behaviour described above in practice - photos and scans of tables to XLSX, OCR running locally, no account required. Feedback on the hard cases (handwriting, rotated scans, dense financial statements) is genuinely useful, because that is where the reconstruction layer still loses.

---

If you have solved the merged-cell case in a more elegant way than "carry spans everywhere", I would like to read about it.