Running OCR entirely in the browser (and making scanned PDFs editable)

A few months ago I needed to change one line in a scanned contract. Every "free" PDF site I tried wanted me to upload the file to their server first. One let me edit for ten minutes and then asked me to pay to download the result.

That's a strange deal for a contract, a passport scan or a payslip. So I started building PDFkind, a PDF editor where everything happens in the browser. The hardest part by far was OCR: reading the text out of a scan, then turning it into real text you can edit.

This post is about how that works, what broke along the way, and the numbers I got. Everything runs client-side; the server only hands out static files.

The constraint: no server, no exceptions

The rule I set myself was simple: the document never leaves the device. No "we delete it after an hour", no API call to a cloud OCR service. That rules out the easy options and leaves two questions:

1. Which OCR engine can run in a browser tab with acceptable quality?
2. How big a download can I ask people to accept?

First try: Tesseract.js

Tesseract.js is the obvious choice. It's mature, easy to use, and works well on clean scans. On my test set it was fine for flat, well-lit documents.

Then I tried phone photos: slightly tilted pages, shadows, uneven lighting. That's what people actually have when they "scan" something. Tesseract mostly fell apart:

Document type	Tesseract.js	PaddleOCR (what I ended up with)
Poor-quality scans	3.5 to 6.4% of characters wrong	about 1%
Phone photos	unusable	10 to 14%

(Character error rate on my own small test set of French and English documents. Not a benchmark, but the gap was big enough to decide.)

The difference comes from how the text is found. Tesseract relies on classical layout analysis, which struggles once the page isn't flat and evenly lit. PaddleOCR uses a neural network to find text lines first, and that detector copes much better with real-world photos.

PaddleOCR in the browser with ONNX Runtime Web

PaddleOCR is normally a Python library, but its models can be exported to ONNX and run with onnxruntime-web in WebAssembly. I use the PP-OCRv5 mobile models:

• a detection model that finds text lines
• a recognition model for Latin-script languages (English, French, Spanish, German and so on)

Together with the WASM runtime that's about 20 MB. It's only downloaded the first time someone uses OCR, and it's cached for 30 days:

// next.config.ts
async headers() {
  return [{
    source: '/ocr/:file*',
    headers: [{ key: 'Cache-Control', value: 'public, max-age=2592000, stale-while-revalidate=86400' }],
  }];
}

One setting worth knowing about: I run ONNX Runtime with numThreads = 1. Multi-threaded WASM needs SharedArrayBuffer, which in turn needs the page to be cross-origin isolated (COOP/COEP headers). Those headers break a lot of third-party embeds. One thread is slower, but a page of OCR still takes a few seconds, which is fine.

OCR happens in two steps: find the lines, then read each line.

Step 1: finding the text lines

The detector (a DBNet model) takes the page image and returns a probability map: for every pixel, how likely it is to be text.

Preparing the input has a couple of traps. The image size must be a multiple of 32, and PaddleOCR models expect BGR channel order, not RGB, normalized with the ImageNet mean and standard deviation:

const MEAN = [0.485, 0.456, 0.406], STD = [0.229, 0.224, 0.225];
// resize so the longest side is at most 1600 px, rounded to multiples of 32
const k = Math.min(1, 1600 / Math.max(canvas.width, canvas.height));
const W = Math.max(32, Math.round((canvas.width * k) / 32) * 32);
const H = Math.max(32, Math.round((canvas.height * k) / 32) * 32);

const px = getPixels(canvas, W, H); // RGBA from a resized canvas
const plane = W * H, input = new Float32Array(3 * plane);
for (let i = 0, p = 0; p < plane; i += 4, p++) {
  input[p]             = (px[i + 2] / 255 - MEAN[0]) / STD[0]; // B
  input[plane + p]     = (px[i + 1] / 255 - MEAN[1]) / STD[1]; // G
  input[2 * plane + p] = (px[i]     / 255 - MEAN[2]) / STD[2]; // R
}
const out = await session.run({ [session.inputNames[0]]: new ort.Tensor('float32', input, [1, 3, H, W]) });

The probability map isn't a list of boxes yet. Turning it into lines is the post-processing that the Python library normally does for you, so I had to rewrite it in TypeScript:

1. Threshold the map (pixels above 0.3 count as text).
2. Flood fill to group connected text pixels into regions.
3. Drop regions that are too small or whose average score is below 0.6.
4. Compute each region's convex hull, then the minimum-area rotated rectangle around it. This is what makes tilted lines work.
5. Expand ("unclip") the rectangle, because the model predicts a slightly shrunk core of each line.

The expansion distance comes from the DBNet paper: area × ratio ÷ perimeter.

const d = (r.w * r.h * UNCLIP) / (2 * (r.w + r.h)); // UNCLIP = 1.6
const w = r.w + 2 * d, h = r.h + 2 * d;

The bug that cost me a day: my first version found the four corners of each box by sorting them by y coordinate. That works on a flat scan. On a tilted photo, the top-right corner of a line can be lower than its bottom-left corner, so lines got read upside down or mirrored. Recognition on photos was garbage and I couldn't figure out why.

The fix was to stop guessing corners from coordinates and derive them from the rectangle's angle instead. I keep the angle within ±45° (swapping width and height when needed), so the width always runs along the text line. Then the corners come out in a known order: top-left, top-right, bottom-right, bottom-left.

Step 2: reading each line

The recognizer expects one straight, horizontal strip of text, 48 pixels high. So each detected line, which may be tilted, has to be cut out and straightened.

Canvas can do this with a single affine transform. Map three corners of the detected box to the corners of the output strip, invert the matrix, and draw the page through it:

function crop(src: HTMLCanvasElement, quad: [number, number][], boxW: number, boxH: number, outH = 48) {
  const w = Math.max(8, Math.round((boxW / boxH) * outH)), h = outH;
  const c = document.createElement('canvas');
  c.width = w; c.height = h;
  const g = c.getContext('2d')!;
  const [p0, p1, , p3] = quad; // top-left, top-right, bottom-left
  // matrix mapping strip coordinates to page coordinates...
  const a = (p1[0] - p0[0]) / w, b = (p1[1] - p0[1]) / w;
  const cc = (p3[0] - p0[0]) / h, d = (p3[1] - p0[1]) / h;
  // ...inverted, so drawing the page lands the line flat in the strip
  const det = a * d - b * cc;
  const ia = d / det, ib = -b / det, ic = -cc / det, id = a / det;
  g.setTransform(ia, ib, ic, id, -(ia * p0[0] + ic * p0[1]), -(ib * p0[0] + id * p0[1]));
  g.drawImage(src, 0, 0);
  return g.getImageData(0, 0, w, h);
}

The recognizer outputs, for each horizontal position in the strip, a probability for every character in its dictionary. Turning that into text is CTC greedy decoding: take the most likely character at each step, merge repeats, drop the "blank" symbol.

// dictionary: blank first, space last (PaddleOCR convention)
const dict = ['', ...chars, ' '];
let text = '', prev = -1, conf = 0, n = 0;
for (let t = 0; t < steps; t++) {
  let best = 0, max = -Infinity;
  for (let k = 0; k < classes; k++) {
    const v = out[t * classes + k];
    if (v > max) { max = v; best = k; }
  }
  if (best !== 0 && best !== prev) { text += dict[best]; conf += max; n++; }
  prev = best;
}

The average probability of the kept characters gives a confidence score. I skip lines below 0.4: they're usually logos, stamps or noise.

Arabic needed a different approach

I'm from Morocco, and a lot of documents here mix French and Arabic, sometimes in the same line. PaddleOCR's Arabic recognizer was disappointing in my tests: it dropped digits and letters often enough to be useless for things like dates and ID numbers.

What worked was a hybrid:

• keep PaddleOCR's detector to find the lines, since it's robust to photos
• read Arabic lines with Tesseract in single-line mode, on a cleaned-up crop of each line

The cleanup makes a big difference with Tesseract. For each line strip I estimate the background with a max filter (the paper is the brightest thing around), blur it, divide the image by it to flatten shadows and uneven light, then stretch the contrast between two percentiles. Tesseract then sees near-black text on near-white paper.

Two more details:

• Line joining. The detector often splits a line at wide gaps, for example after a colon. For Arabic I join pieces that sit on the same row and are close together (gap under 2.5× the line height), then order them right to left. The gap limit matters: without it, a French heading on the left and an Arabic heading on the right got merged into one nonsense line.
• A Tesseract setting that silently drops spaces. I had preserve_interword_spaces: '1' enabled. On Arabic it removed the spaces between words entirely ("شهادةعمل" instead of "شهادة عمل"). Turning it off fixed it. If your Arabic OCR output has words glued together, check this first.

On my Arabic test documents the hybrid gets about 2 to 4% of characters wrong on scans and around 4% on phone photos. Not perfect, but very usable.

From "searchable" to "editable"

Recognizing text is half the job. The classic output of OCR is a searchable PDF: the recognized text is added as an invisible layer behind the image, so you can select and copy it while the page looks unchanged.

That's useful, but it doesn't answer the question people actually ask: "how do I change the text in my scan?" So the editor has an option to go further. Each recognized line gets:

1. a cover in the color of the paper, hiding the original pixels
2. a real text box on top, with the same words, roughly the same font, size, weight, slant and ink color

Getting that to look right is mostly careful measuring.

Paper and ink color. The paper color is the most common color along the edges of the line strip (bucketed to 5 bits per channel so noise doesn't matter). The ink color is the average of the darkest pixels inside the letters. Edge pixels blend with the paper, so averaging all of them would make the text look washed out.

Font size. I measure the height of the ink from the top of the letters to the baseline, then ask the browser how tall the same text is in a candidate font:

ctx.font = `100px ${family}`;
const m = ctx.measureText(text);
const size = inkHeightAboveBaseline / (m.actualBoundingBoxAscent / 100);

Using actualBoundingBoxAscent for the exact text matters. A line with no capitals or tall letters ("one more") is shorter than "Total", and a fixed cap-height ratio gets the size wrong.

Which font? I can't identify arbitrary typefaces, but I can tell serif from sans serif. For each candidate (Helvetica, Times), I draw the recognized text into an offscreen canvas, stretched to the same length as the original, and measure how well it overlaps the original's letter shapes (intersection over union). The best overlap wins. Since a page is usually set in one typeface, a family that clearly wins most lines also gets a bonus on close calls.

Bold. My first attempt compared stroke thickness, and it failed on blurry scans, where everything looks bold. What works is comparing the total amount of ink in a line with how much ink the regular-weight font would use for the same text. Blur spreads ink out but doesn't change the total. And because a faded scan affects every line alike, I compare each line with the page's median rather than an absolute threshold.

Italic. I shear the letter shapes by different amounts and keep the shear at which the vertical strokes line up best (the sum of squared column counts peaks). Upright text peaks at 0; italic peaks at around 0.1 to 0.25.

Width. Finally, a little letter spacing makes the new text exactly as long as the original, so it doesn't overlap whatever comes after it.

The result isn't pixel-perfect. Unusual fonts come out as Helvetica or Times, and handwriting is left alone. But for the common case (fix a date, a name, an amount on a scanned letter) it's close enough that the change is hard to spot.

What I'd tell someone trying this

• Test on phone photos early. Clean scans hide almost every problem.
• ONNX Runtime Web is production-ready, but you have to rewrite the pre- and post-processing that Python libraries normally hide. Budget time for it.
• Cache the models aggressively and load them only when the feature is used. Nobody should download 20 MB just to merge two PDFs.
• Measure with real documents and track a simple error rate. Several of my "obvious" improvements made things worse.

Try it

Everything above is live in PDFkind. It's free, with no signup. Open a scanned PDF in the editor, tick "Make scanned text editable", and you can check in DevTools that your file never goes over the network.

I'd love feedback, especially from anyone who has worked on OCR or PDF internals. Where does it break for you?

Comments

[Launch Gate]

For the paper-colored cover step, I'd test a line crossing a faint stamp or table rule. A rectangle that hides the old text can also hide non-text evidence inside the same box. Can the preview show exactly which pixels will be covered, with a per-line opt-out and an untouched source copy? A second export check would search/copy the PDF after changing a date, to make sure an old hidden OCR layer doesn't still expose the previous value. Those seem like separate acceptance checks from character error rate or visual font matching.

[Madhi Omar]

Really good points, thanks for going through it this carefully.

Each line actually becomes two separate things: a paper-colored cover and a text box on top. You see both in the preview before exporting, so if a cover eats part of a stamp or a table line you can resize it or just delete it for that line. And the original file is never touched, you always get a new PDF.

For the hidden OCR layer, I already strip any text that sits under a cover when exporting, so the old date won’t show up in search or copy-paste.

But you caught a real gap with the image. The cover hides the old pixels, but they’re still there in the embedded scan. I’m going to fix that by burning the covers into the image on export, the same way my redact tool works. And yeah, these need their own tests, separate from OCR accuracy.