Finding the four corners of a page in a phone photo is the step that turns a picture of a document into a copy of one. Everything downstream — the perspective warp, the flattening, the PDF — is easy once you know where the page is.
We do that client-side, in a browser tab, on a grayscale copy of the photo scaled to a long edge of 640 px. The pipeline is the textbook one:
grayscale → blur → Sobel → adaptive edge map → Hough lines → best convex quad
Two near-horizontal lines, two near-vertical lines, score every candidate quadrilateral, keep the best. Disclosure up front: this is from LensUp, which is ours. What follows is the part where the textbook pipeline quietly failed, and the one-line change that nearly doubled how often it works.
The failure: the page fills the frame and detection gets worse
The adaptive edge threshold is the standard trick. Take the histogram of Sobel magnitudes, pick the 90th percentile, call everything above it an edge. It adapts to lighting, exposure and camera noise for free.
It also has a property nobody mentions: it is set by whatever has the strongest gradients in the image. On a photo of a document, that is not the page border. It is the letter strokes. Printed text is black-on-white at high spatial frequency — it produces the sharpest gradients in the frame by a wide margin.
The paper-to-desk step, meanwhile, is weak. Measured on SmartDoc 2015, a page lying on light wood clears the surrounding surface by roughly 14 luma levels. That is the signal you actually need, and it is an order of magnitude quieter than the text sitting in the middle of it.
So the adaptive threshold, doing exactly what it was designed to do, sets itself by the text and prices the page border out of the edge map. And it does this worst in the case you most want to work: when the user fills the frame with the page, so text dominates the histogram and there is barely any desk left to argue for a lower threshold.
The fix is not clever. Cap it:
edgePercentile: 0.9, // gradient magnitudes above this percentile are edges …
minEdgeMagnitude: 14, // … but never below this …
maxEdgeThreshold: 30, // … and never above this: print must not price the page edge out
Measured on 300 real frames, with every other guard unchanged:
| threshold cap | pages located |
|---|---|
| 220 | 23.3 % |
| 30 | 41.0 % |
An adaptive threshold with a hard ceiling is no longer fully adaptive, which felt wrong when we wrote it. It is also the single highest-value line in the file.
Telling paper from a block of printed text
Capping the threshold gets the page border back into the edge map. It also lets a lot of rectangles in that are not pages — and the most dangerous one is a dark block of printed text, because it is genuinely a bright-surrounded rectangle with strong edges.
The first instinct is a contrast magnitude: require the inside of the quad to be meaningfully brighter than the outside. We had that set at 14 gray levels, which is the median paper-on-light-wood step from above.
That was the wrong knob. A threshold at the median of true pages rejects about half of the true pages. What actually discriminates is not how big the step is but which way it points:
minContrast: 8, // 40th percentile of (inside − outside) across each side, in gray levels.
// A page on light wood measures 14 at the median, so 14 rejected half of them;
// the polarity rule is what keeps text blocks out, not this magnitude.
The polarity rule is the actual test: a quad is a page only if it is brighter inside than outside, sampled at two depths, on all four sides. A text block fails that on the sides where more text continues past it. A page on a desk passes it everywhere.
There is still a case that beats local evidence: a large dark region of print can look like "the desk" if you only sample a few pixels out. So before applying a detection automatically — as opposed to just pre-positioning the handles — we probe farther outside, looking for the page continuing past that edge. If the page continues, the edge was not the page edge.
The guards that are product decisions wearing algorithm clothes
Three of the constants in that file are not computer vision. They are decisions about what to do when we are unsure, and they are the ones I would port to any similar project:
borderMargin: 0.02, // a side within 2% of the frame edge is "on the border"
applyMinAreaRatio: 0.25, // a smaller page is only proposed, never applied
minSideSupport: 0.42, // every side needs this much edge support to be applied
A quad with a side lying on the image border is rejected. If the "page edge" is the edge of the photo, the page is not fully inside the frame, and the honest default is the whole frame rather than a crop that silently cuts off whatever was outside it.
The largest well-supported quad wins. This sounds like a tie-breaker and is actually a correctness rule: it makes an inner rectangle — a photo printed on the page, a bordered table — lose to the page around it.
Below a quarter of the frame, a detection is only a proposal. It pre-positions the four draggable handles and waits. Auto-applying a small quad is how you produce the single worst outcome in this whole category: a confident crop that removes two thirds of somebody's passport.
That last one generalises. The result of this module is a proposal and never the truth; every handle stays draggable, and the caller decides from a confidence value whether to apply it silently. When detection fails outright it returns null, and null means keep the entire original image. A scanner that crops wrong is worse than a scanner that does not crop.
Two features that ship turned off
weakEdgeRecovery: 0, // Experimental: desktop cost is not yet justified by coverage gains.
contourRecovery: 0, // Correct proposals improved, but the measured latency gate failed.
Both of these work. Both improve the numbers. Both are disabled in the shipped defaults because they did not clear a latency budget on the hardware people actually hold. Leaving them in the file at 0, with the reason written next to them, has been more useful than deleting them — the next person to look at this does not have to rediscover that the idea was tried and why it is not on.
What I would take away from this
The general shape of the bug is worth more than the specific fix: an adaptive parameter adapts to the strongest thing in your input, and the strongest thing is often not your signal. Text beats page borders. Specular highlights beat document edges. The loudest object in the frame sets your threshold, and if your target is quiet, it gets deleted.
When an adaptive method underperforms, it is worth checking whether it is adapting to something you did not intend before reaching for a better algorithm. In our case the better algorithm was a ceiling of 30.
If you want to watch this run on your own photo, it is at a browser-based document scanner — the detection happens in the tab and your files are never uploaded, so the Network panel is a fair way to check that claim.
Top comments (0)