I work on Red Sheet at W Lab. Students in Japan write answers in red or orange pen, lay a sheet of red plastic over the page, and the answers disappear while the black questions stay. Slide the sheet down and you check yourself one line at a time. Red Sheet is an iPhone and iPad app that does this to a photo of your own notebook. This post is about the engine that finds the ink and hides it.
Why the paper trick works, and why a filter isn't enough
Red plastic passes red light and blocks the rest. Through it, white paper looks red and red ink looks red, so the ink has nothing to stand out against. Black ink reflects no red either way, so it stays dark. Green highlighter is the opposite case: it reflects almost no red, so through the sheet it turns nearly black and blocks the word under it.
The obvious way to copy that on a phone is to take the photo's red channel and tint it. I kept that version in the code as a comparison, and it fails on real photos. A phone photo of a notebook has a shadow across it, warm indoor light, greyish recycled paper. Red ink on a shadowed page is not the same value as the paper next to it, so it shows through as a faint ghost. A plastic sheet gets away with this because your eye adapts. An image doesn't.
So the app does not filter. It works out which pixels are answer ink, and guarantees those are gone.
Step 1: make the paper white
Before any colour decision, the lighting has to come out. The engine estimates the paper level at every point of the page: a morphological closing (a max filter, then a min filter) at quarter scale wipes out the writing and leaves the paper, and a box blur smooths it. Dividing each colour channel by that map sends paper to white under any light, and because the three channels are divided separately, yellowish paper comes out white too.
Pictures in a textbook would wreck that estimate, so coloured figures are found first, as blocks of cells that are saturated and span several hues, and filled in from their borders. Dark photos and panels count as figures too. That rule came from a bug: flattening turned a near-black panel into something that looked like marker.
A PDF page skips this step. It has no shadow to undo.
Step 2: decide what is ink, in Lab
Colour decisions are made in CIELAB, where distances roughly match what people see. Converting every pixel is slow, so the conversion is a lookup table with 64 levels per channel, built once.
A pixel is ink if it is far enough from white. Ink with little chroma is black if it is dark, and paper if it is light, which is how the grey anti-aliased edge of a black letter is kept out. The rest is coloured ink.
Fixed hue thresholds don't survive real pens, so the coloured pixels of each page are clustered with k-means, six clusters with a fixed seed so the result is repeatable. The page tells the engine what its own red looks like. Pale pixels on the rim of a stroke take the label of the saturated pixel beside them, so an orange stroke's faint edge belongs to the stroke and not to some pale marker of a similar hue.
Step 3: pen or highlighter is a question of shape
Hue alone can't separate a pink pen from a pink highlighter. Shape can. A highlighter is a solid band: its bounding box, counting the black text inside it, is mostly filled and about as thick as a character. A pen stroke is thin and its box is mostly empty.
That gives the classes the sheets care about. Red, orange and pink pen vanish under the red sheet. Green highlighter goes dark under it and blocks the word. Under the green sheet it is pink highlighter that blocks, and under the blue sheet a blue pen vanishes. A thin green stroke or a wide blue one is neither, and is never hidden.
Step 4: from pixels to answers you can tap
Hidden ink is not yet an answer. A handwritten answer is several strokes and often several words, and the app needs one box per answer to reveal on a tap.
The ink mask is dilated a little more along the writing direction than across it, and the blobs are labelled with a two-pass union-find. Then come the rules that make a box match what a person would call one answer:
- A blob taller than a line and a half is cut into lines at the valleys of its ink profile.
- Pieces on the same line, close together, with no black ink between them are one answer. The "no black ink between" part stops two answers in neighbouring table cells from merging.
- Specks are dropped. Long thin strokes are underlines: they get hidden, but they are not answers.
Writing direction is measured, not assumed, because Japanese notebooks are often vertical. If tall blobs clearly outweigh wide ones, the page is vertical, and the dilation and the sliding band turn with it.
Each answer gets a key from its ink class and its position on a coarse grid. When detection runs again, answers are matched by key, so anything the user changed by hand stays attached to the right one.
Step 5: the guarantee
Rendering is where "mostly hidden" becomes "hidden". For ink that should vanish, the mask is dilated by two pixels at full strength and one more at half strength, and those pixels are replaced with an estimate of the paper under them: the mean colour of nearby non-ink pixels, with the paper's own grain added back so the patch doesn't look painted. Only then is the sheet's tint applied. Black text and figures are never touched.
The sheet is computed once per page as a whole image. Moving the band only changes a clip rectangle, so dragging stays smooth. Revealing an answer draws the original photo's pixels back inside its box. Until the first render is ready, the page shows blurred under the sheet colour, so an answer is never on screen before it is hidden.
Keeping it fast and testable
The engine is a pure Swift package with plain loops over a pixel buffer, on a working copy whose long edge is 1200 pixels. The min and max filters are the van Herk / Gil-Werman kind, constant time per pixel whatever the radius. Analysis runs off the main thread, a few pages at a time, since each one holds tens of megabytes of per-pixel planes.
I wrote the engine first as a JavaScript mockup in a browser, where tuning a threshold and seeing the result takes a second, then ported it to Swift function by function. The Swift tests run twenty fixture pages, four kinds of page under five lighting conditions including shadow, warm light and recycled paper, and must match the mockup's output and find the same number of answers.
All of this runs on the device. The app's own code makes no network requests, and the camera permission text says what is true: images are never sent to us.
What it still gets wrong
A heading printed in red is hidden just like red pen. A real red sheet hides it too, so I left that alone and gave the user an eraser: tap a wrong hit and it shows for good, even after the page is detected again. For the opposite case, a word printed in black, the user drags a box or runs a marker over it and it hides under every sheet.
Red Sheet is on the App Store for iPhone and iPad as "Red Sheet - Photo & PDF Study": https://w-lab.app/apps/redsheet/
Top comments (0)