Most of Munchable is a barcode scan: point the phone at a pack, get an answer about whether it suits your gut. Sometimes the barcode is one we do not have yet, and then the app asks you to photograph the ingredients panel instead. That screen is 1,256 lines, and almost all of the interesting decisions in it are about one question: will the small print come out sharp enough to read?
The camera is a card on an ordinary page
* THE SCREEN IS AN ORDINARY MUNCHABLE PAGE. Heading, explanation and buttons
* are on the cream page in the app's own type, like every other screen, and
* the camera is a card in the middle of it. Nothing is drawn over the picture
* on a dark band. This is how the scanner apps people already know (Yuka,
* Google's document scanner) lay out a capture step, and it is what lets the
* page say, in a real heading, why the camera is open.
The default instinct is a full-bleed viewfinder with white text floating over it, because that is what a camera app looks like. But this is not a camera app, it is one step in a flow, and a step that cannot explain itself is a step people back out of. A real heading on a real page can say why the camera is open. A caption on a dark gradient cannot say it as loudly, and it is also the layout where your instructions sit on top of exactly the thing you are asking the user to look at.
There are three colour constants for the only things allowed to touch the picture:
/** Deep cocoa. What is behind the picture while the camera warms up. */
export const CAMERA_BG = '#1B120C';
/** Words drawn on the picture (the corner guide's colour, the busy spinner). */
export const CHROME_TEXT = '#FFFFFF';
/** Decorative glyphs inside the dark card only; never used for words. */
export const CHROME_DIM = '#9A8776';
A separate constant for glyphs that may never be used for words is the kind of thing that only gets written after somebody (me) uses the dim one for a sentence and then tries to read it in daylight.
After the shutter, the still takes the live preview's exact place:
* The card holds the live preview and then, after the shutter, the frozen frame
* in exactly the same place with the same cover fit, so the picture simply
* stops.
Same rect, same cover fit, no transition. The user's mental model is that they froze the thing they were looking at, so anything that moves breaks it.
Autofocus off means autofocus on
This is the comment I would put on a poster:
/**
* FOCUS MODE. DO NOT "FIX" THIS TO 'on'. expo-camera's FocusMode
* reads backwards from its name, on both platforms: 'on' means
* focus ONCE and then lock, 'off' means keep focusing as the
* scene moves. Reading small print off a pack the user is still
* steadying needs the continuous one, so native gets 'off'. Web
* keeps 'on' because the browser implementation has nothing
* continuous to ask for.
*/
autofocus={Platform.OS === 'web' ? 'on' : 'off'}
autofocus="off" is the setting that keeps focusing. autofocus="on" focuses once and locks. Every reviewer's instinct, including mine six months from now, is that this is a typo, so the comment is in capitals and the platform split is right there to show it was deliberate.
That is lever one of three.
Lever two: which of the phone's cameras
On iOS, the lens that opens by default is the one that cannot do the job:
* expo-camera opens the plain wide angle lens unless told otherwise. That lens
* has a minimum focus distance of roughly a hand's width, and on its own it
* never switches to the ultra wide, which on every iPhone since the 13 Pro is
* the one that can focus on something held close. The virtual devices ("Back
* Triple Camera", "Back Dual Wide Camera") are the sets Apple's own app uses:
* a single logical camera that changes lens with distance, macro included, so
* a label held wherever the user's hands put it stays sharp.
This is why a label held at a comfortable reading distance comes out as a blur in a third-party camera and sharp in the stock Camera app: the stock app is not using a lens, it is using a virtual device that switches lenses for you. You have to ask for it by name, and the names are strings the OS hands you:
function pickLens(lenses: string[]): string | undefined {
const lower = lenses.map((name) => [name, name.toLowerCase()] as const);
const find = (needle: string) => lower.find(([, l]) => l.includes(needle))?.[0];
return find('triple') ?? find('dual wide') ?? find('dual');
}
Substrings, lowercased, in order of preference, with a comment explaining why it is not an equality check:
* Matched by substring, case insensitively, because the names are localised
* strings and an exact match would silently stop working in another language.
* A miss is harmless: the camera keeps its default lens.
A feature detection that degrades to the default is the right shape for this. The alternative, a device table, is a maintenance commitment that starts failing on hardware released after you stopped updating it.
The picture size is chosen with the same defensive spirit:
/**
* The largest of the picture sizes a camera reports, by pixel count.
*
* The strings are platform shaped, usually "1920x1080", so every entry is
* parsed defensively and anything that is not two numbers is skipped: some
* devices report named presets, and one odd string must not cost us the whole
* list.
*/
const match = /^(\d+)\s*[x×]\s*(\d+)$/.exec(size.trim());
Note the × in the character class. Somewhere out there is a device that reports its picture sizes with a multiplication sign, and a regex that only knows the letter x would have quietly dropped every option that phone had.
Lever three: a 221 line patch
Tap to focus is table stakes for any screen where a human points a phone at small print, and expo-camera does not ship it. So the repo patches it in:
/**
* Focuses (and meters exposure) at a point in the view's own coordinate
* space, in the same units a touch event reports. The camera returns to its
* configured focus mode a few seconds later. No-op on web.
* @platform android
* @platform ios
*/
async focusAtAsync(x: number, y: number): Promise<void> {
await this._cameraRef.current?.focusAt?.(x, y);
}
The Android half is where the real work is, and it is eleven lines of Kotlin:
fun focusAt(x: Float, y: Float) {
val cam = camera ?: return
val point = previewView.meteringPointFactory.createPoint(x, y)
val action = FocusMeteringAction.Builder(point, FocusMeteringAction.FLAG_AF or FocusMeteringAction.FLAG_AE)
.setAutoCancelDuration(4, java.util.concurrent.TimeUnit.SECONDS)
.build()
cam.cameraControl.startFocusAndMetering(action)
}
Two details earn their place. FLAG_AF or FLAG_AE: a tap moves focus and exposure, because on a glossy pack the thing that makes print unreadable is as often a blown highlight as a soft focus. And setAutoCancelDuration(4, SECONDS), which the patch's own comment explains:
* Auto-cancel returns the camera to continuous focus so the next scene is
* tracked again rather than stuck on the last tap.
A tap to focus without an auto-cancel is a tap to lock focus forever, which turns lever one off for the rest of the session. The coordinate mapping is handled by meteringPointFactory, which is the part I would have got wrong by hand: the preview is FILL_CENTER cropped, so the touch coordinates in the view are not the sensor's coordinates.
While writing this I noticed the comment in the app pointed at patches/expo-camera@57.0.4.patch and the file on disk is 57.0.5. A patch filename carries the version it applies to, so a dependency bump renames it and leaves every reference to it stale. That one is now fixed, and it is a decent argument for referring to "the repo's expo-camera patch" instead of a filename.
And one thing deliberately absent:
* There is no zoom control on purpose:
* digital zoom gives the reader nothing a crop does not.
The crop happens before the downscale
The user photographs the whole panel from a distance the lens can actually focus at, then draws a box on the frozen still. Only the box is uploaded, and the order of operations is the whole trick:
/** Long edge for uploads: bigger effective font for the model, smaller upload. */
export const UPLOAD_LONG_EDGE = 1568;
/**
* Optionally crop, then downscale and JPEG compress a captured photo for
* upload. The crop is applied first so a small selection keeps every sensor
* pixel it has instead of being shrunk along with the whole frame.
*/
Downscale first and then crop, which is the order you get if you resize on capture, and a tenth of the frame arrives as a tenth of 1568 pixels. Crop first, and that same tenth arrives at up to 1568 pixels on its long edge: same upload size, roughly ten times the pixels per letter.
The still the box is drawn on is kept deliberately, and its type says exactly how deliberately:
/**
* A frame straight off the shutter, before any crop or downscale.
*
* It is the same OS managed cache file `takePictureAsync` always wrote, kept
* for a few seconds longer, and it is still never written to the photo library
* and never logged.
*/
export interface RawPhoto {
"The same file, kept for a few seconds longer" is a precise claim about what changed when the crop step was added, and it is the kind of sentence that makes a privacy page writable later:
* PRIVACY: photos are transient end to end. They are downscaled into the OS
* managed cache, uploaded from memory, discarded server side, and never saved
* to the photo library or logged. Conditions never ride along with a photo.
Three images per capture, because packs curve:
* A bottle or a pouch wraps its ingredients around a curve, so the list simply
* does not fit in one flat shot.
Checking the photos is its own page
* Photos are checked on their own page afterwards, at full size with a
* labelled Remove under each, rather than as thumbnails on the live camera:
* checking whether words are readable is a job for a big picture and a calm
* page, not a 72pt tile next to a shutter.
A thumbnail strip answers "did I take a photo". It cannot answer "can the words be read", which is the only question that matters here, and if the user cannot answer it they find out two screens later.
Backing out of the flow asks first, because by then there is work to lose:
const ok = await confirm({
title: 'Leave without reading the label?',
message: 'The photos you have taken will be discarded. Nothing has been saved yet.',
confirmLabel: 'Leave',
cancelLabel: 'Keep going',
destructive: true,
});
"Nothing has been saved yet" is there because the other half of this flow is a contribution to shared data, and the honest thing to tell somebody leaving is that walking away costs them their photos and costs nobody anything else.
The one thing the user types
The flow ends with the product's name, and the text read off the label is never offered for editing:
* There is no review-and-edit step for the TEXT: every shared row is an
* unedited machine read backed by an OCR receipt, which is what makes captures
* comparable across users and keeps hand edits out of the shared data. The
* NAME is the one thing typed: it used to come only from what the reader
* happened to see on the pack, and when the photo was of the back, the product
* entered the catalog as "Product <barcode>" or as its first ingredient.
That is the trade. Editable text would improve any single capture and make the whole set incomparable, because there would be no way to tell a corrected read from an invented one. A name is different: a human holding the pack is simply a better source than a photograph of its back, and "named by the person holding it, with the reader's guess prefilled" is both more accurate and less typing.
If the server cannot save a read, the user still gets their answer on that device, with a banner that says so and a retake. Ten named reasons feed that banner, and not one of them is allowed to be a silent failure.
The public side of the data all this feeds is readable without the app: the ingredient answers, the hidden gluten list, and the eight conditions the app checks against.
Top comments (0)