Our viewfinder has no shutter button, and the four rectangles dimming the rest of the frame are not a hole in an overlay

Munchable has a camera screen with no shutter button. You point it at something with text on it, drag a box around the part you actually mean, and let go. The photo is taken on the lift, cropped to what you drew, and read.

The interaction is borrowed from Google Lens, and the first version of it took about forty lines. The version in the app is around two hundred, and almost all of the difference is other things on the screen that were quietly broken by a drag.

The box lives in shared values, the decision lives in JavaScript

Seven shared values: where the finger went down, where it is now, whether a box is showing, and the canvas size for clamping.

const pan = Gesture.Pan()
  .enabled(!busy)
  .maxPointers(1)
  .minDistance(4)
  .onBegin((e) => {
    downX.value = Math.min(Math.max(e.x, 0), viewW.value);
    downY.value = Math.min(Math.max(e.y, 0), viewH.value);
  })
  .onStart(() => {
    curX.value = downX.value;
    curY.value = downY.value;
    shown.value = 1;
    runOnJS(setDrawing)(true);
  })
  .onUpdate((e) => {
    curX.value = Math.min(Math.max(e.x, 0), viewW.value);
    curY.value = Math.min(Math.max(e.y, 0), viewH.value);
  })
  .onEnd(() => {
    runOnJS(finish)(downX.value, downY.value, curX.value, curY.value, viewW.value, viewH.value);
  });

onBegin and onStart are the pair worth understanding, and getting them the wrong way round is the most common bug in a drag.

minDistance(4) means the gesture does not activate until the finger has travelled four points. onStart fires at activation. onBegin fires at touch down. If you seed the origin in onStart, the box starts four points away from where the user put their finger, and on a slow deliberate drag it can be noticeably more. Recording the origin in onBegin and only showing the box in onStart gives you both: the box starts exactly under the finger, and no box appears for a tap.

Everything up to onEnd is a worklet, so the rectangle follows the finger frame by frame on the UI thread even if the JS thread is busy taking a photo. The decision at the end ("is this big enough to be a deliberate selection?") hops to JS with runOnJS, because that is where the code that can act on it lives.

Four rectangles, not a hole

Everything outside the box dims. The obvious implementation is one full-screen scrim with a transparent cut-out, which in React Native means either a mask or an SVG path, both of which are real dependencies and neither of which animates for free from a shared value.

So the scrim is four absolutely positioned views: above the box, below it, and the two strips to its left and right at exactly the box's vertical extent. Each is an animated style reading the same shared values.

const scrimLeft = useAnimatedStyle(() => ({
  opacity: shown.value,
  left: 0,
  width: Math.min(downX.value, curX.value),
  top: Math.min(downY.value, curY.value),
  height: Math.abs(curY.value - downY.value),
}));

Math.min on both axes is what makes a drag from any corner work: there is no normalisation step and no "which direction is this going" branch, because the left edge of the box is simply the smaller of the two x values. The same four lines handle a drag up and to the left as handle one down and to the right.

It is more code than a mask and no dependencies, and it animates on the UI thread like everything else.

The ScrollView has to lose

This canvas sits inside a page that scrolls. A vertical drag across the viewfinder is, from the ScrollView's point of view, a perfectly good scroll gesture, and the ScrollView is the parent, so it wins by default. Which meant the first build of this screen let you draw boxes left and right, and scrolled the page whenever you tried to draw one around a line of text.

The fix is one line, and it is the inverse of the one most people reach for:

if (scrollRef) pan.blocksExternalGesture(scrollRef as ExternalGestureRef);

blocksExternalGesture tells the named gesture to wait for this one to fail before it may activate. Not simultaneousWithExternalGesture, which would scroll and draw at once, and not disabling scrolling, which would break the rest of the page. A swipe that starts on the viewfinder draws; a swipe anywhere else on the page scrolls exactly as it did before. The ScrollView has to be the gesture-handler one, not the react-native one, for this to work at all.

A screen reader user cannot drag a box

This is the part I want to actually argue for, because the easy version of accessibility here is to add a label to the viewfinder and move on, and that produces a labelled control that cannot be operated.

VoiceOver and TalkBack intercept touch. A blind user has no way to drag a rectangle around a region they cannot see, and even if they could, "around the bit you mean" presumes they know where it is in the frame. The gesture is not adaptable. It has to be replaced.

<View
  accessible
  accessibilityRole="image"
  accessibilityLabel={hint}
  accessibilityHint="Double tap to read everything in view"
  accessibilityActions={[{ name: 'activate' }]}
  onAccessibilityAction={(e) => {
    if (e.nativeEvent.actionName === 'activate') readWholeFrame();
  }}
>

The whole canvas is one accessible element, and activating it reads the entire frame instead of a selection. readWholeFrame sets the box to the full view and calls the same callback the drag does, so there is one code path downstream and no "accessible mode" to maintain separately.

That costs something real. A selection is the thing that makes the read accurate, so the no-drag path is working with more text and a worse chance of getting a clean result. It is still enormously better than a control that silently does nothing, and it is honest: the hint says what the action does rather than describing a gesture the user cannot perform.

Mapping a finger to a pixel

The crop is the only maths, and it is short because of a platform fact worth checking in your own app: the camera preview is a centre-cropped cover fit of the sensor frame on both platforms (expo-camera uses resizeAspectFill on iOS and CameraX FILL_CENTER on Android, with preview and still capture sharing an aspect ratio). So a point on screen maps into the photo through one uniform scale and a centring offset:

const scale = Math.max(view.width / photo.width, view.height / photo.height);
const offsetX = (view.width - photo.width * scale) / 2;
const offsetY = (view.height - photo.height * scale) / 2;

Math.max is the cover fit. Math.min would be a contain fit, which would be wrong in a way that only shows up on devices whose preview aspect ratio differs from the one you tested on, which is the worst category of bug to ship.

The whole geometry module is pure functions over plain sizes and rects, with no camera and no React in it, so the mapping can be unit tested without a device. The functions return null when the mapped area collapses below 8 pixels on a side, so a stray flick asks the user to try again instead of sending a sliver off to be read.

What this is for

The reason a crop is worth two hundred lines: the app answers a question about whether a given food suits a given gut condition, and it does that from the words printed on the pack. Narrowing the read to the panel you pointed at is the difference between an answer and a shrug.

You can see the public, non-camera side of the same thing on the website:

• The gastroparesis page is the kind of profile a scan is answering against.
• The answers index has 373 pages built from the same content the app uses, so you can read one before you ever install anything.
• The about page explains what the app does and does not claim, which is the context for why we would rather read less text accurately than more text badly.