Point your camera at a receipt and Claude reads it

Last post ended with an admission: after a real shop, nobody prices 30 items by hand. The prices already exist — printed on a crumpled receipt at the bottom of a shopping bag. So this post is the app's headline feature: photograph the receipt, let Claude's vision read it, and watch the list get priced and ticked in one pass.

It's also the feature that forced three firsts: the app's first native modules (goodbye, JS-only development), its first storage bucket, and — a day later — its first genuinely new data model of the money arc. And it delivered the project's two best war stories so far: an IAM error that looked impossible, and a save that failed silently for an entire session.

TL;DR — The pipeline is a chain of boundaries: the client captures and compresses; the image crosses AppSync as an S3 key (never base64); the Lambda only extracts (Claude vision, strict JSON, domain rules in the prompt); the client only matches (it owns the list); the user confirms in a review sheet with a total-reconciliation line; the stored Receipt is an immutable snapshot. War story 1: owner-scoped S3 rules don't cover Cognito group roles — every user assumed one, so every upload got AccessDenied. War story 2: Amplify's client returns { data, errors } without throwing, and a.json() fields want a string — a raw array failed validation invisibly until errors were surfaced.

(Part 14 of Building CannyCart, a voice-first shopping app I'm building in public. Self-contained — no earlier context needed.)

The pipeline, end to end

camera / photo library
  → resize ~1500px, JPEG q0.6        (expo-image-manipulator)
  → upload to S3                      receipts/{identityId}/{uuid}.jpg
  → parseReceipt({ key })             (custom mutation → Lambda)
  → Claude Haiku vision               (strict-JSON extraction)
  → review sheet                      (match · edit · confirm)
  → normal app mutations              (price + tick, or add as bought)

Two decisions shape everything downstream.

The image travels as an S3 key, never base64 through AppSync. GraphQL payloads have limits and costs; more importantly, uploading first means the original is kept. A receipt is a record — the key lives on the row for audit and re-scan. (The app's other photo feature, scanning a handwritten list, deletes its image after parsing — a list photo is a means to an end, a receipt is an end. Same pipeline, opposite retention policies, each on purpose.)

~1500px wide at 0.6 JPEG is plenty. Claude reads supermarket thermal print fine at that size, and it keeps upload plus vision latency tolerable on a phone connection.

This was also the feature where reload-only development ended: expo-image-picker and expo-image-manipulator are native modules with config-plugin permission strings, so the first camera feature cost a full EAS dev-client rebuild before it could be tested at all. Budget that before you demo.

The prompt is where the intelligence lives

The Lambda is 71 lines and most of the intelligence is English. Excerpts from the real system prompt:

• price is the UNIT price in the receipt's currency (a line's total divided by its quantity) …
• If a line is priced by weight (e.g. "0.62 kg @ £1.50/kg"), set quantity to the weight and unit to the measure ("kg").
• Skip non-item lines: subtotal, tax/VAT, total, change, tendered, card, loyalty/points …
• These receipts are UK, so read ambiguous numeric dates DAY-FIRST: "05/07/2026" is 5 July 2026, not 7 May.
• Receipts rarely print the currency code, so infer it: from the symbol (£=GBP, €=EUR …), from the retailer, or from the tax wording ("VAT" + £ = GBP, "MwSt" = EUR, "Sales Tax" + $ = USD). Only use null when there is genuinely no signal.
• If the image is not a readable receipt, return { "items": [], "store": null, … }.

Three of those earn their keep daily. The day-first rule exists because without it the model reads dates US-style and a July shop lands in May. The currency inference exists because receipts almost never print an ISO code — but they're full of indirect signals. And the typed empty result is the anti-hallucination valve: given a legal way to say "this isn't a receipt," the model stops inventing groceries from a photo of your cat.

The handler around it is defensive plumbing: strip the code fences the prompt already forbade, JSON.parse, throw with a truncated sample if that fails, and return JSON.stringify(parsed) — an a.json() custom mutation returns an AWSJSON string. Remember that last sentence; it comes back with a knife.

Extraction in the Lambda, matching on the client

The Lambda never sees your shopping list. It extracts; the client — which already holds the list in cache — matches. Each extracted line is normalised (same frequentKey normalisation the frequently-bought feature uses) and compared against unchecked items on the active list:

• Match → the item gets the price and a tick: updateItem({ price, checked: true }).
• No match → a new, already-bought item: added priced and checked, straight to Done.

Both are the keyed offline mutations from Part 7, so applying a receipt inherits optimistic updates and offline replay without any new plumbing — same dividend as Part 13, one feature later.

The review sheet is the gate before any of that runs: every line editable (price, quantity), removable, with a match badge — "prices **Milk* on your list"* vs "new item" — and a receipt-total vs items-sum reconciliation line at the bottom. If Claude missed a line, the two numbers disagree and you can see it; a silent gap is the one thing a money feature can't afford.

Later, a persisted preference (addUnmatchedItems) made the "add new items" half optional — turn it off and a receipt is still recorded in full, it just doesn't touch your list.

War story #1: the AccessDenied that looked impossible

First upload: AccessDenied. The storage rule was textbook — an owner-scoped path, authenticated read/write on receipts/{entity_id}/*. The kind of rule you copy from the docs and never think about again.

The cause took a while to see because it lives in a different service. allow.entity("identity") policies the default authenticated role. But this app puts every user in a Cognito group (USERS), and a user in a group assumes the group's role instead. Correct-looking storage rule, wrong role — every upload denied.

The fix is a ManagedPolicy granting the group roles access to the owner-scoped prefix — and where it lives matters. It sits in the storage stack, listing the auth stack's group roles: storage already depends on auth, so the dependency stays one-way. Attach it from the auth side instead (auth → storage) and you've built a circular dependency; the deploy fails. The policy is still named GroupReceiptOwnerAccess — this feature's scar, now a house convention applied to every owner-scoped prefix since.

War story #2: the save that failed silently, all session

Receipts were parsing beautifully, the review sheet applied prices, everything looked done — and no Receipt row ever existed. No error, no red screen, nothing. For a whole session the feature reported success while writing nothing.

Two Amplify behaviors conspired. First: a.json() fields take a JSON *string* — and the code was passing lineItems as a raw array, which failed AppSync validation. Second, the one that made it invisible: Amplify's client doesn't throw — it returns { data, errors }, and a mutation with no onError and an unchecked errors just… proceeds. data was null, errors had the validation failure, and nobody was looking.

// The shape of the fix — every Amplify call now checks errors and throws:
const { data, errors } = await client.models.Receipt.create({
  // …
  lineItems: JSON.stringify(lines), // a.json() wants a STRING, not an array
});
if (errors?.length) throw new Error(errors[0].message);

The write-side rule (stringify) and the read-side rule (tolerate string-or-parsed) are now conventions, and surfacing errors is non-negotiable. Learn your client's error contract before it teaches you: a client that returns failures instead of throwing them turns every unchecked call site into a silent-failure factory.

The Receipt model: the first real new kind

Part 11 and Part 13 kept refusing new models — a template is a flag, an expense is an attribute. A scanned receipt finally passes the new-kind test, for one reason: a receipt is an immutable record, and list items are live state. The items a receipt priced keep changing — renamed, re-priced, cleared. The receipt must not.

Receipt: a.model({
  userId: a.string().required(),
  listId: a.id().required(),
  listName: a.string(),   // denormalised — the /receipts screen spans lists
  store: a.string(),
  imageKey: a.string().required(), // receipts/{identityId}/<uuid>.jpg — the kept photo
  lineItems: a.json(),    // snapshot [{name,quantity,unit,price,category,matchedName}]
  itemsTotal: a.float(),  // sum of the lines we logged
  storedTotal: a.float(), // the receipt's own printed grand total
  currency: a.string(),
  capturedAt: a.datetime(),
  // The date PRINTED on the receipt, kept separate from capturedAt — you might
  // scan a week later, and spend analytics must bucket by when you shopped.
  purchaseDate: a.date(), // date-only: a timezone can't drift it into the wrong day
  // … soft-delete trio
}),

Two fields carry the design. lineItems is a frozen JSON snapshot — the record of what Claude read, independent of what the live items later become. And purchaseDate is an AWSDate, separate from capturedAt: you scan when you get home (or three days later), analytics must bucket by when you shopped, and date-only means no timezone math can ever shift a Saturday shop into Friday.

Receipt history then mounts in two stacks — /expenses/receipts and /more/receipts — as thin routes around one shared screen, so neither tab ever jumps you into the other. Month subtotals stay per currency; a receipt in euros is flagged and excluded from GBP budget maths rather than blended into a confident, meaningless number.

What I took away

• Draw the boundaries first: capture on the client, extraction in the Lambda, matching on the client, confirmation with the user, immutability in the record. Every piece got simpler once its job was singular.
• Domain rules belong in the prompt — day-first dates, currency inference, and a typed way to say "not a receipt" outperform any post-processing.
• Owner-scoped storage rules don't cover group roles. If your users are in Cognito groups, they aren't the role your storage rule policied — grant the group roles from the storage side, or enjoy the circular dependency.
• A { data, errors } client is a silent-failure factory until you check errors everywhere — and a.json() takes a string, not your array.
• Records and live rows are different kinds. The snapshot/live split (plus a separate printed-date field) is what keeps the analytics honest later.

Next up

Receipts price your history; the next input device prices your present: the barcode scanner. A lookup that walks a provider chain across four open product databases, uses Claude as a normaliser (never a generator), and caches every answer in a product table shared by all users — Post 15.

What's your policy for LLM output that writes to the database — review gate, confidence threshold, or straight through? And has a silent { data, errors }-style client burned you too?