"Milk, and a dozen eggs": a voice-to-Claude shopping list that never trusts the AI

I hold the mic and say: "milk, a couple of bananas, a dozen eggs, and some ripe avocados." Two seconds later a tidy list appears — Milk ×1, Banana ×2, Egg ×12, Avocado (ripe) — each already filed under Dairy or Produce. But it doesn't save. It waits for me to say yes.

The interesting part of this feature isn't the AI. It's the deliberate gap I put between the AI and my database.

TL;DR — On-device speech-to-text turns speech into a transcript; a Claude Haiku Lambda turns the transcript into structured JSON items; and — the key decision — that mutation returns the items without saving them. The client shows a review sheet, and rows are only written when the user taps Add. An LLM in your pipeline should never write straight to your data store. Put a human tap between the model and persistence.

(Part 5 of Building CannyCart, a voice-first shopping app I'm building in public. The last three posts were the backend and the build traps; this is the feature all of it existed for.)

The goal, and the one constraint that shaped everything

The goal is boring to state and hard to earn: say what you want to buy, get a clean editable list. Quantities resolved ("a dozen" → 12), items title-cased and singular ("bananas" → Banana), each dropped into an aisle so the list sorts itself.

The constraint is the whole story. An LLM is very good at this and occasionally wrong — it will mishear "eggs" as "legs," miscount, or confidently invent a category. That's fine for a suggestion and unacceptable for a data store. So the rule I built around: Claude never writes to the database. It proposes; the user disposes. Everything below serves that line.

The pipeline in one breath

mic
 └─ expo-speech-recognition (on-device)      → transcript string
     └─ parseShoppingList mutation (AppSync)  → parse-shopping-list Lambda
         └─ Claude Haiku                       → JSON [{name,quantity,unit,category,note}]
             └─ review sheet (client)          → user edits / removes / confirms
                 └─ addItems()                  → ShoppingItem rows created

Five hops, and the LLM sits in the middle with no write access to anything.

Step 1: on-device speech, accumulating a transcript

Speech-to-text is expo-speech-recognition (iOS SFSpeechRecognizer / Android SpeechRecognizer), wrapped in a hook. The one subtlety: the recognizer streams two kinds of result — interim guesses that change as you speak, and final segments that don't. The UI needs both: finalized text stays put while the live guess trails after it, greyed out.

useSpeechRecognitionEvent("result", (event) => {
  const text = event.results[0]?.transcript ?? "";
  if (event.isFinal) {
    setTranscript((prev) => (prev ? `${prev} ${text}` : text).trim());
    setInterim("");
  } else {
    setInterim(text); // live guess for the current utterance
  }
});

Running with continuous: true means it keeps listening across pauses, so you can think mid-sentence. On stop, transcript + interim is the full text to parse.

Step 2: the prompt is the parser

Here's the part people expect to be complicated and isn't. The Lambda barely does anything — the parsing logic lives entirely in a system prompt. This is the extraction contract, trimmed:

You extract items a shopper wants to buy from their spoken words.
Return ONLY a JSON array — no prose, no code fences. Each element:
{ "name": string, "quantity": number, "unit": string|null,
  "category": string|null, "note": string|null }

Rules:
- quantity defaults to 1; convert words to numbers ("a dozen" → 12, "a couple" → 2).
- category: for groceries use aisle names (Produce, Dairy, Bakery, Meat,
  Frozen, Pantry, Drinks, Household); else a short department (Electronics…).
- name is the item only, singular and title-cased ("bananas" → "Banana").
- note captures qualifiers ("ripe", "low-fat", "new") or null.
- Return [] only when the text mentions nothing to buy.

The model is Haiku — cheap and fast, which is exactly right for short structured extraction:

const MODEL = "claude-haiku-4-5";

const message = await anthropic.messages.create({
  model: MODEL,
  max_tokens: 1024,
  system: SYSTEM_PROMPT,
  messages: [{ role: "user", content: text }],
});

And because a model told to emit raw JSON will occasionally wrap it in a code fence anyway, the Lambda is defensive before it trusts the output:

const cleaned = raw.replace(/^```
{% endraw %}
(?:json)?\s*/i, "").replace(/\s*
{% raw %}
```$/i, "").trim();
let parsed: ParsedItem[];
try {
  parsed = JSON.parse(cleaned);
} catch {
  throw new Error(`Model returned invalid JSON: ${raw.slice(0, 200)}`);
}
if (!Array.isArray(parsed)) throw new Error("Model did not return a JSON array");

return JSON.stringify(parsed); // a.json() custom mutations return AWSJSON

Strip fences, parse, assert it's an array, and only then hand it back. If the model returns garbage, the mutation throws and the client shows an error — it does not pass mystery data downstream.

Step 3: the review gate — the whole point

The mutation is declared to return items, not create them:

parseShoppingList: a
  .mutation()
  .arguments({ text: a.string().required() })
  .returns(a.json())            // ← returns parsed items…
  .handler(a.handler.function(parseShoppingList))
  // …and there is no .create() anywhere in the handler. It never persists.

On the client, capture runs through a tiny state machine — the phases are the UX:

type Phase = "listening" | "parsing" | "review" | "empty" | "error";

listening shows the pulsing mic and transcript; parsing shows a spinner while Haiku works; review renders the editable list; empty/error are the honest failure states (heard nothing, or the parse threw). Only review can lead to a write.

The review sheet is where the human tap lives. Every parsed item is an editable row: rename it, step the quantity up or down, see its category chip, delete it, add a blank row by hand, or record more and append. Nothing has touched the database yet. Then:

onAdd={() => {
  const clean = items.filter((i) => i.name.trim());
  addItems.mutate({ listId, items: clean }, {
    onSuccess: () => toast.show(`Added ${clean.length} items`, { type: "success" }),
  });
}}

addItems is the only path that creates ShoppingItem rows — a plain typed create per item, owned by the user's Cognito sub. That single tap is the entire safety model: the AI's output is a draft until a human signs it.

The gotcha: the API key that wouldn't reach the Lambda

One dead end worth saving you. I first wired ANTHROPIC_API_KEY through Amplify's secret() — the "correct" way to handle secrets. In the sandbox, the function kept booting with an undefined key: its deploy-time secret resolution simply wasn't delivering the value to the function. I switched to injecting it as a plain Lambda environment variable, baked at deploy from the deployer's own environment (locally via a sandbox script; on branch builds via Amplify Console env vars):

export const parseShoppingList = defineFunction({
  name: "parse-shopping-list",
  entry: "./handler.ts",
  timeoutSeconds: 30,
});
// backend.ts injects process.env.ANTHROPIC_API_KEY as a normal env var.

Not glamorous, but it deployed and the key was there at runtime. (If you're keeping score with Post 2: this app and environment variables have history.)

Details worth stealing

• AWSJSON can arrive double-encoded. Depending on the layer, a.json() comes back as a string, or a string of a string. The client unwraps until it's an object: while (typeof value === "string") value = JSON.parse(value);
• ShoppingItem.price exists but is unused in v1. It's forward-compat for the Expenses tab — the category Claude infers today is what makes per-aisle spend possible later.
• Ownership is app-layer. Every item carries userId = Cognito sub, every read filters it, and "delete" flips a soft-delete trio rather than removing the row. The AI feature inherits all of it for free.

What's next

The obvious upgrades: stream the parse so items pop in one by one instead of after a beat, and add optimistically so the list feels instant. But the bigger thread is that reserved price field — the next feature is the Expenses tab, turning a checked-off shopping list into per-aisle spend. Claude's categories were always seeds for that.

If you've put an LLM in a real product: where did you draw the line between "the model suggests" and "the model acts"? I'm convinced the review gate is right for anything that writes to a user's data — but I'd like to hear where you'd let the model off the leash.