I built an ofrenda card maker and spent most of the day stopping Gemma from making things up

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

I missed Week 2 of this challenge completely, and I started this one at around 1:30 a.m. on 31 October, the last day of the window. My Week 1 and Week 3 entries (Pixel Leak and Screenshot Sloth) both used Florence-2 to read images, so for the last one I wanted a model that listens instead.

The theme had two halves: build something spooky, or something that honours the past in the spirit of Día de los Muertos. I went with the second. On the Day of the Dead, families build an ofrenda, an altar for someone who has died, with their photo, marigolds, candles, and the food and things they loved. Ofrenda, the app, helps with one small part of that: getting what the person loved onto a card, in the words of the people who knew them.

What I Built

You record someone telling a story about a person they miss, or upload an old voice memo or video. A minute or two is enough. Then:

1. Whisper writes the story down. You pick the language and fix any names it got wrong.
2. Gemma reads it and suggests what goes on the card: favourite food and drink, music, and things the person always said.
3. Gemma also picks one sentence from the story for the card, and the card shows that sentence exactly as it was told.
4. You get the card: papel picado across the top, their photo (or a candle) in an arch of marigolds, and whatever you kept. It downloads as a PNG or prints, and you can save everything as a JSON "memory file" that opens again later.

Both models run in the browser tab. The recording, the photo and the story aren't uploaded anywhere.

Most of my time went into one thing: checking Gemma's suggestions before they reach the card, and marking anything that doesn't hold up so a person can decide. More on why below.

Demo

Try it: https://tarunvashishth.github.io/ofrenda/ (Chrome or Edge on a laptop, because Gemma needs WebGPU)

If you don't have anyone to record, click Try a made-up story. It loads a story about Tomás, a radio repairman who sang Los Panchos badly, made café de olla every Sunday, and hated cilantro. The cilantro is there on purpose: it shouldn't end up on his card, and it doesn't.

The first visit downloads about 950 MB of model weights from Hugging Face (Whisper base and Gemma 3 1B), and they're cached after that. On my Mac, with the GPU doing the work, Whisper transcribed a 51-second recording in 13 seconds and Gemma filled in the card in 11 to 15 seconds across my runs.

Code

tarunvashishth / ofrenda

Record a story about someone you miss. Open models (Whisper and Gemma) turn it into a card for your ofrenda, in your browser. Nothing is uploaded.

Ofrenda 🕯️

Keep their stories in their own words.

Try it: https://tarunvashishth.github.io/ofrenda/ (Chrome or Edge on a laptop: Gemma needs WebGPU)

Record someone telling a story about a person you miss, or drop in an old voice memo or video. Two open models running in your browser tab turn it into a card for your ofrenda:

• Whisper writes the story down, in the language it was told in. You can fix anything it misheard.
• Gemma gathers what they loved: the food and drink, the music, the things they always said. Every suggestion is checked against the story. Anything that isn't in it, or that the story says they disliked is marked so a person decides.
• Gemma also picks the one sentence of the story that says the most about them, and the card shows that sentence exactly as it was told, never a rewrite.

The card (papel picado…

View on GitHub

TypeScript and Vite, no backend. Both models run in a Web Worker. src/gather.ts has the prompts and all the checks, with tests, and scripts/eval.ts scores Gemma on six made-up stories.

How I Built It

Picking the models

Whisper base for the speech and Gemma 3 1B instruct for the reading, both as ONNX builds running on Transformers.js with WebGPU. Gemma is 4-bit, about 760 MB.

I looked at Gemma 4 E2B first, because it can take audio directly and I wouldn't have needed Whisper at all. Its browser build is 2.4 to 3.3 GB, though, and the laptop at a grandparent's house might not cope with that. Gemma 3 1B and Whisper together come to under 1 GB.

Gemma kept putting the wrong things on the card

Before trusting Gemma with anything that goes on a memorial card, I wrote six made-up stories to test it: an abuelo in Mexico, a nani in Jaipur, a mail carrier in Ohio, a tía in Oaxaca (told in Spanish), a diner owner, and an auntie who sang Teresa Teng at every karaoke night. Each one lists what should go on the card and what shouldn't.

The first prompt found 26 of the 32 things I expected, which was fine. The problems were elsewhere:

• Every answer started with "Here's a draft of the card…", and for the Spanish story Gemma wrote a poem first.
• Grandpa Joe's story says, plainly, "He hated olives." Gemma listed olives as one of his favourite foods.
• Auntie Mei's durian and Tía Carmen's coffee, both things they couldn't stand, turned up under things they "always said".

A 1B model will get things like this wrong sometimes, so I stopped expecting the prompt to fix everything and added two checks in plain code. inStory() checks that a suggestion's words actually appear in the story. It ignores accents, so café matches cafe, and it handles plurals and Devanagari. disliked() looks at the few words right after "hated", "couldn't stand", "no soportaba" and similar phrases, or right before "smelled terrible". Olives, durian and café are all in the tests now.

When a suggestion fails a check, the page marks it and leaves the decision to you:

(I added cilantro and tamales by hand for this screenshot, to show both kinds of mark.)

Getting a 1B model to follow the format

Three changes made the biggest difference:

• I start Gemma's reply for it with FOODS:. It can't write a preamble or a poem when the answer has already begun.
• Two short worked examples did more than any instruction I wrote. One has an uncle who never sang, to show that SONGS: none is a fine answer. Gemma doesn't always take the hint: for the mail carrier, who never played music, it still came up with "he'd sing Frank softly". The story check marked it.
• The examples use unusual foods. My first example had black coffee in it, and Gemma put black coffee on the mail carrier's card. Now they use lemon bars, butterscotch candies, masala peanuts and sugarcane juice, so if one leaks into an answer, inStory() catches it.

Why the card quotes the family instead of Gemma

My first version also asked Gemma for "two warm sentences" about the person. For the diner owner it wrote:

Grandpa Joe was a beloved diner owner, known for his hearty meals and friendly nature.

Apart from "diner", nothing in that sentence came from his story. For Nani, whose story is in English with a few Hindi phrases, it replied in Spanish with a Polish word in the middle. Across the six stories, 8 of its 12 sentences said things the story didn't.

So I changed the question. Gemma now picks the one sentence of the story that best shows who the person was, and snapToStory() swaps Gemma's copy for the original sentence, word for word. That swap matters, because Gemma usually copies only part of the sentence and sometimes rewords it. It also never picks a sentence about something they hated. For the mail carrier, the card now says:

My dad, Frank, was a mail carrier in Ohio for thirty-one years, and he knew every dog on his route by name.

Where it ended up

	Found	Not in the story	Things they hated	The card's line
First prompt	26 / 32	3 of 34	3	written by Gemma
Gemma writing two sentences				8 of 12 sentences not backed by the story
Now	27 / 32	2 of 28, both marked	0	5 of 6 in the family's own words*

* Auntie Mei's card has no line: the sentence Gemma picked was about durian, which she hated, so the card leaves it for the family to write.

npm run eval reruns this on your machine, with Gemma on the CPU, in about half a minute per story.

Two bugs worth mentioning

• Whisper dropped the middle of a story. On WebGPU with a 4-bit decoder, a 52-second recording came back without the two sayings or the cilantro. The same audio in Node came back complete. For whisper-base, the fp16 decoder turned out to be smaller than the 4-bit one (105 MB against 124 MB), and with it, plus stitching the 30-second windows together by timestamp, the whole story comes back.
• Gemma doesn't run without WebGPU. Its 4-bit build uses an operator (GatherBlockQuantized) that ONNX Runtime's WASM backend doesn't have. In a browser without WebGPU, the page says so up front: Whisper still works, and you fill in the lists yourself.

If you're building something similar, these posts cover related ground: What breaks when you ship 21 AI tools that never touch a server on onnxruntime and model-size problems, Building Browser-Based Transcription: Workers, Cancellation, and Honest Progress on transcription in a worker, and Run an LLM in the browser: WebGPU did 63 tokens/s, WASM did 1.6 on how much slower the CPU path is.

Caching

The weights download once. On my second run, Transformers.js checked Hugging Face for updates (a couple of dozen small requests), and the 763 MB of Gemma weights loaded from the browser's cache in under a second. A service worker caches the app itself. I haven't done a full run with the Wi-Fi switched off yet, so I'm not claiming it works offline until I have.

Why Does Open Innovation Matter?

The recordings this app is for are people's voices, often the voices of people who have died. Not many families would want to send those to a server, and with Ofrenda they don't have to. After a one-time download, Whisper and Gemma run on your own computer, and you can watch the network tab to confirm the audio never leaves the page.

They also need to last. Memorial and "legacy" apps shut down, and what people stored there goes with them. Ofrenda is a static page built on open-weight models anyone can mirror, and the memory file is plain JSON. If my GitHub Pages site disappeared, your saved stories would still open in a text editor.

Running the exact model locally also let me test it properly. I could score Gemma on six made-up families with a short script, change the prompt, and rerun it in minutes without paying for every call. That's how I found the olives.

And because the code is small and open, it can be changed for your family: Whisper small for strong accents, the card's labels translated, more phrases for the dislike check to look for.

Prize Categories

Best Use of Gemma. Gemma 3 1B does the reading: it gathers the food, music and sayings, and it chooses the sentence for the card, running on WebGPU in the browser. Most of this post is about the checks I needed before I'd trust a 1B model with a memorial card.

My Agent Session

I built Ofrenda with Claude Code, which wrote most of the code and helped draft this post. Everything described here comes from real runs during the build: the olives, the poem, the black coffee, the Spanish-and-Polish sentence and the missing middle of the recording all came up in the eval or in testing, and each fix came with a test or a rerun. The stories in the eval are made up.

What's next

• Card labels in the story's language, not only English
• Gemma on the CPU with an 8-bit build, so it works without WebGPU
• Room for several cards, one for each person on the family's ofrenda