This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass
Here is the scene this is built for.
You are standing at a set of worn stone steps behind the old post office. Someone who grew up here tells you her grandmother cut through them on the way to the market, and that the treads are smooth on one side only. Two years from now that detail exists in your phone and nowhere else, and in four years it may exist nowhere at all.
StorySteps is a PWA for collecting exactly that, and then getting out of the way. Open the app, press record (after asking the person in front of you, which the app reminds you about), speak for a minute, stop. Transcribe it, fix the two words the model got wrong, save. The memory now has a place, a date you recorded it, a date it reportedly happened, a transcript, and the original audio. Then you put the phone back in your pocket and keep walking.
No account. No server. No sync. The models run on the device.
What I Built
A field session is a walk. You name it, and every memory recorded during it belongs to that walk, so the archive reads as a sequence of outings rather than a pile of loose notes.
Each memory carries more than a note. It has a category (oral history, natural feature, local tradition, environmental change, observation, built heritage), a source (firsthand, recollection, observation, supplied reference) and a verification status that is yours to set, never the model's.
The two dates are deliberately separate, and this is the part I care about most:
- recorded: when you captured it. Always known.
-
event date: when it supposedly happened. Often vague, so it accepts
1998,1998-04or1998-04-12and stores the precision alongside it, so a year never silently pretends to be a day.
If the speaker said "sometime in the seventies", the event date is 1970s-ish vague and stays vague. The app will not sharpen it for you.
The map shows every placed memory with a marker coloured by category, a legend, filters, and a results list that stays in sync with it. Location is only requested when you press the button, and anything you type or pick by hand is stored as an estimate, labelled as an estimate, never as a device reading.
The timeline is per place. It puts recorded dates next to remembered event dates so the gap between when you heard it and when it happened is visible instead of hidden.
Export and import exist because the archive should be yours. Export writes a versioned JSON file, with audio as an explicit opt-in because base64 makes it enormous. Import validates every field and the schema version before a single row is written, shows you how many records are new and how many already exist, and applies the whole thing in one transaction, so a malformed file leaves your archive exactly as it was.
Screenshots:
What it refuses to do
- It never invents a transcript. Short or silent recordings are rejected with a reason instead.
- It never marks model output as verified history. Extraction produces suggestions you must accept.
- It never sends a transcript, a recording or a coordinate anywhere.
- It never claims a model is ready when it is not.
Demo
Live demo: https://storysteps.onrender.com
Code
greenbugx
/
StorySteps
Every place has a story. Go outside and find it.
StorySteps
Every place has a story. Go outside and find it.
StorySteps turns walks, conversations and observations into an atlas of local memories: oral histories, disappearing landmarks, local traditions and environmental change. You record a note or a conversation, and it becomes an archive record attached to a place and a date.
Everything stays on your device. There is no account, no server, no analytics and no remote AI. The models that transcribe and structure your notes run in your own browser.
The problem
Local knowledge is fragile. A village elder describes where the old crossing used to be, a mason explains which stones were reused from the railway station, someone notices the last banyan at a crossing is gone. Most of it is written down in a notebook, or not at all, and within a few years the details are gone with the people who held them.
StorySteps is…
React 19, Vite, TypeScript, Tailwind v4, Dexie over IndexedDB, Leaflet, LiteRT-LM, Transformers.js, Zod, vite-plugin-pwa, and 29 tests. pnpm install && pnpm run dev and it runs, with no API key and no environment file.
How I Built It
Two open models, both inside the browser:
| Model | Size | How it runs | Fetched when |
|---|---|---|---|
| Whisper base, quantised ONNX | about 76 MB, plus about 26 MB for the ONNX runtime | Transformers.js in a Web Worker, WebGPU when a real adapter answers, WASM otherwise | first time you press Transcribe |
| Gemma 4 E2B, web build | about 1.9 GB | LiteRT-LM on the main thread | when you press Download model in Settings |
flowchart TD
A[Walk and conversation] --> B{Input}
B -->|Typed note| C[Original text]
B -->|Audio| D[Whisper in a worker]
D --> E[Transcript, audio kept]
C --> F[Gemma extracts fields]
E --> F
F --> G[Zod validation]
G --> H{A human accepts it?}
H -->|Yes| I[Dexie and IndexedDB]
I --> J[Archive, timeline, map marker]Whisper runs in a worker, and the audio never enters React state
MediaRecorder writes chunks straight into IndexedDB as blobs. The blob is decoded and resampled to 16 kHz mono, checked for length and for actual silence, and only then handed to the worker. Silence and near-silence are rejected before the model is invoked, because there is no point spending 76 MB and 13 seconds to learn that nothing was said. My own fake-microphone test recording tripped that guard, which is the nicest possible confirmation that it works.
The worker reports download progress, so the first run tells you it is fetching about 76 MB rather than appearing to hang.
Gemma streams, because 1.9 GB will not fit twice
The obvious way to use Gemma is to hand LiteRT-LM a URL, but that means a 1.9 GB download every session and no offline use. Instead the download is streamed through a TransformStream into the Cache API, with progress and a cancel, and the engine is handed cache.match(...) as a stream. The browser never holds two copies in memory, a reload costs nothing, and the app has no way to reach for the network on a later run.
The runtimes are served from this origin, not a CDN
LiteRT-LM and onnxruntime-web both default to fetching their WASM from jsDelivr. That would quietly break local AI the moment the signal drops, which is the one thing this app cannot afford. A small Vite plugin copies both runtimes out of node_modules at build time and serves them from /runtime/litert and /runtime/ort, where the service worker caches them on first use.
The schema refuses sloppy data
Partial dates are validated as real dates, 1998-13 is rejected, coordinates must appear as a pair or not at all, and an event date's declared precision has to match its own length. That last one sounds fussy until you remember that a year and a day look identical in a string, and a database full of fake precision is worse than one full of honest gaps.
Bugs I hit along the way
Whisper died on WebGPU with webgpuInit is not a function. I had pointed onnxruntime at the ort-wasm-simd-threaded.jsep factory. In the version installed here, WebGPU has moved out of jsep, so the runtime asked the module for an entry point it does not define, then reported that no backend was available:
asyncify.mjs -> 1 (webgpuInit)
jspi.mjs -> 1
jsep.mjs -> 0 <-- what I was loading
plain .mjs -> 0
Repointing at asyncify fixed it, and WebGPU transcription now runs with a real adapter. There is now a test that fails if that factory ever stops exporting webgpuInit, because I verified it catches the original bug by reintroducing it.
Offline, the app quietly lost its typography. The service worker precached js, css, html and svg but not woff2, so with no connection the headings fell back to system serif. Nothing errored, it just looked wrong. Fonts are precached now, 18 entries and 1.74 MB, and I check it by reading the computed font family on an offline reload.
The map rendered inside a fraction of its own box. leaflet.css was never imported, so panes and tiles had no size rules and the map drew into about a quarter of its container with the rest blank. It looked like a tile outage, which is exactly what it was not.
The onboarding gate fought the router. A flag read on first render redirected to the welcome page after you had already left it, because the state was initialised as "not done" instead of "not known yet". Small bug, immediate bounce.
Selects displayed raw values like observation instead of Observation, because the component needs its item labels declared. Every control now has an accessible name, which I checked by walking all five routes and asserting no button, link or input is nameless.
Why Does Open Innovation Matter?
Because of what the data is.
An oral history with coordinates attached is not something you want sitting on a server you do not control, especially when the people in it never agreed to that. With a closed API, every transcript, every recording and every location would leave the device, there would be a per-token bill attached to a hobby that happens on a walk, and the entire thing would die the moment the signal does.
Open weights meant the models are a detail rather than a dependency. I could build the part that actually matters, the quiet collecting and the honest archive, and let the AI sit behind a toggle that is off by default.
It also meant I could go and read what onnxruntime actually ships. The bug above was not a support ticket or a vague "model sometimes does not work", it was a grep away, and the fix was mine to make. Being able to read the runtime is half the value of it being open.
Both models ran on my machine with a real WebGPU adapter, and Whisper also on the WASM fallback, producing the same transcript either way. It also transcribed with every non-localhost request blocked, which is the proof that it needs no network.
My Agent Session
Build StorySteps, a privacy-first offline-first PWA that turns walks, conversations and observations into an atlas of local memories. (Full 500 line specification abridged here.) Hard rules: no backend, no account, no API key, no analytics, no em dashes in any file, do not scaffold a new project or replace the package manager, do not commit or stage anything, and never fake a capability. Model inference and transcription must run locally; map tiles may need the network. Ship a README with honest claims about what was actually tested.
Repository audit first, before any edits. Node 26.7.0, pnpm 11.3.0, Vite 8, React 19, Tailwind v4, shadcn/ui on Base UI. The template shipped Dexie, Leaflet, LiteRT-LM and Transformers.js but no tests, no router and a bare counter page.
Data model first, since everything else depends on it. Partial event dates need explicit precision, otherwise a year and a day look identical to the schema. Coordinates are nullable but must appear as a pair.
Dexie with two versions: v1 plus a v2 upgrade that backfills the fields added during the build, so an early archive survives. Audio is a separate table so metadata queries never drag blobs through.
Local AI is the part most likely to quietly lie, so it got the most scrutiny. Gemma 4 E2B is 1.9 GB, which is far too big to hold twice in memory, so the model streams into the Cache API and is handed to LiteRT-LM as a stream.
The LiteRT-LM and ONNX runtimes load their WASM from a CDN by default, which would break local AI offline and send a request to a third party. A small Vite plugin copies both runtimes out of node_modules and serves them from this origin under /runtime/litert and /runtime/ort.
Verification was browser-driven wherever possible, because the acceptance criteria are behavioural. Typed memories, search, editing, deletion, export and import were exercised through the real UI.
no available backend found. ERR: [webgpu] TypeError: j(...).webgpuInit is not a function when I press Transcribe on the Device
That is a real bug and it is mine. The WebGPU path pointed at the ort-wasm-simd-threaded.jsep factory. In the onnxruntime-web version installed here, WebGPU has moved out of jsep, so the runtime called a function the factory never defines, then reported no backend at all.
Fixed the paths, dropped the unusable 28 MB jsep pair from the build, and added a one-shot fallback so a WebGPU failure lands on WASM with an honest message instead of a dead end. Added a regression test that asserts the configured factory really exports webgpuInit.
Whisper was proven on the real model rather than mocked: an 11 second speech sample transcribed correctly in about 13 seconds including the first download, and about 3 seconds once cached. Re-running with every non-localhost request blocked produced the same transcript, which is the evidence that it works offline.
change the project name to StorySteps please
Renamed the product name everywhere, and the identifiers too, not just the strings: the IndexedDB name, the Dexie class, the archive format id and export filename, both Cache API names, the manifest name and the icon files. Flagged the two real consequences: an archive exported under the old format string no longer imports, and a browser profile with old data starts empty.
please add MIT license to the project please
Adding MIT surfaced a problem worth deciding out loud: react-leaflet 5.0.0 is not MIT, it is Hippocratic License 2.1, which adds a termination clause, arbitration and an indemnity and binds users to human rights compliance. Leaflet itself is BSD-2-Clause. Asked whether to keep it and document the exception, or drop it for a hand written Leaflet wrapper. The author chose to keep it, so the README states the exception plainly and names the single file that would have to change for a purely MIT dependency set.
Final pass found a real offline defect: the service worker precached js, css, html and svg but not woff2, so offline the display face silently fell back to system serif. Fonts are precached now (18 entries, 1.74 MB), verified by resolving getComputedStyle on an offline reload.
What I would tell someone considering this build: everything except Gemma inference is verified in a real browser, the map tile layer genuinely needs the internet, and browser storage is neither encrypted nor durable. Gemma is implemented and its runtime loads, but the test browser returns a null WebGPU adapter, so the 1.9 GB download, generation and the extraction review screen are unproven on real hardware and the README says so.
Prize Categories
- Best Use of Gemma
- Best Use of Render
Top comments (0)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.