This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass
The detail you photograph picks the path. Gemma reads the photo live in the desk demo. Photo: Francesco Bini, Wikimedia Commons, CC BY-SA 4.0, cropped.
I live and write code in Istanbul. The city is full of old fountains with poems carved into the marble. Most of us walk past them and look at our phones.
The Bereketzade Fountain faces Galata Tower. One side of it has a gilded inscription and a brass spout. In my game, this side of the fountain is a fork.
If you photograph the inscription, Gemma sees "gold calligraphy inscribed on a rectangular stone plaque". The cue is writing, and the next stop is a small mosque downhill. If you photograph the spout, Gemma sees "a brass spout set into a carved marble fountain wall". The cue is water, and the next stop is a second fountain on a street corner.
These are real readings from the desk demo. In 10 of 10 pairs of live runs, the two photos led to different stops.
Forking Paths is a walking mystery for your phone. Gemma 4, an open-weight model, writes a short case from facts about the streets around you. Every fact the narrator tells you comes with its link. The narrator reads each clue aloud, so your eyes stay on the street. At each fork, the detail you photograph picks where you walk next.
The demo video shows both ways through the fork, in the game's own ElevenLabs voice.
What I Built
You choose a walk of 15 or 25 minutes and tap Start the walk. The narrator is an old Istanbul ferry captain who became an archivist. He asks one question, and the answer is a real fact at the last stop.
Each case asks one question. The answer to this one is at the end of this post.
Each clue names something you can see. It never uses a compass direction. Then the app says Lock your phone. Go. and stays quiet, so you hear the street.
At the place, you tap I'm here. GPS alone is enough to pass the stop. Then the look line tells you what to photograph. Gemma describes your detail, and the narrator tells you one fact about the place. The screen shows the source of the fact: Wikipedia, or a Google result that SerpApi fetched.
At a fork, each way out has a cue. A cue is one word, such as writing or water. The look line gives a hint, but it does not choose for you: "Two things here keep a story: one you can read, one that flows. Photograph the one you trust, up close."
Gemma finds a cue in your photo, and the cue picks the next stop. The narrator's bridge line then connects your detail to that stop.
Screens from a scripted browser run.
All routes end at the same answer. At the end, the case file also shows the road you did not take.
Gemma's readings from a run of the app. Photos: Wikimedia Commons, CC BY-SA 4.0 (Francesco Bini, Basak, Ali Osman Dilekoğlu, Another Believer).
Forking Paths is for anyone who wants a reason to walk for 15 minutes and look closely at an old part of town.
Demo
The video at the top is the desk demo at phone size. It is 1 minute 54 seconds long.
Try it from your desk: clone the repo, run the app and open /demo. The README shows how.
You do not need an account or GPS. Play the saved Karaköy case with The gilded inscription. Then play it again with The brass spout, and the route changes. You can also tap Use your own photo. If the live model fails, the sample photo uses its stored result, and the screen says so.
No walk needed. Photos: Francesco Bini, Wikimedia Commons, CC BY-SA 4.0, cropped.
Generate a quest near me writes a 15-minute case around you, or in a city you pick. It takes about two minutes. If a saved case starts within about 500 m, the app uses that case again. It has been tested in 8 cities: Istanbul, London, New York, Berlin, San Francisco, Lisbon, Rome and Prague. It works in city centres, where many places have facts. You can also pick one of the tested cities.
Code
Forking Paths
A voice-first walking mystery for your phone, written by an open-weight model.
Gemma writes a short case from facts about the streets near you, and every fact comes with its link A narrator reads each clue, so you can lock your phone and walk At each fork, the detail you photograph picks your next stop. Every route ends at the same answer.
Demo video: YouTube · Write-up: DEV
Try it from your desk
You need Node 22, pnpm and Docker.
cd apps/web
cp .env.example .env.local # add GOOGLE_GENERATIVE_AI_API_KEY for live Gemma
pnpm install
docker run -d --name forking-paths-mongo -p 127.0.0.1:27017:27017 mongo:8
pnpm db:check # connects and creates the indexes
pnpm dev # then open http://localhost:3000/demo
The desk demo plays a saved case in Karaköy, Istanbul, with sample photos, so it needs no GPS It also plays with no API keys: its voice clips are in the repo, and…
The code is MIT licensed. 621 unit tests pass. 12 browser checks also pass against a production build at 390 × 844, with no paid API calls. The README explains how to run the game with hosted or local Gemma, and how to deploy it to Render.
How I Built It
How the parts fit together.
Gemma does the language and the vision work. Research, the rules and the fork decision are plain code. MongoDB stores the cases and the walks. Sentry traces each request, and photos stay out of the traces.
I designed the game and directed the build. Coding agents (Claude Code) wrote much of the code and ran the checks and evals behind the numbers in this post. Every language-model call in the product goes to Gemma.
The writer
First, code fetches the facts, numbers them and measures possible routes. Each hop is at most 250 m. Then gemma-4-31b-it writes the case in one call, with thinking on and no tools. It picks a route and returns the case as JSON. Every line that states a fact cites fact IDs.
A validator of 665 lines then checks the case. It rejects unknown fact IDs, numbers that are not in the cited facts, compass words, unsafe wording and broken routes. If it finds a problem, it sends its messages back to Gemma for one repair:
// apps/web/src/quest/validate.ts (trimmed)
const unknown = factIds.filter((id) => !factById.has(id));
for (const id of unknown) add(`${label} cites ${id}, which is not one of the facts.`);
if (unknown.length) return; // its numbers are checked once it cites real facts
for (const num of numberTokens(text)) {
if (!known.has(num.value))
add(`${label} says ${num.raw}, which is not in its cited facts (${list(factIds)}); use only numbers from those facts.`);
}
51 broken quests test the validator, one for each rule. With the current prompt, 9 of 10 first drafts pass the first time. Almost all of the time to make a case is that one Gemma call:
A Prague case in Mastra Studio. The write step took 1 min 36 s of the 1 min 38 s total.
The voice
The game talks at each stop. It is quiet while you walk.
ElevenLabs voices the narrator. When a walk starts, the app voices each line with eleven_v4. It caches the clips and keeps the voice under a daily cap. An ambience plays under the clues, and two short sounds play for "found" and "not here".
The 21 clips of the demo case are in the repo, so the desk demo makes 0 ElevenLabs requests.
The voice budget of the demo case: 2,320 credits for 18 lines, a 3-line sample and 3 sounds.
Facts and their sources
Research runs in code, around a start point rounded to about 100 m. First it uses the geo search of Wikipedia. Then SerpApi grounds the writer in fresh results: Google Maps, plus up to four Google searches for the places with the fewest facts. A case uses at most 6 searches.
In a live run at the Karaköy start, 2 of 8 places came only from SerpApi: Galata Tower and the Camondo Stairs. (The demo case itself cites Wikipedia.) At the Pantheon in Rome, Maps raised the number of places with a fact from 11 to 21.
Two sources. Each fact shows its query and its link.
An automated check found all 30 sampled facts in their sources. 22 were word for word on the page. The other 8 were Google Maps descriptions in the stored SerpApi responses, and a coding agent matched them.
The SerpApi account counter: 34 searches before the live tests, 108 after.
The photo check and the fork
At the fountain, the screen says This is a fork.
The phone makes your photo smaller (at most 1024 px) and re-encodes it, so no EXIF data leaves the phone. The server does not keep the photo.
gemma-4-26b-a4b-it gets the photo, the name of the stop and the cues of the fork. It answers in JSON. It says what it sees (never people), whether the photo can be from this place, how sure it is, and which offered cue it found.
The cue field is an enum of the fork's cues. The enum is there because the small E4B model once answered "crest", a word from the hint, instead of symbol-or-emblem. With the enum, it went from 15/20 to 18/20. Then code makes the choice:
// apps/web/src/mastra/agents/reader.ts: at a fork, the cue must be one of the offered cues, or null
const cue = offered.length ? z.enum(offered as [Cue, ...Cue[]]).nullable().catch(null) : z.null().catch(null);
// apps/web/src/quest/fork.ts: null hands over to Gemma's pick by meaning, then to the nearest way
export function decideByRules(place: Place, verdict: Pick<Verdict, "cue"> | null) {
if (place.links.length === 1) return decision(place, place.links[0], "only-link");
const byCue = linkForCue(place, verdict?.cue);
return byCue ? decision(place, byCue, "cue") : null;
}
Gemma writes the bridge line from what it saw and from the facts of the next stop. The validator's text checks apply to this line too. Through the app, a fork took 3,722 ms at the median.
One eval photo fooled both Gemma models. It shows the spout, but also the gilded panels above it. The 26B model saw "a brass water spout" and still chose writing. The E4B model on my laptop also chose writing. Only 2 of the 34 eval photos fooled both models. So the look line now asks for the detail "up close". In the desk demo, the close crop read water in every run.
After the spout photo, the cue is water, and the walk goes to a different street.
Paused walks
Between stops, your phone screen is dark, and nothing runs on the server.
Two Mastra workflows run the game. The generate-quest workflow writes a case. The play-quest workflow suspends at every stop, and your photo resumes it. In 2 of 2 scripted runs, a paused walk survived a server restart.
Mastra saves where each walk is paused. The rules and the photo stay outside it: a paused snapshot is 2,071 bytes and has no image.
play-quest in Mastra Studio, waiting for a photo.
A prompt rule for legends
At the small mosque, the narrator tells you about a secret passage.
Old cities have many legends. A legend needs careful words, such as "is thought to", "says" or "circa". The validator checks IDs and numbers, but it cannot see when a careful word disappears. Gemma's first draft of the demo case dropped three of them:
| Gemma's first draft | Its next draft, with one new rule |
|---|---|
| "They were designed to prevent the banker's children from falling …" | "It is thought that the shape was designed to stop the children … from falling …" |
| "… built between 1870 and 1880 …" | "… built circa 1870-1880 …" |
| "A secret passage once led from here to the harbor." | "The mosque's inscription says that a secret passage … ran from Galata Tower … down to the old harbour at Karaköy." |
This is the new rule in the prompt:
Keep their hedges ("is thought to", "says", "circa"): a legend stays a legend. Never claim that an inscription or a detail says something unless a fact says so.
The next draft passed the first time. One line still said more than its facts ("another hidden link between the lanes"), so it was rewritten by hand. The quest file lists all 8 hand edits.
Measuring the writer prompt
The most common error in first drafts was a compass word (5 of 22 drafts). Gemma copied each one from its own facts. But "the northwest corner" does not help when your phone is in your pocket. The fix was to name the compass word next to the fact in the prompt:
// apps/web/src/mastra/agents/writer.ts (trimmed): a note under each place's facts in the prompt
const note = compass.length
? `These facts use the compass word ${compass.map((w) => `"${w}"`).join(", ")}: cite them, but in clues, hints and link reasons name a landmark instead.`
: "";
Three versions of the prompt ran with the same 10 starting points, the same facts and the same model. Runs with no draft are not counted:
| Writer prompt | First drafts that passed |
|---|---|
| As first merged | 2 of 7 |
| v2: stricter rules on directions, hints and ties | 0 of 6 |
| v3: each fact's compass words named; street names allowed | 8 of 9 |
Version 2 got worse: its rule "Choose its answer place first" sent the start straight to the answer in 2 of 6 drafts. Version 3 removed that rule.
Bugs and fixes
- Tracing kept a photo. Mastra's span store saved the image from a photo call. Now a span processor removes images first: 0 image bytes in 160 spans.
- The first fork on the laptop ignored the photo. The model was not loaded yet, so the read took 10,108 ms, over its 10 s budget. Now the app loads the model when the walk starts, and reads take 3.5 to 3.7 s.
- AI Studio's free tier hid its wait time. The 429 response had no Retry-After header. The wait, "Please retry in 46.1s.", was only in its body. Now the app reads it and waits once, inside its 240 s limit. Two more cases then finished.
- A rough GPS position counted as exact. A position with 1 to 3 km of error would tell a player at the right place that they were "too far". Now a position worse than 150 m counts as no GPS.
How I tested it
- The desk-demo fork runs above: two photos, two stops, in 10 of 10 pairs.
- 30 live case generations in 7 cities.
- 34 labelled photos from Wikimedia Commons, through both photo readers, with 0 errors in 68 calls.
- 8 unsafe test lines. The validator blocks all of them.
- 12 browser checks, from both ways through the desk demo to a scripted play in street mode, with GPS set by the script.
Why Does Open Innovation Matter?
Only one model call sees something private: the photo check. It gets a photo taken where you stand. ElevenLabs only gets the text of the case. SerpApi and Wikipedia only get a point rounded to about 100 m.
The photo check can run on your own machine; the demo uses Google's API. When the app runs on your machine, one setting moves the photo check to Gemma 4 E4B on Ollama:
GEMMA_READER_PROVIDER=ollama # the photo check only; GEMMA_PROVIDER=ollama moves every role
OLLAMA_MODEL=gemma4:e4b
It is the same Gemma 4 family: 26B/31B in the cloud, E4B on my laptop. Both readers ran on the same 34 photos, on the same day:
| Photo check | Gemma 4 26B, AI Studio free tier | Gemma 4 E4B, Ollama, M1 Pro 16 GB |
|---|---|---|
| Right cue, 28 street details (chance: 12.3 of the 25 with a fitting cue) | 26 (93 %) | 22 (79 %) |
| Plausible or not, right | 34 of 34 | 34 of 34 |
| Per photo, p50 / p95 | 3,592 / 5,573 ms | 4,178 / 5,953 ms |
| Desk-demo fork, two photos to two stops | 10 of 10 | 10 of 10 |
| Who sees the photo | Google; free-tier inputs may be used to improve its products | my laptop |
The prompt and the schema are the same. Only the place where the photo is read changes.
The trade-off. E4B is a much smaller model, and it runs on a busy 16 GB laptop. It got four more cues wrong. Its first read took 17.8 s while the model loaded. So the app loads it when the walk starts, and gives it 25 s per photo instead of 10. The writer stays on the 31B model in the cloud.
What open weights made possible. With a closed vision API, every photo goes to one place, and the vendor picks the model. Gemma's open weights (Apache 2.0) let the photo check also run on my laptop. They also let me pick a model size for each job: 31B to write, 26B to read fast, and E4B where the photo stays on a machine I control. The eval is in the repo, so anyone can run it again.
AI Studio listed 62 models for my key, and only 2 of them were Gemma. So "open weights only" is a check in the code:
// apps/web/src/lib/model.ts: AI Studio also serves closed Gemini models, so its IDs must be Gemma
if (provider === "google" && !/^gemma-/i.test(modelId))
throw new ModelConfigError(`Only Gemma models may run on Google AI Studio (got "${modelId}")`);
if (provider === "ollama" && /(^|[:-])cloud$/i.test(modelId))
throw new ModelConfigError(`Ollama cloud models are not allowed (got "${modelId}")`);
Prize Categories
-
Best Use of Gemma: Gemma 4 31B writes each case from numbered facts in one JSON call. 26B reads each photo, picks one cue from a fixed list and writes the bridge lines. With thinking set to
minimal, a one-word answer went from 5.5 s to 0.55 s. E4B runs the same photo check on a laptop through Ollama. -
Best Use of ElevenLabs: ElevenLabs gives the open-source agent its voice. The ferry-captain narrator speaks with
eleven_v4, with a clip cache and a daily cap. ElevenLabs also made the ambience, the two short sounds and the narration of the video. - Best Use of SerpApi: SerpApi grounds the writer in fresh Google Maps and Google Search results. A case uses at most 6 searches, and each fact shows its query and its link.
-
Best Use of Mastra: Mastra orchestrates the agent over open models. The
generate-questworkflow writes a case. Theplay-questworkflow suspends the walk at every stop until a photo resumes it, even after a server restart.
Credits
- Model: Gemma 4 by Google (Apache 2.0), through Google AI Studio and Ollama.
- Code: Mastra, the AI SDK, the MongoDB driver, sharp and Playwright (Apache-2.0). Next.js, React, zod, Tailwind CSS, Vitest and the Sentry, ElevenLabs and SerpApi SDKs (MIT).
-
Facts and voice: Wikipedia (CC BY-SA 4.0, shortened or translated) and Google results through SerpApi. ElevenLabs (
eleven_v4, the premade voice George). - Photos: Wikimedia Commons, CC BY-SA 4.0, cropped: Francesco Bini, Basak, Ali Osman Dilekoğlu, Another Believer. The eval photos are credited in the repo.
- Fonts: Geist, Fraunces, IBM Plex Mono (SIL OFL).
The answer
Back to the question about the stairs. Both photos lead you to the Camondo Stairs at the end. The answer keeps its careful words: "It is thought that the shape was designed to stop the children of the banker who built them from falling down the steps if they slipped."
Then the case file shows the street you did not walk: "Had you photographed the water, you would have gone to Hasan Ağa Fountain."
Photos: Wikimedia Commons, CC BY-SA 4.0, cropped (Francesco Bini, Basak, Another Believer).
Run the desk demo and take the other photo at the fork. Then tell me in the comments what your own street would hide.
Top comments (0)