TrailSage

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

I Built a Pocket Naturalist That Speaks Hindi and English, and Refuses to Make Things Up

I pointed my phone at a plant I'd walked past a hundred times. Five seconds later, a calm voice told me its name, a fact I didn't know, and whether it was rare, first in English, then in Hindi. Then it told me to put the phone away.

That's TrailSage: point, shoot, learn. A bilingual pocket naturalist powered by an open-weight Gemma model, with an ElevenLabs-narrated recap of your whole walk at the end.

What I Built

TrailSage is a mobile-first web app for anyone who walks past living things and wonders about them: hikers, gardeners, birders, kids on a school trip.

The loop is deliberately short:

1. Point your camera at a plant, bird, insect, fungus, or animal
2. Identify. An open-weight vision model (Gemma 4) answers in under 80 words: NAME, one genuinely surprising FACT, and STATUS (common or rare)
3. Hear it instantly. The answer is spoken aloud by the phone's own TTS, in English and then in Hindi. No waiting on a cloud voice on a trail with one bar of signal
4. Every sighting lands in a nature journal (stored on-device)
5. At the end of the walk, hit "Recap my walk". ElevenLabs' "George, the warm storyteller" voice narrates your journey like a nature-documentary outro

It also speaks in, not just out: a 🎤 button takes voice questions (Web Speech API), and everything, questions and answers, works in English, हिंदी, or both.

The design goal underneath everything: the app should want you to look up, not down. Identify, listen, pocket the phone. The recap even ends with "the screen stayed dark, the grass got touched."

Demo

• Live app: https://trailsage.onrender.com/
• Video demo, outdoor walk, camera identify, bilingual voice, George's recap: [https://youtu.be/aY-6SQH-mbU?si=sTf8OWjOEDFGFLw9]
• The honesty screenshot I'm most proud of. I fed it an abstract wallpaper and it answered, bilingually:

NAME: Not a living thing.
NAME: कोई जीवित वस्तु नहीं है।

No invented species, no confident nonsense.

Code

GitHub: https://github.com/shuchi111/trailsage. server.js (Express, ~200 lines), public/index.html (zero-build frontend), instrument.js (Sentry preload), plus TESTING.md, a full test guide covering every endpoint, expected timings, and every failure mode I hit (more on that below).

How I Built It

One Node/Express service, no build step, everything else is browser-native. The journal stays on the phone. Each action sends one small JSON body: the photo, a question, or the sighting titles.

Architecture

trailsage/
├── public/index.html   zero-build UI (camera, voice, journal, speech)
├── server.js           Express API: identify, ask, recap
├── instrument.js       Sentry preload (node --import, before the server)
└── render.yaml         single web service on Render

Layer	Lives in	Does
Trail UI	public/index.html	Camera or file to JPEG, mic, on-device journal, spoken answers
API	server.js on Render	Three JSON routes, plus /api/health
Tracing	instrument.js	Sentry on every /api/* call (no-op if SENTRY_DSN is unset)
Vision + chat	OpenAI-compatible LLM	Gemma 4 first; gemini-flash-lite if it stalls
Walk narration	ElevenLabs	George reads the English recap; browser TTS if that fails

Phone browser
├── camera or file upload → JPEG, long side 768px (~100 KB)
├── mic → Web Speech API (hi-IN or en-IN)
├── journal → localStorage (the photo stays on the phone)
└── speechSynthesis → English, then Hindi
        │
        │  one small JSON body
        ▼
Express on Render          Sentry traces every /api call
├── POST /api/identify → Gemma 4 (vision)
│                        └── no answer in 40s → gemini-flash-lite
├── POST /api/ask      → same models, text only, last journal titles
└── POST /api/recap    → ElevenLabs George (mp3)
                         └── no key, 402, or error → phone speaks instead

What crosses the network

Action	Request	Response
Identify	POST /api/identify: JPEG (~100 KB) and lang (en / hi / both)	NAME, one FACT, STATUS, and which model answered
Ask	POST /api/ask: question, journal context, lang	Short bilingual answer
Recap	POST /api/recap: sighting titles only	English and Hindi script, and either George's mp3 or a reason code (no-key, no-credits, invalid-key, tts-failed) so the phone speaks it

Swap the model with two env vars (LLM_BASE_URL, LLM_MODEL). Same code talks to Google AI Studio, the Hugging Face router, or a local Ollama.

The design rule baked into every layer: degrade, don't die. Model fallback, TTS fallback, camera fallback. The hiker never sees an error screen.

The provider odyssey, or: my open-source model got delisted twice mid-hackathon

The plan said "Gemma 3 4B via Hugging Face router." Reality had opinions:

1. 403: insufficient permissions to call Inference Providers. Fix: re-issue the fine-grained token with the "Make calls to Inference Providers" checkbox.
2. 402: no remaining credits. Free-tier inference credits were gone, and buying pre-paid credits wasn't an option for me.
3. So I surveyed the alternatives: Groq, no Gemma. OpenRouter, Gemma delisted (only Gemini models remain). Cloudflare Workers AI, Gemma listed, but text-only, and I need vision. By late 2026, hosted vision-capable Gemma had quietly disappeared from most serverless providers.
4. Then I checked the one key I already had, a Google AI Studio key, against its models endpoint, and found this: gemma-4-26b-a4b-it. Google now hosts its newest open-weight Gemma on the same OpenAI-compatible endpoint, free tier included.

The entire migration was one environment variable:

- LLM_BASE_URL=https://router.huggingface.co/v1
+ LLM_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai

Same code, still an open-weight Gemma (now Gemma 4, a MoE with 4B active params, fast), zero cost per identification. The whole provider matrix the app ships with:

Provider	LLM_BASE_URL	Cost
Google AI Studio (Gemma 4, current)	https://generativelanguage.googleapis.com/v1beta/openai	free tier
Hugging Face router (Gemma 3)	https://router.huggingface.co/v1	pre-paid credits
Local Ollama (gemma3:4b)	http://localhost:11434/v1	free, offline

Taming a thinking model (the part that ate my weekend)

Gemma 4 is a thinking model, and the OpenAI-compat layer ships its reasoning inside the response content:

<thought>The user wants me to identify... * 8 paragraphs * ...</thought>NAME: ...

Three real problems, three fixes, all in the repo:

1. Thought leakage. Answers arrived as walls of reasoning. The fix is a stripThought() pass. Note the second half, which handles the nastier case where the reasoning never closes its tag because it ran past the token limit:

function stripThought(text) {
  let out = String(text || "")
    .replace(/<(thought|think)>[\s\S]*?<\/\1>/gi, "")   // closed thoughts
    .replace(/<\/?(thought|think)>/gi, "");
  // unclosed <thought> left behind when reasoning outran the token limit
  const openIdx = out.search(/<(thought|think)>/i);
  if (openIdx !== -1) out = out.slice(0, openIdx);
  return out.trim();
}

Plus a "think briefly (under 80 words of reasoning)" instruction in the prompt. Cheaper to prevent the spiral than to survive it.

2. The invisible 105-second timeout. I set a 35s per-attempt timeout and requests still took 105s. The OpenAI SDK retries failures twice by default. My "35s timeout" was secretly 3×35s inside one call:

for (const model of [LLM_MODEL, FALLBACK_MODEL]) {     // Gemma 4 → flash-lite
  if (answer) break;
  try {
    const r = await llm.chat.completions.create(
      { model, messages, max_tokens: 3000 },
      { timeout: 40_000, maxRetries: 0 },              // the bug fix: own your retries
    );
    answer = stripThought(r.choices?.[0]?.message?.content || "");
  } catch { /* next model gets a turn */ }
}

3. When the thinker stalls anyway. That same loop is the fix: Gemma 4 gets 40 seconds, then gemini-flash-lite-latest (~2s, never thinks) answers. The response's model field always says who answered, so the fallback is honest, not hidden.

The rule I ended up with: the hiker never sees an error. Every layer has a degradation path. Model fallback, ElevenLabs to browser TTS, camera to photo upload, studio voice to phone voice.

ElevenLabs: the 402 that wasn't about money

Recaps worked instantly in testing but returned 402 for everyone else. The full error body finally explained:

paid_plan_required: Free users cannot use library voices via the API.

The hardcoded "Rachel" voice is a paid library voice. My key was fine; my voice choice wasn't. GET /v1/voices listed the premade voices my tier can use, and TrailSage now narrates with George, "Warm, Captivating Storyteller", which is honestly a better fit for a nature-walk recap. The voice ID is an env var, and the UI reports why it fell back (no-key / no-credits / invalid-key / tts-failed) instead of a generic apology.

Bilingual everything

One lang field (en / hi / both) flows from a header toggle into the prompt, the answer, and the speech. Bilingual answers are "English paragraph, blank line, Hindi paragraph", which makes the client-side speech trivially two-voiced:

if (lang === "both") {
  const parts = text.split(/\n\s*\n/);      // English, blank line, हिंदी
  speakPart(parts[0], "en-US");
  speakPart(parts[parts.length - 1], "hi-IN");
}

The server-side half is a single instruction appended to every prompt:

both: "Reply in two parts separated by one blank line: first the full answer
       in English, then the same answer in Hindi (Devanagari script).",

The 🎤 mic listens in hi-IN or en-IN to match the toggle, and the recap text itself is bilingual (though George narrates the English line for now. Devanagari TTS on free-tier premade voices is my next experiment).

Sentry in ESM-land

@sentry/node v9 has no ESM default export. (await import("@sentry/node")).default is undefined, which crashed the server the moment a DSN existed. The fix is the documented-but-easy-to-miss pattern: init in a separate instrument.js, preload it with node --import ./instrument.js server.js (baked into npm start). Every identify/ask/recap shows up with latency and token counts. The graphs in this post come from that.

Why Does Open Innovation Matter?

Because it saved this project, live, on deadline weekend. Not in principle. In fact.

A closed-API version of TrailSage dies at step 2 of my odyssey: when the credits ran out, there'd be nothing to swap to, because the model would belong to someone else. The open-weight version just moved. Same code, same API shape, new home, from Hugging Face's router to Google AI Studio hosting the same model family, in one line. When Groq and OpenRouter delisted Gemma, that portability was the difference between a submission and a post-mortem.

What open weights specifically made possible:

• Zero cost per identification. A kid can identify 200 plants on a class trip and it costs nothing.
• One env-var portability, demonstrated twice in one weekend: HF router, Google AI Studio, and (locally) Ollama at http://localhost:11434/v1.
• A real offline path. The same code can point at an Ollama running in Termux on the phone. A fully offline pocket naturalist is one environment variable away (phone CPU makes it slow, but it works, and it's on my roadmap).
• Photos stay in open-weight pipelines. No closed vendor's retention policy between a hiker's camera and their answer.
• Fine-tunable someday. A Gemma tuned on regional flora isn't a negotiation with a vendor; it's a training run.

Closed APIs sell you intelligence. Open weights hand you the keys and the engine.

What I Want Next

A whole walk in airplane mode. The camera, the journal, and the spoken English-then-Hindi answer already live on the phone. The dream is the model living there too, so a trail with no signal still gets a name, a fact, and a recap. Slower, yes. The screen still goes back in the pocket.

Prize Categories

• Gemma (Featured): the core, gemma-4-26b-a4b-it, open-weight, vision, bilingual naturalist answers (with a documented one-line path back to google/gemma-3-4b-it on the HF router)
• Render (Featured): hosts the single service (render.yaml, npm start)
• ElevenLabs (Partner): George (premade voice) narrates the walk recap, with honest reason-coded fallbacks to browser TTS
• Sentry (Partner): traces on every /api/* call via the ESM --import preload.