DEV Community: Eidan Rosado

Linterna: keep the voice behind the memory

Eidan Rosado — Tue, 27 Oct 2026 22:06:12 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine.

What I Built

A transcript can tell you what someone said. It cannot keep the little pause before a punchline, the way they pronounce your name, or the laugh halfway through a story.

I built Linterna, a small, local oral-history app for keeping those things together: the recording, the words, and a way to find the memory again.

For this week's theme, I leaned toward remembrance. A lantern is something you light to keep a person or a story present. In Linterna, you tap a watercolor lantern, tell a memory, and keep it. Smaller lanterns float independently through the night sky, with an occasional shooting star crossing above them.

The interaction is deliberately small. There is no account to create and no conversation feed to manage. Pick a story by date, search its words, and listen.

Demo

The GIF uses synthetic speech to demonstrate the actual browser recording, transcription, save and playback flow. No family recording or private memory is included. The transcription runs against my existing private speech service.

You can tap to start and tap again to finish, or press and hold. Once the recording ends, you can preview your voice and edit the transcript before saving. If transcription is unavailable, the recording can still be saved. Your story does not disappear because a model request failed.

Play recording plays the original voice. Read transcript is a separate narration option. That distinction matters: a generated voice should never quietly replace the person who told the story.

Linterna runs locally. There is no public instance collecting visitors' family recordings.

Code

EdyVision / linterna

Linterna

Keep a voice, not just a transcript. Linterna is a local oral-history lantern record a memory, keep the original audio, find its words later and hear it again No accounts. Memories and recordings stay together in the local SQLite database.

The demo uses synthetic speech, with real private speech inference and local storage.

Requirements

Python 3.11 or newer and make.
A current Chrome, Edge or Safari browser with microphone access.
FFmpeg is optional, recommended for seekable browser recordings. On macOS, brew install ffmpeg; on Debian/Ubuntu, sudo apt install ffmpeg.
Node.js 22+ is needed only for the recorder tests, not to run the app.
A compatible speech endpoint is optional. Recording, local saving, manual transcripts and original-voice playback work without one.

Run

git clone https://github.com/EdyVision/linterna.git
cd linterna
make install
make start

Open http://127.0.0.1:8787. Allow the microphone when you choose to record Keep that terminal running; Ctrl+C…

View on GitHub

Source code and setup instructions

The application code is MIT licensed. The browser interface is plain JavaScript, HTML and CSS, with a FastAPI server and SQLite storage.

How I Built It

The speech models do practical work here:

NVIDIA Nemotron ASR transcribes the recording through my privately hosted, compatible speech endpoint. The tested deployment uses the nemotron-3.5-asr-streaming-0.6b.q8_0 model. NVIDIA documents the underlying Nemotron ASR family here.
Magpie TTS provides optional narration for a written transcript. It is useful for text-only memories, but the saved original recording remains the primary playback option. NVIDIA's speech documentation describes the TTS service.

Although the ASR model supports streaming, this version transcribes a completed recording. I am not claiming live captions or a streaming speech interface.

The browser uses getUserMedia and MediaRecorder to capture real audio. A separate WAV conversion sends mono, 16 kHz audio to ASR, while the original recording is kept for playback. The service URL and model ID are configurable; my infrastructure is not required to run someone else's copy.

SQLite stores the audio and transcript in the same memory record. That keeps saving and deletion together, without loose recording files becoming separated from their stories. SQLite FTS and a small TF-IDF ranking pass support topic searches. The latter is local text ranking, not an embeddings model or an LLM pretending to remember your family.

A few less visible details made a real difference:

Recording data can arrive after Stop is pressed. I wait for the final recorder event before offering the audio to save.
A late microphone permission result cannot restart a cancelled recording.
Failed transcription keeps the draft audio available.
Existing text-only memories survive the schema migration.
Browser recordings can lack useful duration metadata. When FFmpeg is installed, the server remuxes the container without re-encoding the voice, so playback is seekable.
Configured speech credentials stay on the server and are bound to their configured origin. They are not returned to the browser or committed to the repository.

Validation includes ten backend tests and five recorder/WAV tests, plus a real browser run through capture, private transcription, saving, page refresh and original-audio playback. The GIF comes from that working flow.

Why Does Open Innovation Matter?

An oral-history tool should help you keep something, not create another dependency you are afraid of losing.

Open-weight speech models let me run transcription and narration on infrastructure I control. I do not need a new transcription subscription to capture each story, and the app does not need a particular vendor's account system to work.

The recordings and transcripts live in an ordinary local SQLite database. The code is available, the speech endpoint can be changed, and the original voice is retained independently of the model output.

That does not make every configuration automatically private or offline. My tested setup uses private network speech services. Point Linterna at an external provider and that provider receives the audio or text you send. Browser speech captions can also depend on the browser vendor. Keeping that boundary clear is part of building something trustworthy.

I also kept the transcript editable. Speech recognition can get names and words wrong, including this demo's spelling of “Linterna.” The original recording remains there to check against. A model's best guess should not become the only surviving version of a memory.

I used Codex to help develop and validate Linterna and prepare this write-up. The recording, transcription, storage and playback described here were exercised, rather than represented by a mock transcript.

My aim for this final week was simple: make something small enough to finish and useful enough to keep. A voice, a story, and a lantern you can return to.

I taught my release checklist to wait for me

Eidan Rosado — Fri, 23 Oct 2026 04:01:48 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady.

What I Built

I shipped a CLI release. Homebrew was still installing an older version.

That is a particularly annoying kind of failure. The release exists, the build has finished, and you are ready to tell people to try it. Then you discover that “published” and “installable” are two different things.

For a small open-source project, the release checklist tends to live in the maintainer's head. Check the tag. Check the assets. Check the package version. Check the links. Remember what failed. Come back later and remember why.

I wanted to stop carrying that context around.

Miilo Release Caretaker is a release-review workflow that gathers evidence, explains what needs attention, and waits for a person to return.

Mastra runs the workflow and preserves its state. Gemma investigates the evidence. Miilo can display the resulting work and attention through an optional local integration.

The important boundary: the current release checks use labeled sample facts. The Mastra execution, SQLite persistence, live Gemma inference, evidence tools and local native CLI bridge are real. Automatic GitHub, Homebrew and documentation-site inspection are the next adapters, not a capability I am claiming today.

This week's project establishes the part I need to trust before attaching it to real releases: can the workflow pause, survive interruption, explain its findings and preserve the evidence when I return?

Demo

No voiceover required. This captioned GIF condenses three excerpts from the same verified Linux recording:

The journey starts with a sample package at 1.1.0 when the release expects 1.2.0. Gemma reads the evidence, explains the mismatch and the workflow waits for review. A separate process then reopens the saved run and exports its observation to Miilo.

Finally, an explicit recheck switches the sample source to corrected facts. Current attention clears; the original failed evidence remains. The command does not edit a real formula or publish a release.

Code

Miilo Release Caretaker on GitHub · Apache-2.0

The base workflow runs independently of an installed Miilo CLI. With Node.js 22.13 or later:

git clone https://github.com/miilo-ai/miilo-release-caretaker.git
cd miilo-release-caretaker
npm ci --ignore-scripts
npm run check
npm test
npm run journey -- start stale-package

That last command creates a persisted sample review. Live model investigation is explicit opt-in through an owner-only configuration file; ordinary tests never call my model endpoint or yours.

The model configuration guide explains endpoint, model and credential-file configuration. The optional Miilo bridge requires the matching local SDK/CLI changes. Those changes are not yet a published Miilo feature.

Release Caretaker is the new Week 3 project. The broader Miilo observer project predates this challenge.

How I Built It

Mastra gives the work a lifecycle

A checklist script can report a mismatch. I needed the review itself to survive after the command exits.

The workflow runs three parallel check groups: release, package and links. They join into one report, pass through investigation, and reach a human-review step. This is the actual composition from the repository:

.parallel([
  checkStep("release", checks),
  checkStep("package", checks),
  checkStep("links", checks),
])
  .then(aggregate)
  .then(investigationStep(investigator))
  .then(reviewStep())
  .then(verificationStep(checks))
  .commit();

Mastra supplies execution, agent tools and suspension/resumption, with its LibSQL storage adapter backing saved workflow snapshots. The application adds report revisions, deterministic verdicts and the rules for what a review means.

A failed check remains evidence with expected and actual values. A source throwing an error exercises bounded retries. These are different events: finishing the workflow does not make its release checks pass.

Review is tied to a SHA-256 revision of the report. If the submitted revision does not match, the workflow suspends again. Acknowledging a finding preserves its failed verdict. Requesting verification reads the source again and produces a separate verification report.

That distinction matters when I return after a break. My review should describe the evidence I actually saw, and “I looked at it” should never mean “the problem is fixed.”

Gemma makes the evidence easier to act on

The version comparison does not need an LLM. Explaining a collection of findings and proposing a useful next step is where I wanted model assistance.

I used Gemma 3 12B, served on hardware I already own through a configurable OpenAI-compatible endpoint. A Mastra Agent receives the report's evidence identifiers and can use only two tools:

list_evidence: list findings in the current report.
read_evidence: inspect one finding by its identifier.

It cannot run shell commands, install a package, browse arbitrary URLs or change a repository. The final explanation must cite evidence the agent actually read through a tool. Knowing an identifier exists is insufficient.

The endpoint used for the recording supports text chat rather than native function calls. An explicit JSON tool protocol translates Gemma's request into a tool call that Mastra executes. The model does not execute the tool itself. A separate native-tool mode is covered by disposable-server tests.

In the recorded Omarchy run, Gemma inspected the formula evidence and explained the 1.2.0 versus 1.1.0 mismatch using two model requests and one evidence-tool invocation in 11.3 seconds. That is one observed case, not an accuracy benchmark. The endpoint did not report token usage, so token counts and inference cost remain unknown.

Citation checks constrain the explanation; they do not prove the diagnosis is correct. Deterministic code owns the verdict, and a model error leaves that evidence available for human review.

Recovery needed more than a happy-path restart

I tested reopening a saved review in another process. Then I killed a worker during an active parallel check.

The second test exposed a different problem: a branch can finish reading before the parent parallel group commits its state. Restarting can repeat work that appeared complete.

The adapter now checkpoints validated read results by run, scenario, check group and initial/verification stage. Recovery reuses committed results and repeats the unfinished read. A fresh verification cannot quietly reuse the initial stage's facts.

This establishes a tested recovery boundary for the current read-only workflow. It does not establish exactly-once external writes or power-loss durability. A read interrupted before its checkpoint commits may repeat.

Miilo gives the workflow somewhere to be seen

Mastra remains the execution engine. Caretaker normalizes selected saved state into Miilo's workflow observation contract; the native CLI projects it into work items, attention and evidence.

The practical payoff is that a review request can appear alongside other work rather than disappearing into a terminal scrollback. After verification, current attention can clear without deleting the failed evidence that explains the run's history.

The bridge uses owner-only atomic file replacement and explicit enrollment. Tests cover interrupted exports, malformed files, stale sources and isolation between sources. The domain depends on typed ports; framework, storage, model and process details stay outside it.

Why Does Open Innovation Matter?

The two partner technologies doing the work here are Mastra and Gemma. SQLite stores state, TypeScript implements the application, and Miilo supplies observation.

Mastra lets me build on inspectable workflow and agent machinery. Gemma lets me run open-weight inference on existing hardware. Endpoint and model selection are configuration, so the workflow's domain does not require one model vendor or hosted subscription.

This deployment uses a model-serving machine on a private network, so it is not an entirely offline laptop application. It avoids a new public inference service for the demo; existing hardware and electricity still have costs.

More broadly, the checks, review rules and adapter interfaces are open to inspection. Someone can reproduce the sample journey, replace a check source or connect a compatible model without accepting my infrastructure as a dependency.

The Lazy Payoff

I want to stop remembering the release checklist without surrendering the release decision.

The current prototype demonstrates that handoff: findings persist, Gemma explains the evidence, review can wait, and verification preserves history. The next step is to connect bounded read-only GitHub, formula and link adapters so it can review Miilo's actual releases.

I have not measured hours saved or deployed automatic monitoring of real releases. What I have verified is concrete:

All 28 Caretaker tests passed on macOS, Omarchy and the repository's first Node 22 Linux CI run.
Seven local SDK/CLI integration tests and 50 native CLI regression tests passed on Omarchy.
Live Gemma investigation completed from both macOS and Omarchy.
The recorded journey ended with zero current attention records while retaining historical failed evidence.

The repository documents dependency-review and deployment limits. This is a local prototype, not a qualified public service.

For this week's Slow and Steady theme, the useful part is giving the workflow permission to wait. I can leave the terminal without asking a model to make the final decision or relying on myself to reconstruct the whole story later.

Prize Categories

Best Use of Mastra: parallel checks, bounded retries, durable review, cross-process resumption and a scoped evidence-tool agent.
Best Use of Gemma: live, locally served Gemma inference for cited explanations of release evidence.

Miilo: Your AI, in sight

Eidan Rosado — Sun, 18 Oct 2026 06:48:16 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

What I Built

I build software for clients, and AI agents have become part of that work. Keeping track of several sessions introduces a surprisingly ordinary problem: which one is working, which one is waiting, and which one needs me to make a decision?

I wanted a useful answer at a glance. I also wanted it to be a little fun.

Miilo gives your AI workflows a tiny robot crew. Each robot represents an observed system or group of sessions. Working robots move, resting robots close their eyes, and a ?! makes a request for attention hard to miss. Their colors follow your launcher theme.

In Compact, the crew fits into a small space. In Detail, you can see the tasks behind each system. Select a task to inspect its notes, evidence and file changes. When you need help understanding it, Explain brings a self-hosted Gemma model into the conversation.

The immediate audience is freelancers, independent developers and small teams using several coding sessions or automation systems. They need to understand what happened before they approve the next step.

The CLI, SDK, widgets and native Omarchy plugin are now public, with versioned GitHub releases and a Homebrew tap. My commercial hypothesis is that some of these users will pay for custom provider integrations, private deployment and support. Sponsorship can help fund the shared tools. Those are hypotheses I want to validate with real users, rather than invent a subscription price for the post.

Demo

End-to-end video walkthrough coming soon....

The walkthrough follows a complete journey:

Start with an empty Miilo.
Connect four named systems.
Switch between Compact and Detail, then filter what needs attention.
Open a review task and inspect its notes, check result and colored file changes.
Ask Gemma to explain it.
Confirm a supported approval and watch the observed state update.

The recording uses curated task fixtures with live Gemma inference. The approval shown affects an isolated demonstration provider. Existing-session observation is tested separately; the demo does not imply that every coding harness exposes approval controls.

Preview clips

Watch Tokyo Night — Orbit

Watch Osaka Jade — Quiet

Watch task inspection

These three short previews use sample systems and tasks. The separate end-to-end mobile recording will show the longer onboarding and explanation journey.

Project home: miilo.tech.

Code

The entry's main pieces are:

Miilo CLI: standalone observation of explicitly selected existing sessions, plus optional remote connections.
Miilo widgets: the shared robot experience, Rust helper and Omalauncher integration.
Miilo for Omarchy: the native desktop plugin.
Miilo SDK: the provider contracts and shared foundation.
Homebrew tap: installation for macOS and Linux.

Install the CLI with:

brew install miilo-ai/cli/miilo

Versioned downloads are available for CLI v0.2.2, SDK v0.2.0, widgets v1.4.0, and Omarchy plugin v0.1.0. Source availability does not imply an approved marketplace or AUR listing.

The broader Miilo planning and SDK predate this challenge. During Week 2, I built the standalone observer CLI, launcher experience, native Omarchy plugin, and public website, building on that earlier foundation. The CLI repository began October 13, widgets October 14, and the separate native plugin October 16. The website is now live at miilo.tech, hosted on DigitalOcean. Many thanks to our sponsors!

Omalauncher, OWL, and the upstream Quickshell renderer provide the platform. The vendored renderer retains its upstream notices and source provenance. Miilo adds observation integration, original robot behavior, workflow inspection, and model-backed explanations.

How I Built It

Observation starts with a deliberate choice

Miilo observes explicitly selected existing local sessions. Codex and Cursor are available in the packaged CLI; five additional adapters are merged for CLI 0.3.0, with release packages pending. Discovery does not start an agent or silently enroll everything it finds. Local observation can run without a Miilo account, a cloud backend or a listening HTTP service.

For remote automation, users can explicitly connect a provider endpoint. A provider can supply richer workflow state, evidence and supported actions. Local transcript activity alone does not establish authoritative approval or execution state.

Rust owns collection and helper behavior. Domain/application boundaries keep that logic separate from filesystem, HTTP, process and presentation adapters. The native CLI is complemented by a Node client; OWL and QML provide the launcher and desktop presentation.

Seven local agents, one crew

Compatibility update, October 18, 2026: Miilo now has seven local observation adapters in source. Here is when each was added to the Miilo CLI, alongside the agent versions we checked:

Agent	Added in Miilo CLI	Validated agent version or source
Codex	0.2.0	Existing local sessions and live observation
Cursor	0.2.0	Existing local transcripts
Claude Code	0.3.0	2.1.300
OpenCode	0.3.0	1.19.3
Devin CLI	0.3.0	3000.11.92
Grok CLI	0.3.0	1.0.60, local history format 1
Hermes	0.3.0	0.22.2, interactive CLI sessions

Codex and Cursor support is already released. CLI 0.3.0 is merged, with versioned release assets pending. The matching launcher updates are widgets/helper 1.5.0 and native Omarchy plugin 0.3.0, also merged and awaiting publication. Check the repository release links above before updating.

Claude Code, OpenCode and Hermes passed real reply and live follow-up checks. Devin CLI and Grok CLI passed real-reply and selected-session observation checks. Cursor passed real transcript discovery/import; its incremental updates have fixture coverage, while a fresh live follow-up remains unverified.

These are local session adapters, not integrations with every similarly named cloud product. Claude Desktop/Cowork, Devin Cloud, Grok web chat and Hermes one-shot/messaging sessions are outside this scope. The adapters observe bounded visible assistant activity; they do not grant approval, rejection or retry authority. Full version-qualified details live in the CLI compatibility matrix.

The robots share one visual language

Compact and Detail use the same observed state. A system's robot remains recognizable as you move into its tasks.

The orbital tracks and moving lights share their geometry, so the lights stay on their paths. Sleeping systems have muted rings. Different offsets keep the crew from moving in perfect synchronization. A brief electrical zap adds personality to active work.

These effects are decorative. The task notes and evidence remain the basis for decisions. Reduced and disabled motion options are available.

Gemma explains the evidence

For the recorded explanation, I served Google Gemma 3 12B IT on infrastructure I control, using Transformers/PyTorch behind an OpenAI-compatible endpoint. The task’s Explain button calls its provider’s explanation capability; the demonstration provider delegates that request to Gemma. General text chat uses a separate configurable helper path.

The explanation path supplies bounded context for the selected task: its observed state, review notes, linked evidence and available file changes. Gemma turns that context into a short explanation.

The model does not receive authority to approve a task. Where a provider advertises an eligible action, the helper rechecks the observed revision and requires confirmation before submitting it. Changed revisions are rejected, and mutation requests are not automatically retried.

That boundary matters when the answer sounds convincing. A generated explanation is still an explanation; provider state and evidence establish what happened.

Test the actual paths

Validation included 264 official OWL tile/screen evaluations, 36 desktop scenarios, 54 exact portrait comparisons, and the plugin's 24 Node tests. Rust tests cover HTTP behavior, stale revisions and the explanation deadline.

We also clicked through the actual Android emulator. That exposed a real timing issue: a valid model response could exceed the ordinary eight-second request deadline. Read-only explanations now have a twenty-second deadline, while observation and action deadlines stay unchanged.

The published CLI artifacts were also checked on macOS, Linux and Windows, and the public Homebrew installation was validated on macOS and Linux. Installed Omarchy acceptance remains a separate host check. Voice is deferred from this release.

Why Does Open Innovation Matter?

I can choose where inference runs. Self-hosting Gemma lets me keep the explanation service on infrastructure I control. It avoids a mandatory third-party model API bill, although hardware, electricity and operation still cost money.

The model is replaceable. Users can configure an OpenAI-compatible endpoint instead of depending on my model server. Gemma is the open-weight model demonstrated here; it is not a requirement to use one particular hosted AI vendor.

The observation tools can stand on their own. A developer can use the local CLI without signing up for a managed service. Shared provider contracts give other people a way to add integrations.

These choices also make the business question more interesting: can useful open tools lead to sustainable integration and support work?

Privacy still requires care. Selected summaries and evidence can contain sensitive information. Users need to understand the context they share with a configured model; self-hosting alone is not a guarantee of security.

Development and this write-up used AI assistance. The demonstrated workflows were reviewed and exercised, including the actual emulator interaction.

My First User

Released on October 17, 2026. Here are the latest download counts across the projects:

GitHub release-asset downloads, including supporting files, repeat downloads and automation. These are not unique users or verified installations.

Prize Categories

Best Use of Gemma: Gemma 3 12B IT performs the live task explanation shown in the demo. It receives selected workflow context and evidence through an OpenAI-compatible service running on infrastructure I control.

Best Use of DigitalOcean: The public Miilo website is deployed on DigitalOcean App Platform at miilo.tech, with a DigitalOcean starter URL. The demonstrated Gemma inference runs on separate, self-hosted infrastructure.

Best Use of a .tech Domain: I claimed miilo.tech through the Hacktoberfest offer. It is now the public website address, serving the DigitalOcean deployment over HTTPS: miilo.tech.

Foliage Hunter: a Colorado leaf guide on a fine-tuned Gemma 3

Eidan Rosado — Sat, 10 Oct 2026 22:33:21 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

Foliage Hunter is a pocket leaf guide for peeping fantastic fall foliage. You walk, you find a leaf, you snap it. A Gemma 3 model fine-tuned on 18 Colorado species names the tree, Magpie TTS speaks a short guideline, and the find gets pinned to a winding trail of everything you have spotted this season. The screen is meant to be the shortest part of the walk.

It is for anyone on the Front Range who wants to know what they are looking at without stopping to read. The loop is deliberately short. One tap to shoot, a few seconds of a leaf wreath spinning, and the guide starts talking before the whole reply has even been synthesized. You keep walking. Asking the guide free-form questions about the current find (mic in, spoken answer out) is planned for a follow-up; this release identifies and speaks the find, and the mic path only works against the built-in demo guide.

The part that makes it a Colorado guide and not a generic plant app is a LoRA adapter I trained this week on iNaturalist photos of the species you actually meet here: quaking aspen, narrowleaf cottonwood, Gambel oak, boxelder, Rocky Mountain maple, chokecherry, serviceberry, peachleaf willow, plus the urban Front Range imports (green ash, Siberian elm, silver maple) and the fall-color classics (sugar maple, northern red oak, American sweetgum, ginkgo), and the three conifers that frame several trails (ponderosa, Douglas-fir, blue spruce).

Demo

Trail landing	Identification	Past finds

Below are six test finds. The clips are Magpie TTS output the phone played. DEV (like GitHub READMEs) does not allow inline <audio>, so the players live on the project Pages site: Hear the guide.

Find	Listen	What the guide said
Quaking aspen	Play 16 s	That looks like quaking aspen. Nice find. Leaf colors include ember, bronze, gold, and rust. Famous for fluttering leaves and Colorado gold in fall. Bark and buds feed wildlife, not a human snack.
Sugar maple	Play 14 s	That looks like sugar maple. Nice find. Leaf colors include amber, ember, rust, and gold. Famous for tapped sap made into maple syrup. Leaves and raw parts are not snacks.
American sweetgum	Play 16 s	That looks like American sweetgum. Nice find. Leaf colors include bronze, amber, rust, and ember. Famous for star-shaped leaves and spiky gum balls. Not a food plant.
Northern red oak	Play 16 s	That looks like northern red oak. Nice find. Leaf colors include scarlet, burgundy, rust, and amber. Famous for pointed-lobed leaves and strong fall red. Acorns are high-tannin and need processing.
Ginkgo	Play 16 s	That looks like ginkgo. Nice find. Leaf colors include amber, bronze, ember, and burgundy. Famous as a living fossil with fan-shaped leaves. Seeds need careful prep; the fleshy coat is foul and irritating.
Boxelder	Play 13 s	That looks like boxelder. Nice find. Leaf colors include amber, olive, bronze, and burgundy. A maple relative common in yards and washes. Grown for shade, not fruit.

There is no hosted URL on purpose. The app runs in Expo Go on your phone and talks to a model you host. The README's "Run it" and "Taking it outside" sections get you from clone to trail.

I took it outside

I took it out for about ten minutes in Castle Rock, Colorado, on the afternoon of October 10. Phone on cellular, Mac at home on the tailnet, guide server in the lab. Each find now also stores an on-device place name (city and state from the phone's geocoder, no Google Maps API), so the drawer can show where I was standing when I snapped the leaf. Two finds that stuck with me: American sweetgum (star leaves turning hard) and green ash on a street strip. Both pinned as Castle Rock, CO.

American sweetgum	Green ash

The first attempt failed in the field. Every photo came back as a 500 from the gateway, and the server log said PIL.UnidentifiedImageError. Real iPhones shoot HEIC and the simulator never did, so the guide server had never seen one. The fix was to re-encode on the phone with expo-image-manipulator (resize to 1280 px, JPEG at 0.82) before upload, which also cut the payload to a fraction of the original. After that the finds in the table above came in one after another, and the trail filled in as I went.

Code

EdyVision / foliage-hunter

Hacktoberfest Challenge Week 1 - Outdoor Activity Application with Open Weight Models Used

Foliage Hunter

Foliage Hunter is a pocket leaf guide for Colorado's fall. You walk, you find a leaf, you snap it. A Gemma 3 model fine-tuned on Colorado species names the tree, a guide voice tells you what you are looking at, and the find is pinned to a winding trail of everything you have spotted this season. The screen is meant to be the shortest part of the walk.

Why this exists

Built for the Hacktoberfest Open-Source AI Challenge: Week 1 on DEV (October 5–11, 2026). The week's theme was Touch Grass: build something with open-source AI at its core that gets people off the screen and into the world. Fall foliage was literally on the prompt's list of examples, and the Front Range was peaking. Entered under #hf26challenge and the Best Use of Gemma category.

Why open models matter here

A regional model is only possible with…

View on GitHub

MIT. The adapter, the split manifests, and the held-out results are public on Hugging Face:

Adapter: EdyVision/foliage-hunter-gemma3-12b
Experiments: EdyVision/foliage-hunter-experiments

How I Built It

Two open models are in the live loop. Everything else is on-device code or a curated catalog.

Role	What runs
Name the leaf	Gemma 3 12B IT + LoRA (vision identify only)
Speak the guide line	NVIDIA Magpie TTS 357M + NanoCodec (GGUF)
Species notes	Curated catalog on the phone
Leaf colors	On-device JPEG palette
Place on the find	Phone GPS + platform reverse geocode
Free mic conversation	Planned (`/v1/voice/turn`); demo guide only in this release

  phone (Expo Go)                       home lab
 ┌───────────────────┐   Tailscale    ┌──────────────────────────────────────┐
 │ camera / library  │ ─────────────▶ │ LiteLLM  (alias: gemma-foliage)      │
 │ jpeg palette      │                │   └─▶ guide-server (FastAPI + PEFT)  │
 │ species traits    │ ◀───────────── │        Gemma 3 12B IT + LoRA r=16    │
 │ Magpie playback   │   audio/wav    │ nemo-speech (GGUF)                   │
 │ GPS + place name  │                │   Magpie TTS 357M                    │
 │ trail + history   │                │                                      │
 └───────────────────┘                └──────────────────────────────────────┘

The eyes: Gemma 3 12B IT plus a LoRA. I pulled 80 research-grade iNaturalist photos per species through the public API (Colorado place filter for the local tiers), 1,440 photos from 729 observations, and split them 80/20 stratified per species. Training is PEFT LoRA on all-linear modules, r=16, alpha=32, 600 steps, lr 2e-4, with the loss masked to the answer tokens only. The prompt is closed-set on purpose. A general VLM will happily say "maple" for a boxelder; the task I actually have is "which of these 18 is it":

"Identify the plant in the photo.\n"
f"Choose exactly one common name from this list: {choices}.\n"
"Reply with that common name only. No punctuation, no extra words."

The job runs through the Ray Jobs API on a DGX Spark in my home lab. Evaluation is generate-and-match on the 288 held-out photos: 85.1% (245/288). The adapter is 284 MB, base weights are not re-uploaded, and the model card carries the Gemma terms. Gemma is the specialist here: it names the leaf and nothing else. When the Gemma 4 training path on this GPU is unblocked, the LoRA job takes a new BASE_MODEL_ID and the app will not know the difference.

The voice: NVIDIA Magpie TTS, as GGUF. Speech out is Magpie TTS Multilingual 357M under NeMo-Speech.cpp on the same box. After identify, the app constructs a short spoken line from the species name, on-device palette colors, and a curated trait note, then streams it sentence by sentence so the first clip plays while the rest is still being synthesized. Clips save next to the find; reopening a past find replays with no network. A free conversation about the current find (speech in, spoken answer out) is next: that needs a real /v1/voice/turn on the guide path. This release does not ship that route, so the mic only answers through the built-in demo guide.

The app: Expo SDK 57, hexagonal. ports/ holds the interfaces (GuideClient, Speech, Camera, PaletteReader, FindHistory, Locator), adapters/ the Expo and HTTP implementations, application/ the use-cases, and one file (composition/wire.ts) reads env vars and assembles everything. That is what makes the models swappable; the identify flow never learns which TTS or which VLM it is talking to. Leaf colors are extracted on the phone from the JPEG bytes, place comes from the phone's location APIs, and without a server URL the app falls back to a built-in demo guide so the UI still works offline.

export interface Speech {
  speak(text: string, audio?: SpokenAudio, options?: SpeakOptions): Promise<void>;
  stop(): void;
  discardSaved(uris: string[]): Promise<void>;
}

The path outside: Tailscale. The phone and the Mac sit on the same tailnet. make start-ts makes Metro advertise the Mac's 100.x address so the Expo Go QR code works on cellular, and the guide URL points at the lab's tailnet host. No port forwarding, no public endpoint.

Things that bit me

HEIC. Covered above. Conversion was necessary.
License filters leak. My corpus builder accepts cc0, cc-by, cc-by-sa, cc-by-nc, and the manifests still contain 72 -nd and -nc-sa photos because the match was too loose. Images were private training only and are not redistributed, and the real per-photo breakdown is now in the README and the dataset card.

What is next

Free conversation with the guide about the leaf you are holding: mic audio in, a spoken answer that can describe what the model sees, with the current find as context. The UI affordance is already there; the server route and ASR wiring are the missing piece.

Why Does Open Innovation Matter?

A regional model only exists because the base is open. No hosted vision API is going to learn the difference between a Gambel oak and a northern red oak on a Colorado trail for me. With google/gemma-3-12b-it as an open base, a 284 MB adapter and 1,440 photos turn a general model into a local one, and I can publish the adapter and every split manifest so someone in Vermont or Oregon can retrain it for their species in an afternoon. An API with training capabilities can get us to this point, but for self-funded researchers and students, that can become costly as they learn their edge cases or restrictions they may need to set.

Photos and location never leave hardware I control. Every leaf photo is taken somewhere, usually near where I live. Inference runs in my own lab and the phone reaches it over a private tailnet. Place names for finds are resolved on the phone. No third-party Maps key, no per-image bill, no photo or location on a server I do not own. For more private applications such as health and wellness apps, this is everything to researchers. Before we publish sanitized datasets to the cloud, our data must live somewhere and be processed somehow.

Everything is swappable. The app speaks plain OpenAI-compatible HTTP. Point EXPO_PUBLIC_GUIDE_MODEL at a different alias and nothing in the client changes. When the device TTS sounded like a GPS unit, I swapped in Magpie behind the same Speech port in an afternoon. When the Gemma 4 training path on this GPU is unblocked, the LoRA job takes a new BASE_MODEL_ID and the app will not know the difference.

It costs nothing to run and it degrades gracefully. Marginal cost per identification is electricity. Color extraction is on-device, saved clips replay offline, and the demo guide keeps the UI alive with no network. The screen stays short because nothing in the loop is waiting on a vendor.

Prize Categories

Best Use of Gemma. Gemma 3 12B IT is the model that identifies the leaf: closed-set vision over 18 Colorado species, LoRA published, served from my own hardware. Magpie handles speech. The short screen time on the trail is the product of that split.

Built solo. Thanks to the iNaturalist observers whose photos taught the model what a Colorado leaf looks like; every one of them is credited by observation ID in the experiments dataset.

DEV Community: Eidan Rosado

Linterna: keep the voice behind the memory

What I Built

Demo

Code

EdyVision / linterna

Linterna

Requirements

Run

How I Built It

Why Does Open Innovation Matter?

I taught my release checklist to wait for me

What I Built

Demo

Code

How I Built It

Mastra gives the work a lifecycle

Gemma makes the evidence easier to act on

Recovery needed more than a happy-path restart

Miilo gives the workflow somewhere to be seen

Why Does Open Innovation Matter?

The Lazy Payoff

Prize Categories

Further Reading

Miilo: Your AI, in sight

What I Built

Demo

Preview clips

Code

How I Built It

Observation starts with a deliberate choice

Seven local agents, one crew

The robots share one visual language

Gemma explains the evidence

Test the actual paths

Why Does Open Innovation Matter?

My First User

Prize Categories

Foliage Hunter: a Colorado leaf guide on a fine-tuned Gemma 3

What I Built

Demo

I took it outside

Code

EdyVision / foliage-hunter

Hacktoberfest Challenge Week 1 - Outdoor Activity Application with Open Weight Models Used

Foliage Hunter

Why this exists

Why open models matter here

How I Built It

Things that bit me

What is next

Why Does Open Innovation Matter?

Prize Categories